Why algorithm choice shapes every trajectory dataset
A zebrafish trajectory tracking system is only as good as the algorithms running underneath the camera. The hardware captures pixels. The software decides what those pixels mean. Two labs can use the same camera, the same arena, and the same lighting, yet end up with very different datasets because one pipeline handles occlusion well and the other does not. Tracking algorithms sit at the center of that difference.
The job breaks into three parts. Detection finds the fish in each frame. Association links detections across frames so that a fish keeps the same identity. Smoothing estimates the true path when detections are noisy or missing. Different algorithm families handle those parts with different assumptions, and those assumptions determine what the data can support.
Background subtraction and thresholding remain the first line of defense
Most pipelines start with a way to separate fish from background. Background subtraction builds a model of the empty arena and flags pixels that differ from that model. Thresholding then turns the difference map into a binary mask. Morphological operations clean up small holes and stray pixels.
This approach is fast and works well when the background is static and the contrast is decent. A dark fish against a bright arena is easy. Problems appear when lighting drifts, reflections move, or bubbles and debris enter the frame. A shadow can look like a fish. A fish near the wall can merge with the edge. Background subtraction also struggles when fish overlap, because the mask becomes one blob instead of two.
Centroid tracking and nearest neighbor matching handle simple arenas
Once fish are detected, the simplest way to link frames is centroid tracking. The centroid of each blob is computed, and the nearest centroid in the next frame is assumed to be the same fish. Nearest neighbor matching is easy to implement and works well for a single fish or a few fish that stay far apart.
The failure mode is identity swap. When two fish cross paths, the nearest centroid may belong to the wrong fish. The tracker swaps identities, and the trajectory data becomes a mess. This is not a rare edge case. Zebrafish are social and often swim close together. A pipeline that ignores identity persistence will produce beautiful looking tracks that are biologically wrong.
Kalman filters bring motion prediction into the pipeline
A Kalman filter adds a motion model. Instead of assuming the fish stays still between frames, the filter predicts where it should be based on previous velocity. The prediction is compared with the new detection, and the state estimate is updated. This smooths noisy measurements and helps when a detection is missing for a frame or two.
The standard Kalman filter assumes linear motion and Gaussian noise. A zebrafish does not always cooperate. It can turn sharply, dart, or hover. Extended Kalman filters and particle filters relax some of those assumptions, but they add complexity. The practical benefit is still real. Even a simple constant velocity model reduces jitter and improves track continuity in open water.
Hungarian assignment and multi-object tracking keep identities straight
The Hungarian algorithm solves the assignment problem. Given a cost matrix that scores how likely each detection is to match each existing track, the algorithm finds the one-to-one matching that minimizes total cost. Cost can combine distance, appearance, and motion prediction. This is the backbone of multi-object tracking methods such as SORT and DeepSORT.
-
Predict each track position using a motion model.
-
Compute a cost matrix between tracks and new detections.
-
Solve the assignment with the Hungarian algorithm.
-
Update matched tracks and handle unmatched detections as new or lost.
DeepSORT adds an appearance descriptor, often a small neural network, so that fish can be re-identified after occlusion. That helps when two fish cross and then separate. The cost is more computation and more tuning. The assignment step is only as good as the cost matrix, and a poorly weighted matrix can still produce swaps.
| Algorithm family | Strength | Weakness | Best fit |
|---|---|---|---|
| Background subtraction | Fast, simple | Sensitive to light and debris | Static arena, high contrast |
| Centroid with nearest neighbor | Easy to implement | Identity swaps | Single fish or low density |
| Kalman filter | Smooths noise and predicts | Linear motion assumption | Open field, moderate speed |
| Hungarian assignment | Optimal one-to-one matching | Cost matrix tuning | Multi-fish tracking |
| Deep learning tracker | Handles occlusion and appearance | Needs data and GPU | Complex scenes, high variability |
Deep learning trackers raise accuracy and change hardware needs
Deep learning has moved into zebrafish tracking in a big way. Object detectors such as YOLO and Faster R-CNN can find fish without background subtraction. Pose estimation tools like DeepLabCut can track body parts, not just centroids. Re-identification networks can compare appearance across frames. Transformers and Siamese networks are also used for tracking.
The gains are real in crowded or variable scenes. The costs are also real. Training data has to be labeled, often frame by frame. Inference needs a GPU if the frame rate is high. Models trained on one lab's lighting and fish strain may not generalize to another. A deep learning tracker is not a drop-in replacement for a classical pipeline. It changes the hardware budget, the annotation workload, and the validation strategy.
A field example where algorithm mismatch caused identity swaps
A behavioral core facility at a university in southern China was running a multi-tank tracking setup. The initial pipeline used background subtraction and nearest neighbor matching. The tracks looked clean in single fish trials. In group trials, the data showed impossible jumps. Fish appeared to teleport across the arena and then back.
The problem was identity swap during crossings. The team added a Kalman filter for prediction and replaced nearest neighbor with Hungarian assignment. A simple appearance feature based on tail shape was added to the cost matrix. The teleporting stopped. The tracks still needed manual review during dense schooling, but the dataset became usable. The lesson was not that one algorithm is always better. It was that the algorithm has to match the social behavior of the animals.
Matching algorithms to the biological question and the rig
No single algorithm family wins everywhere. A simple centroid tracker is fine for a single larva in a 24-well plate. A deep learning tracker may be overkill for that same assay but essential for a crowded adult tank. The right choice depends on fish density, arena complexity, lighting stability, and the biological question.
It also depends on the rig. A stable camera mount, even illumination, and a clean background make classical algorithms work much better. A shaky mount or flickering light forces the software to compensate for problems that should have been solved mechanically. Tracking accuracy is a system property, not just a software feature.
Hunter Lab (Hunter) builds zebrafish laboratory equipment with attention to the mechanical and optical stability that tracking algorithms depend on. The manufacturing and quality control side supports the consistency that multi-rig behavioral studies require. For labs that want tracking data to reflect biology rather than pipeline artifacts, that stability is part of the foundation.