Data science & ML
Tracking by Detection & Optical Flow
Trading detector calls for speed in a person tracker
A person tracker that runs a detector only every K frames and follows boxes with optical flow in between, with the box update and the choice of K worked out in a course assignment.
Key result Re-detecting every 8 frames ran at 2.8 frames per second against 1.0 with a detection on every frame, on a 48-frame CPU test: a throughput figure only, since there is no ground truth to score tracking quality.
Technical lab Status: Completed Academic
Workflow diagram
The problem
Running a detector on every frame is accurate but slow. Optical flow can carry the boxes between detections, but how often must the detector run?
My Role & Leadership
Project Owner · Technical Lead · Developer. Owned and implemented the box update and the choice of the re-detection interval in a course assignment: each box moves by the median displacement of its valid flow tracks (at least six points, otherwise the box is dropped), and a sweep over K chose re-detecting every 8 frames.
- Led the evidence rule: the median, not the mean, displacement, so points that slide onto the background cannot drag the box, and a box with too few valid points is dropped rather than guessed.
The algorithm
Every K frames a pretrained detector finds people; in between, points inside each box are tracked with pyramidal Lucas-Kanade flow, and each box moves by the median displacement of its points.
What I developed
- A box update from the median displacement of the box's valid flow tracks, robust to up to half of them being wrong.
- A minimum of six valid points per box, below which the box is dropped rather than guessed.
- A sweep of the re-detection interval K and the choice K = 8, with the reasoning for it.
Result
- Detector on every frame (K = 1)
- 1.0 frames per second
- Every 8 frames (K = 8, chosen)
- 2.8 frames per second
- Once per clip (K = 48)
- 4.9 frames per second
Source: Saved notebook timing, 48-frame clip at 10 fps, CPU.
Implementation decisions
- The median, not the mean, displacement: a few points that slide onto the background cannot drag the box with them.
- Drop a box with too few valid points instead of moving it on weak evidence.
What was provided
- The pretrained person detector, the Lucas-Kanade tracker with point seeding, the tracking loop and the timing harness.
- The test clip is OpenCV's sample video.
Key technologies
Known limits
Throughput only: there is no ground truth, so tracking quality was judged by eye. Boxes move but do not change size. The test clip shows identifiable pedestrians, so no frame is shown.