AI systems
Breaking SAM 2
Stress-testing video segmentation
A 52-clip benchmark of when SAM 2 video segmentation fails: it copes with extreme changes of scale, but breaks when the object itself is cut, poured or melted.
Key result With a verified first box, SAM 2 failed on 12 of 14 clips where the object is cut, poured or melted, and on none of 14 control clips (as reported in the study).
Research study Status: Completed Academic · team of 4
Workflow diagram
Research question
Video segmentation benchmarks mostly contain objects that keep their shape, size and appearance. The study asks when, how and under which transformation SAM 2 fails, and separates a wrong first box from later drift.
My Role & Leadership
Project Owner · Technical Lead · Developer. Owned the study and led its technical direction in a four-student research team, and worked hands-on in its engineering and analysis.
- Led the study design: a 52-clip benchmark in four categories that scores whether the first mask lands on the right object apart from how long tracking survives.
- Set the evaluation rules: a failure rule fixed before any model ran, and time to failure counted from each clip's event frame.
Key findings
- Object cut, poured or melted
- 12 of 14
- Illumination change
- 5 of 10
- Extreme scale or viewpoint
- 1 of 14
- Control clips
- 0 of 14
Source: Report Table 1: SAM 2 with a verified first box; failure means two consecutive frames below J = 0.5.
On the topology clips, SAM 2 failed a median of 2.0 s after the event. Grounding DINO failed to initialise on 11 of 52 clips; on the other 41, Grounded-SAM 2 was 2.0 J&F points below SAM 2. Shifting the first box cost 5.5 J&F points and enlarging it 6.6.
Method
The benchmark keeps two questions apart: whether the first mask lands on the right object, and how long tracking survives once the object changes. Per-frame scores show how a failure happens, not only when.
Choose 52 clips and their categories, an event frame and a fixed text prompt for each, before running any model.
Prompt frame 0, either with a verified box (SAM 2) or with text through Grounding DINO (Grounded-SAM 2).
Propagate the mask through the clip and score every frame: J, F, area error, connected components and temporal IoU.
Apply the failure rule and measure time to failure from the event frame, with recovery and abrupt-or-gradual labels.
Repeat with a shifted and an enlarged first box to measure prompt sensitivity.
What the team built
- A benchmark of 52 clips in four categories (topology or state change, extreme scale or viewpoint, illumination, controls), fixed before any model was run.
- Two pipelines: SAM 2 prompted with a verified box, and Grounded-SAM 2 initialised from a text prompt.
- A pre-registered failure rule (two consecutive frames below J = 0.5) with survival analysis from each clip's event frame.
- A prompt-sensitivity test: the verified box shifted by 10% and enlarged by 20%.
Methodological choices
- The failure rule was fixed before any model ran, so the threshold could not be tuned to the results.
- Initialisation is scored apart from tracking: a text-prompted run that starts on the wrong object is an initialisation failure, not drift.
- Time to failure is counted from each clip's event frame, so the study measures how soon after the change a model breaks, not only whether it does.
Built on
- SAM 2.1 and Grounding DINO are third-party pretrained models: the study evaluates them and trains neither.
- Every clip comes from a public research dataset: M3-VOS, VOST, DAVIS 2017 and VOT-ST2022.
Limitations
A small benchmark with possible selection bias, one checkpoint per model, event times estimated from ground truth, and possible overlap between SAM 2.1's undocumented training data and DAVIS. No video frame is shown: the clips belong to third-party datasets.
Key technologies
Evidence
- A three-page research report by the four authors, with the method, Table 1 and the limitations.
- The numbers are as reported in the report; they were not rerun for this site.