Segmentation-guided association refinement in multiple object tracking

Multiple Object Tracking (MOT) aims to detect all objects in a video sequence and maintain consistent identities across frames. While Tracking-by-Detection (TbD) remains the dominant paradigm due to its effectiveness, its reliance on object detectors makes it vulnerable to failures, particularly in occlusion scenarios. These failures often result in fragmented trajectories and degraded tracking performance. In this work, we propose SGAR-MOT—which stands for Segmentation-Guided Association Refinement in Multiple Object Tracking—a novel framework that enhances TbD pipelines by recovering object tracks that a conventional pipeline would consider lost due to detection errors. SGAR-MOT introduces the Object Recovery Protocol (ORP), which leverages segmentation masks and integrates a video segmentation method with a Vision Transformer-based module, the Mask Instance Resolver (MIR), to assess track continuity. In this way, SGAR-MOT incorporates mask-level evidence and a learned identity-consistency verification step through MIR to determine whether a track should be recovered. Experimental results across multiple benchmarks, including MOT20, SportsMOT and VisDrone, demonstrate that SGAR-MOT consistently outperforms its baseline tracker and achieves competitive results against state-of-the-art methods. These improvements highlight the effectiveness of integrating segmentation-guided reasoning into a conventional TbD pipeline, providing a hybrid tracking paradigm that is more robust to detector failures. The code is available at https://github.com/ManuelBGomez/SGAR-MOT.

Palabras clave: Multiple Object Tracking, Tracking-by-Detection, Segmentation-Guided Object Recovery,