UGO: Unified Architecture for General Multi-Object Tracking by Segmentation

Advances in Neural Information Processing Systems (NeurIPS) 2026

• Unified detection and tracking. A shared Hiera backbone combines GECO2-based instance localization with SAM2 frame-to-frame propagation, producing compatible pixel-wise outputs from visual exemplars.
• Training-free consolidation. Energy minimization resolves overlaps, duplicates, and detector–tracker conflicts into mutually exclusive instance masks while suppressing poorly supported hypotheses.
• Hierarchical adaptive memory. Category-level memory adapts the detector’s visual exemplars from reliable tracks, while recent-appearance and distractor-resolving memories improve instance segmentation and identity preservation.
• Reliable track management. Consolidated detections initialize tracks, corrected masks update their memories, and detector confirmation validates trajectories while collapsed tracks terminate cleanly.

Paper: NeurIPS 2026
Source code: GitHub

BibTeX citation:

@inproceedings{Pelhan_2026_NeurIPS,
  author    = {Pelhan, Jer and Lukezic, Alan and Kristan, Matej},
  title     = {UGO: Unified Architecture for General Multi-Object Tracking by Segmentation},
  booktitle = {Advances in Neural Information Processing Systems},
  year      = {2026},
  url       = {https://arxiv.org/pdf/2609.37339}
}