Segmenting and Tracking Every Pixel

Explore video panoptic segmentation in this comprehensive conference talk. Dive into the challenges of assigning semantic classes and track identities to every pixel in a video. Learn about the introduction of two new benchmark datasets, KITTI-STEP and MOTChallenge-STEP, designed to provide long video sequences for studying long-term pixel-precise segmentation and tracking under real-world conditions. Discover the novel evaluation metric, Segmentation and Tracking Quality (STQ), which balances semantic and tracking aspects for sequences of arbitrary length. Examine the evolution of visual scene understanding, the importance of segmentation and tracking, and the process of dataset creation. Explore existing datasets, annotation processes, and track length distributions. Gain insights into metric design, formal definitions, and the components of STQ. Investigate extensions to tracking single-frame baselines and review results on the KITTI-STEP dataset. Access valuable resources and gain a comprehensive understanding of this cutting-edge research in computer vision and machine learning.

Intro
Overview
Why Segmentation matters
Why Tracking matters
Evolution of Visual Scene Understanding
From Tracking to STEP
Contributions
Existing Datasets
Annotation Process
Dataset Comparison (Training Set)
Tracking the most salient classes
Track length distribution
Why a new metric?
Metric design
Formal Definition
Association Quality (AQ)
Segmentation Quality (SQ) We use Intersection-over-Union (loU) to measure segmentation quality
Segmentation and Tracking Quality (STO)
Panoptic-DeepLab
Extensions to Tracking Single-frame baselines
Results on KITTI-STEP
Resources
Summary