1) An image encoder (e.g. a hierarchical ViT) encodes the frame. 2) The user provides a prompt (point, box, mask) on a chosen frame. 3) A mask decoder generates the mask of the indicated object. 4) In video, a streaming memory module stores the object's features and propagates the mask to subsequent frames, allowing corrections via additional prompts.
Classical segmentation requires training for specific object classes and separate tracking models. PVS lets you interactively isolate any object (zero-shot) and track it in video with a single, unified model.
Encodes the frame into feature representations (e.g. hierarchical ViT Hiera).
Official
Encodes user prompts (points, boxes, masks).
Official
Generates the pixel mask of the indicated object.
Stores object features between frames and propagates masks in video.
Official
Long occlusions or fast appearance changes can cause loss of the object.
Segmenting and tracking many objects simultaneously increases compute cost.
Meta introduces the Segment Anything Model (SAM) and the promptable segmentation task for images (zero-shot).
SAM 2 generalizes PVS to video with a streaming memory, unifying real-time segmentation and tracking.
Time complexity: ~O(1) na klatke (przetwarzanie strumieniowe). Space complexity: O(bank pamieci + cechy klatki).
Throughput is limited by the per-frame image-encoder cost and the size/handling of the memory bank for long sequences and many objects.
Points, boxes or masks as interactive input.
Larger encoders (e.g. Hiera-L) give higher quality at the cost of speed.
How many previous frames/objects the model remembers.
The output depends on the given prompts and frame.
Single-frame segmentation is parallel; video propagation is sequential.
The ViT encoder and per-frame inference benefit from GPU accelerators; smaller variants also run on weaker hardware.
