Robots Atlas>ROBOTS ATLAS
Multimodal

PVS

ActivePublished: 31 July 2026Updated: 31 July 2026Published
Key innovation
Generalizing promptable segmentation from images to video: the model takes prompts (points, boxes, masks) on any frame and returns and propagates object masks over time, unifying segmentation with tracking.
Category
Multimodal
Abstraction level
Paradigm
Operation level
ModelInference
Use cases
Data annotation (images and video)Video editing and post-productionRobot perception and manipulationMedical and scientific segmentationInteractive computer-vision tools

How it works

1) An image encoder (e.g. a hierarchical ViT) encodes the frame. 2) The user provides a prompt (point, box, mask) on a chosen frame. 3) A mask decoder generates the mask of the indicated object. 4) In video, a streaming memory module stores the object's features and propagates the mask to subsequent frames, allowing corrections via additional prompts.

Problem solved

Classical segmentation requires training for specific object classes and separate tracking models. PVS lets you interactively isolate any object (zero-shot) and track it in video with a single, unified model.

Components

Image encoderFeature extraction

Encodes the frame into feature representations (e.g. hierarchical ViT Hiera).

Official

Prompt encoderPrompt interface

Encodes user prompts (points, boxes, masks).

Official

Mask decoderSegmentation output

Generates the pixel mask of the indicated object.

Streaming memoryTemporal tracking

Stores object features between frames and propagates masks in video.

Official

Implementation

Reference implementations
SAM 2 (facebookresearch/sam2)
Meta FAIR
Official
Implementation pitfalls
Tracking loss under occlusionMedium

Long occlusions or fast appearance changes can cause loss of the object.

Fix:Additional corrective prompts, longer memory, larger encoders.
Cost with many objectsMedium

Segmenting and tracking many objects simultaneously increases compute cost.

Fix:Smaller encoders, limiting the number of objects, batching.

Evolution

2023
SAM - promptable image segmentation
Inflection point

Meta introduces the Segment Anything Model (SAM) and the promptable segmentation task for images (zero-shot).

2024
SAM 2 - PVS for video
Inflection point

SAM 2 generalizes PVS to video with a streaming memory, unifying real-time segmentation and tracking.

Hyperparameters (configurable axes)

Prompt typeHigh

Points, boxes or masks as interactive input.

Encoder sizeHigh

Larger encoders (e.g. Hiera-L) give higher quality at the cost of speed.

Tracking memory lengthMedium

How many previous frames/objects the model remembers.

Computational complexity

Time complexity: ~O(1) na klatke (przetwarzanie strumieniowe). Space complexity: O(bank pamieci + cechy klatki).

Compute bottleneck

Image encoder and tracking memory

Throughput is limited by the per-frame image-encoder cost and the size/handling of the memory bank for long sequences and many objects.

Depends on
Koszt enkodera na klatkeDlugosc sekwencji / liczba obiektow

Execution paradigm

Primary mode
Conditional

The output depends on the given prompts and frame.

Activation pattern
Input dependent
Routing mechanism

Parallelism

Parallelism level
Partially parallel

Single-frame segmentation is parallel; video propagation is sequential.

Scope
Inference
Constraints
!Mask propagation depends on previous frames (memory).

Hardware requirements

Primary

The ViT encoder and per-frame inference benefit from GPU accelerators; smaller variants also run on weaker hardware.