SpaCeFormer: Real-Time Open-Vocabulary 3D Instance Segmentation Without Proposals

Imagine asking a robot: “bring me the Snoopy plush from the corner of the room.” For this to work, the robot needs to find the Snoopy in 3D space — a category it has likely never seen during training. This is the goal of open-vocabulary 3D instance segmentation: given a 3D scene and an arbitrary text query, return the 3D mask of every matching object.

The catch? Existing methods are either painfully slow (minutes per scene), produce fragmented masks, or both. Our new paper SpaCeFormer (project page) shows that you can do this in 0.14 seconds per scene — about 3,900× faster than prior state-of-the-art — and still achieve the best zero-shot mAP on ScanNet200.

TL;DR: A transformer that combines spatial windows with Morton-curve attention, trained on a new multi-view-consistent 3D instance dataset, predicts 3D masks directly from 200 learned queries — no proposals, no multi-stage pipelines.

Why Open-Vocabulary 3D Is Hard

The dominant recipe for open-vocabulary 3D segmentation has been:

1. Generate 3D proposals with a class-agnostic instance segmenter
2. Project each proposal back to 2D images
3. Encode the 2D crops with CLIP to get a category embedding
4. Match to text queries at inference time

This pipeline has two big problems:

Problem 1: It’s slow. Methods like OpenMask3D take 547 seconds per scene — minutes of processing for one room. Open-YOLO-3D and similar methods cut this to ~16s, which is still way too slow for any interactive application.

Problem 2: Fragmented data. To train these models, the field has relied on processing 3D scenes view-by-view with 2D foundation models like SAM2. But single-view processing produces fragmented 3D masks (one chair becomes three pieces) and inconsistent captions (“red chair” / “leather seat” / “office armchair” for the same object).

SpaceFormer fixes both problems.

Fix #1: A Multi-View-Consistent Dataset

We built ScanNet3D-OV, the largest open-vocabulary 3D instance dataset:

• 604K instances across 7.4K scenes from ScanNet
• Each instance has a 3D mask + multi-view-consistent caption

The key insight is that the data generation pipeline must be multi-view-aware from the start.

Multi-view mask clustering aggregates partial 2D SAM2 detections into geometry-consistent 3D instances. A VLM then sees multiple representative views simultaneously to produce a single coherent caption per instance.

Multi-view mask clustering. Instead of processing one frame at a time, we collect SAM2 masks from many views and cluster them by 3D consistency: a cluster must have visibility > 0.3 across views and consensus > 0.9. This produces complete instances — the chair as a single mask, not three fragments. The multi-view approach achieves 54.3% recall on ground-truth instances, dramatically better than single-view methods.

Multi-view captioning. Instead of asking a VLM to caption one cropped image at a time (which produces incoherent labels), we present K representative views simultaneously to the VLM with a structured prompt. The VLM sees the object from all sides and produces a single caption that describes intrinsic properties consistently.

Fix #2: Space-Curve Attention

The second insight is architectural. Most 3D transformers serialize point clouds along Morton (Z-order) curves — space-filling curves that map 3D coordinates to 1D indices while preserving locality on average. The advantage: every attention block sees a fixed number of tokens.

But here’s the problem: Morton curves have jumps. Two points adjacent in 3D can land far apart in the serialized order, breaking spatial coherence within an attention window. For dense prediction tasks like instance segmentation, where you need sharp boundaries, this is a real issue.

Spatial windows (right) keep geometrically adjacent points in the same attention group. Morton curve serialization (left) provides structured diversity but can split nearby points across windows.

Our solution: combine both. SpaceFormer interleaves two attention types:

• Spatial window attention — fixed geometric extent (e.g., $H \times W \times D$ voxels), variable token count. Preserves local geometry, gives 28.6% better spatial coherence than Morton alone.
• Morton curve attention — fixed token count, variable spatial extent. Provides structured diversity and lets distant points exchange information.

Windows use shifted partitions across layers (like Swin) to ensure cross-window connectivity. The combination gives the best of both worlds: sharp local boundaries and long-range context.

Fix #3: Proposal-Free Decoding with 3D RoPE

The third piece: no proposals at all. SpaceFormer predicts instance masks directly from 200 learned queries via cross-attention with the encoded point features.

The decoder uses 3D rotary positional embeddings (RoPE) to encode absolute spatial relationships between queries and point features. After T=3 iterations of cross/self-attention, each query produces a mask, a foreground score, and a CLIP embedding.

The trick is 3D rotary positional embeddings (RoPE). Backbone attention uses relative displacements within windows, but the decoder needs absolute spatial encoding to ground each query in the scene. We extend RoPE to 3D, encoding $(x, y, z)$ via three orthogonal rotation blocks.

Each query produces three things:

1. Instance mask — dot product of the query embedding with point features
2. Foreground score — is this query a real instance?
3. CLIP embedding — the open-vocabulary semantic descriptor

At inference time, you compute cosine similarity between each query’s CLIP embedding and your text query. Done. No proposal networks, no NMS heuristics, no multi-stage post-processing.

Headline Results

The headline numbers:

• 2.8× higher mAP than the prior best proposal-free method on ScanNet200
• 3,900× faster than OpenMask3D
• 119× faster than Open-YOLO-3D, with higher accuracy
• Beats methods that use 2D images as input, while only consuming the 3D point cloud

Zero-Shot Novel Categories

The most fun result: SpaceFormer generalizes to categories completely absent from ScanNet200’s vocabulary.

Left: input 3D scene. Right: SpaceFormer’s predicted mask for the query “Snoopy plush”. The model has never seen this category in training labels — it learns the open-vocabulary descriptor space from CLIP.

We tested queries like “Snoopy”, “X-mas decoration”, “sewing machine”, “ceiling fan”, “globe”, “stroller”, “cistern” — none of which appear in the ScanNet200 label set. SpaceFormer finds them all.

Why It Works: A Coherent Story

Three pieces fit together:

1. The dataset (multi-view consistency) gives the model clean targets — coherent 3D masks paired with coherent captions. Garbage in, garbage out: prior methods were stuck because their training data was already fragmented.

2. The backbone (space-curve attention) preserves local geometric coherence so the model can predict sharp instance boundaries. Pure Morton-curve transformers blur these boundaries by splitting adjacent points across windows.

3. The decoder (proposal-free with 3D RoPE) skips the entire multi-stage pipeline. Every query directly predicts a mask in one forward pass, and the CLIP embedding head gives open-vocabulary semantics for free.

Each fix addresses one bottleneck of the prior pipeline. Together, they collapse a multi-minute multi-stage system into a single 0.14-second forward pass.

Limitations and What’s Next

• Indoor only. All experiments are on indoor scans (ScanNet, ScanNet++, Matterport3D, ARKitScenes, Replica). Outdoor LiDAR would need different sampling strategies.
• Fixed query count. 200 queries works well for typical room-scale scenes but may saturate in extremely dense scenes.
• Gap to multi-modal methods. OpenTrack3D with 2D + 3D inputs still reaches 26.0 mAP on ScanNet200; we close most of this gap from 3D alone, but the modality combination gives the absolute best numbers.

The bigger picture: real-time 3D scene understanding is finally tractable. Bringing inference time from hundreds of seconds down to a tenth of a second changes what applications become possible — interactive robot manipulation, on-the-fly AR object selection, live indoor navigation queries. The proposal-free architecture also opens a clean path to extending the same recipe to outdoor scenes, dynamic objects, and multi-frame temporal understanding.

Citation

@article{choy2026spaceformer,
  title     = {SpaCeFormer: Space-Curve Transformer for Open-Vocabulary
               3D Instance Segmentation without Proposals},
  author    = {Choy, Chris and Lee, Junha and Park, Chunghyun
               and Cho, Minsu and Kautz, Jan},
  journal   = {arXiv preprint},
  year      = {2026}
}