LabelMaker 🎨

Automatic Semantic Label Generation from RGB-D Trajectories

3DV 2024

1ETH Zürich 2ETH AI Center 3Google 4Microsoft
Paper arXiv Code

LabelMaker bundles a collection of state-of-the-art segmentation models with different sets of predicted classes in a neural field. LabelMaker can refine existing annotations and produce highly accurate 2D as well as 3D labels on ScanNet. At the same time, it opens new possibilities to rapidly label large-scale datasets without human effort.

ARKit LabelMaker 🏠

A New Scale for Indoor 3D Scene Understanding

1ETH Zürich 2Stanford 3Google 4Microsoft 5Uni Bonn / Lamarr Institute
Paper arXiv Code Dataset ScanNet200 SOTA

We complement ARKitScenes dataset with dense semantic annotations that are automatically generated at scale. This produces the first large-scale, real-world 3D dataset with dense semantic annotations. Training on this auto-generated data, we push forward the state-of-the-art performance on ScanNet and ScanNet200 with prevalent 3D semantic segmentation models.



Video

Qualitative results on ARKitScenes

LabelMaker can automatically create highly accurate 2D and 3D semantic segmentations. Below we show 3D semantic segmentation on ARKitScenes reconstructions.
Ashcan Backpack Bag Bannister Basket Bathtub Beanbag Bench Blanket Book Bookshelf Bottle Box Bucket Cabinet Ceiling Chair Chest of drawers Clock Column Curtain Cushion Decoration Display Door Doorframe Floor Footstool Furniture Lamp Laptop Light Microwave Oven Paper Paper cutter Picture Plant Printer Radiator Sack Stairway Table Teddy Tray Wall Window
Apparel Ashcan Bannister Basket Bed Blanket Book Bookshelf Bottle Bowl Box Cabinet Ceiling Chair Chest of drawers Clock Coffee table Curtain Cushion Desk Display Door Doorframe Floor Gat Lamp Light Mat Mirror Picture Radiator Shoe Shower stall Soap dispenser Stand Stool Swivel chair Table Teddy Telephone Towel Vent Wall Window
Ashcan Backpack Bed Blanket Book Bookshelf Bottle Box Cabinet Ceiling Chair Chest of drawers Clock Computer Computer keyboard Countertop Crate Curtain Cushion Decoration Desk Display Door Doorframe Fan Floor Footstool Kettle Lamp Mat Picture Plant Printer Radiator Swivel chair Table Teddy Telephone Towel Tray Wall Wardrobe Window

BibTeX

@inproceedings{Weder2024labelmaker,
title = {{LabelMaker: Automatic Semantic Label Generation from RGB-D Trajectories}},
author={Weder, Silvan and Blum, Hermann and Engelmann, Francis and Pollefeys, Marc},
booktitle = {International Conference on 3D Vision (3DV)},
year = {2024} }
@inproceedings{Ji2024arkitlabelmaker,
title = {{ARKit LabelMaker: A New Scale for Indoor 3D Scene Understanding}},
author={Ji, Guangda and Weder, Silvan and Engelmann, Francis and Pollefeys, Marc and Blum, Hermann},
year = {2024} }