Segment Everything Everywhere All at Once - Summary

Arxiv URL: https://arxiv.org/abs/2304.06718v1

Authors: Xueyan Zou, Jianwei Yang, Hao Zhang, Feng Li, Linjie Li, Jianfeng Gao, Yong Jae Lee

Summary:

The paper presents SEEM, a promptable, interactive model for Segmenting Everything Everywhere all at once in an image. It introduces a versatile prompting engine for different types of prompts, including points, boxes, scribbles, masks, texts, and referred regions of another image. The model can efficiently handle multiple rounds of interactions with a lightweight prompt decoder and has a strong capability of generalizing to unseen user intents.

Key Insights & Learnings:

• SEEM is a promptable, interactive model for Segmenting Everything Everywhere all at once in an image.
• The model introduces a versatile prompting engine for different types of prompts, including points, boxes, scribbles, masks, texts, and referred regions of another image.
• SEEM can efficiently handle multiple rounds of interactions with a lightweight prompt decoder.
• The model has a strong capability of generalizing to unseen user intents.
• A comprehensive empirical study is performed to validate the effectiveness of SEEM on various segmentation tasks.

Terms Mentioned: Segmentation, Prompting, Transformer, Visual understanding, Semantic segmentation

Technologies / Libraries Mentioned: GPT, T5, DETR, CLIP, X-Decoder