Rethinking Prototype-based Similarity Learning for Few-Shot Object Detection

An overview of ReSet. Recent prototype-based similarity learning methods construct class prototypes solely from visual support features, often suffering from inter-class similarity margin collapse and insufficient visual cues for localization. ReSet addresses these problems by leveraging text features as semantic anchors to retain only co-activated semantic channels, enlarging inter-class similarity margins for better class separation, and injecting hierarchical ViT features into autoregressive regression stages for precise localization.

Abstract

Few-shot object detection aims to detect novel object categories from only a few labeled examples, avoiding costly large-scale annotation. Recent prototype-based similarity learning approaches enable training-free adaptation by matching query features with class prototypes. However, they suffer from two fundamental limitations: (i) class confusion arising from inter-class similarity margin collapse, and (ii) insufficient visual cues for precise localization, as similarity scores capture only class-level semantic affinity while providing limited spatial information. To address these issues, we introduce two complementary components. Text-Anchored Semantic Mask (TSMa) leverages class-level text features as semantic anchors to identify semantically aligned channels through channel-wise interaction between visual and text features. By suppressing style-induced spurious responses and emphasizing class-intrinsic signals, TSMa enlarges inter-class similarity margins and mitigates class confusion. We further propose Stage-Aligned Hierarchical Autoregressive Regression (SHARe), which reformulates localization as a hierarchical autoregressive process that progressively refines bounding boxes across multiple stages. SHARe leverages the layer-wise characteristics of ViT representations by aligning feature abstraction levels with regression stages: deeper layers guide early coarse localization, while shallower layers rich in edge and texture cues refine spatial details in later stages. Experiments on COCO demonstrate a new state of the art, outperforming the previous best by +10.1 nAP, with extensive analysis validating each component.

Core Contributions

We identify two fundamental bottlenecks of prototype-based similarity learning in few-shot object detection: inter-class similarity margin collapse causing class confusion, and insufficient visual cues for precise localization.
We propose Text-Anchored Semantic Mask (TSMa), which leverages text features as semantic anchors to identify semantically co-activated channels, suppress style-induced spurious responses, and construct class-discriminative Semantic Prototypes.
We introduce Stage-Aligned Hierarchical Autoregressive Regression (SHARe), which reformulates localization as progressive box refinement by injecting hierarchical ViT features in a stage-aligned manner, from high-level semantic cues to low-level spatial details.
Extensive experiments on COCO and Pascal VOC demonstrate state-of-the-art performance across one-shot and 10/30-shot settings, validating the effectiveness of text-anchored semantic prototypes and hierarchical autoregressive localization for prototype-based FSOD.

Why Should We Rethink Prototype-based Similarity Learning?

Prior few-shot object detection methods often require fine-tuning on scarce novel-class examples, which can lead to instability and overfitting. Prototype-based similarity learning avoids this by using query–prototype similarity for training-free novel-class detection, but it over-relies on visual-only similarity for both classification and localization. Visual-only prototypes can contain style-related or instance-specific cues shared across categories, narrowing inter-class similarity margins and causing class confusion. Moreover, similarity scores mainly capture semantic affinity, not the spatial evidence needed to localize object boundaries, shapes, and scales. ReSet rethinks this paradigm by constructing text-anchored Semantic Prototypes to make similarity more class-discriminative, and by injecting hierarchical ViT features to recover spatial cues for precise box refinement.

The analysis below illustrates these two bottlenecks. Visual-only similarity produces small gaps between the top-ranked class and competing classes, whereas ReSet enlarges the margin by suppressing ambiguous prototype responses. In addition, layer-wise ViT visualizations show that localization-relevant cues are distributed across depth: deeper layers capture semantic context, while shallower layers preserve spatial details such as edges and textures.

Comparison of average query–prototype similarity scores. ReSet produces a larger gap between the top-ranked prototype and competing class prototypes, indicating improved inter-class separability and reduced class confusion.

Layer-wise ViT visualizations showing the distribution of localization-relevant cues across different network depths.

Key point: the central issue is not prototype-based similarity itself, but relying on visual-only similarity for both classification and localization. ReSet makes similarity more class-discriminative with text-anchored Semantic Prototypes, and complements similarity-only localization with hierarchical ViT visual cues for progressive box refinement.

Proposed Method

ReSet follows a simple principle: build class-discriminative evidence for what to detect and progressively recover spatial evidence for where to localize. Given a few support examples, ReSet first constructs visual prototypes and refines them into Semantic Prototypes by using text features as semantic anchors. This suppresses style-induced, class-agnostic channels and enlarges inter-class similarity margins for more reliable class separation. For localization, ReSet progressively refines object boxes by injecting hierarchical ViT features, using high-level semantic cues for coarse localization and low-level spatial cues for precise boundary refinement. This design enables both robust classification and accurate localization in prototype-based few-shot object detection.

Overall architecture of ReSet. Our framework consists of two main components: Text-Anchored Semantic Mask (TSMa) and Stage-Aligned Hierarchical Autoregressive Regression (SHARe). TSMa constructs Semantic Prototypes by identifying channels co-activated by visual and text features, thereby mitigating inter-class similarity margin collapse and class confusion. SHARe reformulates localization as a hierarchical autoregressive refinement process, injecting multi-level ViT features in a stage-aligned manner to compensate for the limited spatial cues of similarity scores.

Quantitative Results

ReSet achieves state-of-the-art few-shot detection performance on COCO and Pascal VOC. Compared with prior prototype-based similarity learning methods, ReSet consistently improves novel-class AP (nAP) in both 10-shot and 30-shot COCO settings. On Pascal VOC, ReSet achieves the best overall average nAP50 across three novel splits and multiple shot settings. These quantitative results demonstrate that ReSet improves classification reliability by mitigating inter-class similarity margin collapse, while enabling more precise box localization by injecting hierarchical ViT features to recover spatial cues.

Quantitative comparison on COCO (top) and Pascal VOC (bottom) few-shot object detection benchmarks. Ours (ReSet) achieves state-of-the-art performance across diverse shot settings.

Qualitative Results

ReSet produces more reliable classification and precise localization across diverse few-shot detection scenarios. Compared with DE-ViT and PiDiViT, it corrects class-confused predictions and yields cleaner bounding boxes with fewer redundant detections. These qualitative results support the claim that TSMa mitigates inter-class similarity margin collapse, while SHARe compensates for insufficient visual cues through hierarchical ViT feature injection.

Qualitative comparison between DE-ViT, PiDiViT, and Ours (ReSet). ReSet provides more accurate classification and precise localization.

Citation

@misc{heo2026rethinkingprototypebasedsimilaritylearning,
      title={Rethinking Prototype-based Similarity Learning for Few-Shot Object Detection}, 
      author={KunHo Heo and Seungjae Kim and Wongyu Lee and SuYeon Kim and MyeongAh Cho},
      year={2026},
      eprint={2606.23069},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2606.23069}, 
}