Abstract:

To address the limitations of 3D Gaussian Splatting (3DGS)—namely, unnatural object-level representations, the inadequacy of explicit semantics for representing complex real-world objects, and the difficulty of ensuring cross-view consistency of pseudo-instance masks— Query-Conditioned Object Response Fields (QC-ORF) was proposed as a weakly prompted framework for constructing QC-ORF in 3D Gaussians. In 3DGS, a rendered pixel is usually produced by the continuous alpha compositing of multiple Gaussians along the viewing direction, and Gaussian primitives are not naturally aligned with real object boundaries. Therefore, directly assigning discrete object labels to individual Gaussians makes the resulting representation susceptible to interference from supporting surfaces, contact regions, occlusions, and pseudo-mask noise. Instead of relying on explicit categorical semantics or treating pseudo-masks as stable cross-view instance-level ground truth, QC-ORF models objects as query-triggered continuous Gaussian memberships. This formulation builds a continuous object response field and provides a unified object-level intermediate representation within the 3DGS framework. Specifically, the original 3DGS representation was augmented with continuous feature and foreground branches, enabling each Gaussian primitive to carry object-relevant feature information in addition to geometry, opacity, and appearance parameters. A multi-channel rendering mechanism was adopted to render RGB colors, semantic features, foreground probabilities, and depth-related information within a unified Gaussian field. During training, dense teacher features extracted from visual foundation models were distilled into the Gaussian feature branch, while pseudo-instance masks generated by SAM2 were utilized as weak prompts rather than strict object labels. Foreground-union supervision was introduced to distinguish object-related regions from the background, and single-frame intra-instance compactness was enforced to reduce local feature variance within each pseudo-instance region. Furthermore, a depth-consistency gating strategy was employed to modulate the reliability of depth-related supervision, ensuring that inconsistent monocular depth or unreliable rendered depth did not dominate the learning of the object response field. These constraints jointly facilitated the learning of stable object-relevant responses under weak and noisy supervision. During inference, object queries were instantiated by seed prompts. Positive and negative prototypes were constructed from the seed view or multiple seed views, and query responses were formulated through positive-sample attraction, negative-sample suppression, and foreground bias. The resulting target-correlated continuous object responses can be rendered as two-dimensional response maps and further decoded into downstream object-level outputs, including single-view segmentation, multi-view aggregated segmentation, and hierarchical grouping. Compared with direct hard-threshold segmentation, the continuous response field preserves the response gradients between target and non-target regions, which is beneficial for handling incomplete masks, uncertain boundaries, and supporting-surface interference. Experiments on public datasets and additionally captured real outdoor 3DGS scenes demonstrated the effectiveness of QC-ORF. The proposed method achieves favorable performance on primary object response metrics, including AUROC, AUPRC, and response gap, while exhibiting robust detail recovery in downstream segmentation tasks. In challenging scenarios characterized by incomplete training masks, complex local structures, partial occlusions, and prominent interference from supporting surfaces, QC-ORF consistently recovered cleaner and more complete target details. Visual results on real outdoor scenes further showed that stable object responses can be obtained under complex illumination, natural backgrounds, and reconstruction noise. These results indicate that stable and continuous object fields can be learned in 3D Gaussian splatting (3DGS) without explicit semantics, using only pseudo-masks as weak prompts, and that the learned response field can serve as a unified intermediate representation for object-level segmentation, hierarchical decoding, and potential scene editing.

Key words: 3D Gaussian splatting, explicit semantics-free, weak prompt learning, object response field, depth consistency gating, hierarchical grouping