Modern foundation models achieve remarkable results, but they often face performance bottlenecks when deployed in novel, unseen environments.
To bridge this gap, our latest research, in collaboration with the GLADIA Lab at Sapienza University of Rome, focuses heavily on Open-Vocabulary Semantic Segmentation, a highly challenging domain application where models must identify and pixel-level segment an unbounded set of categories using textual queries. In this post, we are sharing a glimpse into how we are enabling these models to seamlessly handle real-world distribution shifts on the fly.
The Foundation: Semantic Library Adaptation (SemLA)
Our current trajectory builds directly upon our paper, Semantic Library Adaptation: LoRA Retrieval and Fusion for Open-Vocabulary Semantic Segmentation (SemLA). Built upon the CAT-Seg backbone, a state-of-the-art architecture designed for mapping images and textual queries into a shared semantic space, SemLA introduced a novel paradigm for test-time domain adaptation.
Traditionally, domain adaptation requires costly offline fine-tuning or access to source data, which can compromise privacy and slow down deployment. SemLA circumvents this by utilizing a library of LoRA (Low-Rank Adaptation) adapters indexed via CLIP embeddings. By treating CLIP as a "domain navigator", the SemLA framework dynamically retrieves and merges the most relevant adapters based on their proximity to the target domain. This allows the system to construct an ad-hoc, optimized model for every specific input without any additional training.
The Evolution: Data-Free Semantic Routing
Building on the intuition of SemLA, we are pushing the boundaries further by developing a fully data-free routing solution. While our previous iterations relied on dataset centroids, our new approach removes the dependence on any training centroids or private datasets entirely.
Instead of looking at external dataset embeddings, we are "looking inward". LoRA adaptation weights inherently contain rich semantic information that can be actively exploited for online domain adaptation. By examining the internal weights of these adapters, or observing their intrinsic responses to a very small set of public inputs, we can generate unique "fingerprints" for each adapter.
This fingerprinting enables the system to route inputs dynamically. By analyzing the intrinsic properties of the data, the model automatically determines the most suitable adapters and fuses them during inference. Remarkably, this data-free routing approach achieves performance comparable to the original SemLA framework, while operating entirely without access to any training data or stored dataset centroids.
The Road Ahead
While our dynamic semantic routing yields highly encouraging results, we are continuously iterating to build a more granular and robust system.
Our immediate next steps involve conducting systematic ablations utilizing new, highly robust LoRA adapters to ensure peak stability. We are also analyzing the qualitative behavior of our models to better understand how specific semantic components drive the signal used for routing. Finally, our team is exploring the highly ambitious goal of achieving multi-LoRA segmentation on a strict per-patch basis. If successful, this will unlock an unprecedented level of granularity, allowing different regions of a single image to dynamically utilize entirely different adapter combinations.
We look forward to sharing our complete findings, alongside fully reproducible code and artifacts, with the open-source community in the near future.
References
-
Qorbani, R. et al. (2025). Semantic Library Adaptation: LoRA Retrieval and Fusion for Open-Vocabulary Semantic Segmentation. CVPR 2025. Paper
-
Cho, S. et al. (2024). CAT-Seg: Cost Aggregation for Open-Vocabulary Semantic Segmentation. CVPR 2024. Paper
-
Hu, E. J. et al. (2022). LoRA: Low-Rank Adaptation of Large Language Models. ICLR 2022. Paper
-
Radford, A. et al. (2021). Learning Transferable Visual Models From Natural Language Supervision. ICML 2021. Paper