Anonymous Submission
Anonymous affiliation
Under Review at AAAI 2027
Paper (Under Review) arXiv (Coming Soon) Code (Coming Soon)

Figure 1: Framework of VIE-CLIP and geometric interpretation of the multi-task loss.

Abstract

Despite the success of vision-language contrastive learning models such as Contrastive Language-Image Pre-training (CLIP), the inherent information asymmetry between vision and language modalities remains an important challenge: images contain rich visual details, whereas textual descrip-tions provide compressed semantic abstractions, resulting in incomplete cross-modal correspondence. To bridge this gap, we propose Visual Imagination Enhanced CLIP (VIE-CLIP), a framework that leverages visual imagination to in-troduces generative visual priors into vision-language rep-resentation learning. A pretrained diffusion model generates visual priors from textual descriptions, allowing richer vis-ual semantics to participate in representation learning. To alleviate semantic inconsistencies introduced by generated visual priors, a Semantic Correction Fusion Module adap-tively refines these priors under the guidance of original text representations. The corrected representations are then aligned with real images. Without modifying pretrained vi-sion-language encoders, the proposed framework consist-ently improves zero-shot classification performance over the original CLIP baseline. Experiments on ImageNet-1K demonstrate that incorporating visual imagination effective-ly alleviates CLIP’s information asymmetry and provides a promising direction for improving vision-language repre-sentation learning.

Key Contributions

  • Visual Imagination Framework: We propose VIE-CLIP, which bridges CLIP’s information asymmetry by reconstructing latent visual semantics before cross-modal alignment.
  • Semantic Correction Fusion Module (SCFM): We introduce SCFM to adaptively refine generated visual priors using textual semantic constraints, balancing semantic consistency and visual richness[cite: 1].
  • Parameter-Efficient Gains: Experiments demonstrate consistent improvements in zero-shot image classification without modifying pretrained vision-language encoders[cite: 1].