Q-Align: Alleviating Attention Leakage in Zero-Shot Appearance Transfer via Query-Query Alignment

Namu Kim*1, Wonbin Kweon*2, Minsoo Kim*3, Hwanjo Yu†3
1KT Corporation, South Korea, 2University of Illinois Urbana-Champaign, USA, 3Pohang University of Science and Technology (POSTECH), South Korea
* Equal contribution   † Corresponding author
Paper arXiv Video Code

Abstract

We observe that zero-shot appearance transfer with large-scale image generation models faces a significant challenge: Attention Leakage. This challenge arises when the semantic mapping between two images is captured by the Query-Key alignment. To tackle this issue, we introduce Q-Align, utilizing Query-Query alignment to mitigate attention leakage and improve the semantic alignment in zero-shot appearance transfer.

Q-Align incorporates three core contributions: (1) Query-Query alignment, facilitating the sophisticated spatial semantic mapping between two images; (2) Key-Value rearrangement, enhancing feature correspondence through realignment; and (3) Attention refinement using rearranged keys and values to maintain semantic consistency. We validate the effectiveness of Q-Align through extensive experiments and analysis, and Q-Align outperforms state-of-the-art methods in appearance fidelity while maintaining competitive structure preservation.

Video

Query-Query Alignment

Zero-shot appearance transfer relies on cross-attention to map the appearance of a reference image to a target structure. Existing diffusion-based methods rely on query–key attention, which often suffers from attention leakage—unintended texture spillover to unrelated regions.

Q-Align introduces Query–Query alignment to compute a clean semantic correspondence between source and reference images. Instead of matching queries to keys, we directly align queries in the shared feature space of the diffusion model, enabling cleaner and sharper semantic matching.

Q-Q Alignment Illustration

Once query correspondences are established, we perform Key–Value rearrangement to align feature memory slots across images. This directly enforces spatial consistency and prevents misaligned texture propagation. The aligned KV pairs are then injected back into the cross-attention layers, refining appearance transfer without distorting structure.

The entire pipeline works without training, masks, or per-image optimization, making it efficient, general, and plug-and-play on any Stable Diffusion backbone.

KV Rearrangement Pipeline

Attention Visualization

We visualize high-resolution attention maps to analyze attention leakage in prior approaches. Baselines exhibit blurry and inconsistent cross-image attention, incorrectly attending to background or irrelevant regions. Q-Align produces sharp, compact, and semantically consistent attention patterns that clearly correspond to object parts, even under large viewpoint and style differences.

Attention Visualization

Animation / Additional Results

Q-Align delivers strong performance across a wide spectrum of categories and styles. It preserves the structure of birds, cars, animals, and architecture while transferring rich, fine-grained appearance details like feathers, fur, metallic reflections, glass transparency, and artistic painting styles.

Our method consistently outperforms baselines in cases with strong pose differences, complex textures, and cluttered backgrounds.

Additional Result
Additional Result

Highlights

BibTeX

@inproceedings{kim2025qalign,
  title={Q-Align: Alleviating Attention Leakage in Zero-Shot Appearance Transfer via Query-Query Alignment},
  author={Kim, Namu and Kweon, Wonbin and Kim, Minsoo and Yu, Hwanjo},
  booktitle={BMVC},
  year={2025}
}