MG-MTTA

ACM Multimedia 2026 · MM ’26

Majorization-Guided Test-Time Adaptation for Vision-Language Models under Modality-Specific Shift

Lixian Chen1 Mingxuan Huang2 Yanhui Chen1 Junyi Lin1 Yang Shi1

1 Guangdong University of Technology · 2 Sun Yat-Sen University

Reliability-aware fusion for multimodal test-time adaptation under modality-specific shift.

Paper Code BibTeX
Four-panel motivation figure comparing balanced fusion, text-biased fusion, entropy-only adaptation, and the reliability-aware MG-MTTA gate.
Motivation. Entropy-driven adaptation can sharpen an incorrect prediction when an unreliable modality dominates fusion. MG-MTTA regulates modality contribution using reliability and conflict cues before confidence sharpening.
Abstract

Vision-language models can face asymmetric visual and textual shifts at deployment. These shifts expose a multimodal failure mode in which an unreliable branch remains overconfident, dominates fusion, and causes entropy-based test-time adaptation to sharpen an incorrect prediction. We model this behavior as doubly stochastic posterior mixing and cast adaptation as constrained de-mixing.

MG-MTTA freezes both encoders and updates only a lightweight fusion module. Running-anchor consistency estimates relative branch drift, while cross-modal conflict regulates modality dominance before entropy sharpening. On ImageNet, MG-MTTA improves top-1 accuracy from 57.97% to 66.51% under textual shift and from 21.68% to 26.27% under joint shift.

Method

Reliability-aware posterior control

MG-MTTA separates modality selection from confidence improvement: the fusion gate controls which branch should dominate, then entropy minimization sharpens the fused posterior.

01

Frozen evidence

Visual and textual encoders remain fixed; adaptation is confined to a lightweight fusion module.

02

Reliability cues

Running-anchor consistency tracks branch drift, while JS and ranking disagreement capture cross-modal conflict.

03

Guided fusion

A soft gate prior limits unreliable modality dominance before posterior sharpening.

Fusionhf = αhv + (1 − α)ht
Objectiveℒ = ℒent + λgℒgate + λdℒdiv
Results

Strongest gains under asymmetric reliability

Textual shift
57.97→66.51
+8.54 pp over Source-only
Joint shift
21.68→26.27
+4.59 pp over Source-only
Textual WMC46.50 → 28.21−18.29 pp
Joint WMC58.63 → 49.07−9.56 pp
Textual L525.48 → 65.88+40.40 pp
Trainable params.41.0Kencoders frozen

Scope. MG-MTTA controls late fusion and complements representation-level adaptation; it does not reconstruct evidence already lost inside an encoder.

BibTeX
@inproceedings{chen2026mgmtta,
  title     = {Majorization-Guided Test-Time Adaptation for Vision-Language Models under Modality-Specific Shift},
  author    = {Chen, Lixian and Huang, Mingxuan and Chen, Yanhui and Lin, Junyi and Shi, Yang},
  booktitle = {Proceedings of the 34th ACM International Conference on Multimedia},
  year      = {2026},
  doi       = {10.1145/3767308.3836429}
}