A failure mode, made explicit.
We identify the confidence–reliability mismatch behind wrong-more-confident (WMC) predictions in multimodal adaptation.
ACM Multimedia 2026 MM ’26
Majorization-Guided Test-Time Adaptation
for Vision-Language Models
under Modality-Specific Shift
1 Guangdong University of Technology2 Sun Yat-Sen University
01 THE PAPER
Abstract
When images and text shift differently, a vision–language model can trust the wrong modality. Entropy-based test-time adaptation may then make an incorrect prediction more confident, without correcting the unreliable fusion behind it.
We introduce MG-MTTA, a majorization-guided approach that controls modality contribution before sharpening confidence. It uses branch drift and cross-modal disagreement to guide a lightweight fusion gate, while keeping both encoders frozen.
Experiments show improved robustness under textual and joint shifts, along with fewer wrong-more-confident predictions. The findings highlight the value of adapting how modalities are combined when their reliability changes.
02 THE PROBLEM
More confident.
Still wrong.
When images and text shift differently, confidence stops being a reliable guide. A degraded modality can dominate fusion—and entropy minimization can make the wrong answer even sharper.
ENTROPY-ONLY
Biased fusion Sharper wrong prediction
Controls confidence, without checking modality reliability.MG-MTTA
Reliability-guided fusion Controlled sharpening
Regulates which modality drives the fused decision.
A principled control perspective.
Majorization frames adaptation as constrained posterior de-mixing, with sufficient conditions for beneficial concentration and a modality-dominance threshold.
A small update, where it matters.
Keep both encoders frozen. Adapt only a lightweight fusion module, evaluated through robustness, reliability diagnostics and objective ablation.
03 THE METHOD
Reliability first.
Confidence second.
Separate modality selection from confidence improvement. Observable branch drift and cross-modal conflict guide the fusion gate; entropy then sharpens the fused posterior.
01 / EXTRACT EVIDENCE
Vision encoderVisual logits hv
Text encoderTextual logits ht
Both backbones stay frozen.
02 / ESTIMATE RELIABILITY
Drift + disagreement
- Running-anchor consistency
- Jensen–Shannon divergence
- Class-ranking disagreement
Build a soft prior over the fusion gate.
03 / ADAPT THE FUSION
ONLY THIS MODULE UPDATESReliability-guided gate
Control modality contribution.
Then sharpen the prediction.
LOGIT-LEVEL FUSION
hf = αhv + (1 − α)ht
The gate controls the relative weight of each modality.
ADAPTATION OBJECTIVE
ℒ = ℒent + λgℒgate + λdℒdiv
Sharpen confidence, constrain dominance, preserve diversity.
What does majorization tell us?
Doubly stochastic mixing makes a posterior less concentrated. Majorization characterizes a candidate de-mixing direction, but entropy reduction alone can still sharpen the wrong class. The analysis therefore includes ranking preservation and proximity to the clean decision margin.
These are sufficient conditions involving latent clean quantities. Anchor consistency and conflict are practical control signals; they do not certify correctness for each test sample. See Section 3.4 of the paper ↗.
04 THE EVIDENCE
Better fusion.
Measurable gains.
The strongest gains appear when the two branches differ in reliability. Matched adaptation baselines use the same trainable parameters, isolating the effect of fusion control.
ACCURACY (%) 0 — 100
Textual: label-preserving prompt perturbations, averaged over L1–L4. The visual input remains unchanged. Three-seed averages; Table 1.
View the complete benchmark table
| Method | CIFAR-100C | ImageNet-C | Textual | Joint |
|---|---|---|---|---|
| Source-only | 35.80 | 25.45 | 57.97 | 21.68 |
| TENT | 37.96 | 25.64 | — | — |
| SAR | 41.42 | 29.70 | 56.84 | 25.61 |
| TPT | 36.20 | 24.90 | 56.05 | 24.07 |
| BATCLIP | 42.21 | 30.70 | 60.61 | 29.29 |
| Entropy-only | 37.96 | 26.39 | 55.40 | 21.68 |
| Entropy + Diversity | 38.10 | 26.29 | 55.38 | 21.76 |
| MG-MTTA (Ours) | 38.11 | 26.32 | 66.51 | 26.27 |
Visual: 15 common corruption types at severity 5. Textual: L1–L4. Joint: severity-5 visual corruption combined with L1–L4 text perturbations. TENT is a visual-only reference.
Where it helps. MG-MTTA leads this comparison under textual shift. BATCLIP achieves higher visual-only and joint accuracy. Fusion control complements adaptation of representations and alignment.
FAILURE DIAGNOSIS · FIGURE 3
Fewer confident mistakes.
A wrong-more-confident (WMC) event is an incorrect adapted prediction with higher confidence than before adaptation.
Textual shift
−18.29 pp46.5028.21%
Joint shift
−9.56 pp58.6349.07%
Entropy-only → MG-MTTA. Lower is better. Separate diagnostic stream, distinct from the L1–L4 benchmark.
STRONGEST TEXT-SIDE PROBE · TABLE 2
When text degrades severely.
Under the L5 textual probe, entropy and diversity alone do not explain the recovery.
Source-only25.48%
MG-MTTA65.88%
+40.40 percentage points
Entropy-only: 22.11% · Entropy + Diversity: 22.14%
L5 is a separate ablation condition and is excluded from the main benchmark.
Frozen encoders. Lightweight updates.41.0K trainable parameters1.64 GB peak memoryTable 1 · NVIDIA H20
05 BUILD ON THIS WORK
Read. Reproduce. Cite.
@inproceedings{chen2026mgmtta,
title = {Majorization-Guided Test-Time Adaptation for Vision-Language Models under Modality-Specific Shift},
author = {Chen, Lixian and Huang, Mingxuan and Chen, Yanhui and Lin, Junyi and Shi, Yang},
booktitle = {Proceedings of the 34th ACM International Conference on Multimedia},
year = {2026},
doi = {10.1145/3767308.3836429}
}