SOURCE-LINKED INTELLIGENCE

TwinICL: Diagnosing Multimodal In-Context Learning through Paired Counterfactuals

arXiv · AI, language, vision and robotics · article · Sep 14, 2026 · UTC

In-context learning (ICL) enables models to infer tasks from demonstrations, but existing benchmarks generally lack matched text and image versions needed to compare ICL performance across modalities. We introduce TwinICL, a procedurally generated benchmark providing such pairs for controlled comparison. Across six open-weight models and 38 tasks, multimodal ICL consistently underperforms text-only ICL, with gaps varying by task family. To test whether this gap can be recovered, we target visual access, task framing, and reasoning through three interventions. Their combination recovers strong

Read original source ↗ Open in workspace

recordType

paper

region

Global