MedProb: Probing Internal Representations of Vision-Language Models for Medical Question Answering

Nourbakhsh, Erfan; Yang, Ke; Rios, Anthony

Paper

Abstract

Medical visual question answering (Med-VQA) is often assumed to require medical fine-tuning, large models, or complex multi-agent pipelines. We revisit this assumption with MedProb, a lightweight probing framework that predicts multiple-choice Med-VQA answers from frozen VLM representations without free-text generation. Across PATH-VQA, SLAKE, and VQA-RAD, MedProb recovers substantially more answer-relevant signal than prompting and performs stronger than medical VLMs and agentic systems. Probing also reduces the apparent gap between small and large models compared to prompting, suggesting that smaller VLMs contain more recoverable Med-VQA signal than generation-based evaluation reveals. Across 14 matched general-purpose and medical VLM pairs, medical adaptation does not consistently improve this linear decodability. Finally, free-text generation exhibits an answer-position bias of up to 10 percentage points, whereas MedProb also has positional bias, however, it is impacted differently than prompting. Our main results target the multiple-choice/multiclass Med-VQA setting; we additionally show the probe can be extended to open-ended generation via a rejection-sampling scoring procedure.

Experiments

Method Overview

We study Med-VQA as multimodal multiple-choice classification and compare what a VLM produces through prompting with what can be read from its internal representations using simple linear probes.

Overview of MedProb: evaluated VLMs and prompting, probing, and fine-tuned prompting settings.

Figure 2. Overview of the MedProb evaluation framework. Left: general-purpose and medical VLMs evaluated, grouped by backbone. Right: prompts with system prompt, medical image, clinical question, and options (A–D), under three settings: direct prompting (top), MedProb linear probing of frozen hidden states (middle), and fine-tuned then prompted VLMs (bottom).

Results: Main Results

Prompting vs. MedProb

Across PATH-VQA, SLAKE, and VQA-RAD, MedProb recovers substantially more answer-relevant signal than prompting and outperforms agentic systems and many medical VLMs. The gap is especially large on PATH-VQA, where prompting often collapses while probing stays high.

Figure 3. Prompting vs. MedProb accuracy for general (Gemma-4B/27B) and medical (MedGemma-4B/27B) VLMs on VQA-RAD and PATH-VQA.

Model	Method	PATH-VQA		SLAKE		VQA-RAD		Average
Model	Method	F1	Acc	F1	Acc	F1	Acc	F1	Acc
MAM	Agentic	51.85	51.80	39.92	59.51	39.31	59.31	39.62	56.87
MMedAgent	Agentic	36.11	37.00	28.30	43.13	59.80	58.17	41.40	46.10
UCAgents	Agentic	60.55	61.19	78.09	77.10	64.17	71.10	62.36	69.80
HuatuoGPT-Vision-34B	Medical VLM	60.61	61.80	77.89	76.86	83.57	75.66	74.02	71.44
Aloe-Vision-72B-AR	Medical VLM	67.70	69.20	86.80	80.24	75.73	79.46	76.74	76.30
UniMedVL-14B	Medical VLM	52.84	48.00	66.46	58.07	66.24	58.94	61.85	55.00
MedVLM-R1-2B	Medical VLM†	46.73	58.00	49.29	61.20	58.71	51.71	51.58	56.97
InfiMed-RL-3B	Medical VLM†	79.79	79.60	88.16	81.92	83.43	74.90	83.79	78.81
MedMo-8B	Medical VLM†	65.99	67.80	80.64	72.29	73.39	58.94	73.34	66.34
Medix-R1-30B	Medical VLM†	73.00	73.00	82.46	83.61	70.32	72.24	75.26	76.28
InternVL3-1B	Prompting	62.01	62.80	83.68	75.66	76.12	65.02	73.94	67.83
InternVL3-1B	MedProb	82.09	82.20	88.70	83.13	81.51	72.62	84.10	79.32
Qwen3-VL-2B	Prompting	55.78	56.80	68.73	77.83	64.72	64.64	63.08	66.42
Qwen3-VL-2B	MedProb	83.67	83.80	89.02	84.10	80.79	73.00	84.49	80.30
Gemma-4B	Prompting	53.94	57.00	68.93	61.69	57.96	52.47	60.28	57.05
Gemma-4B	MedProb	86.33	86.40	86.43	79.76	80.33	72.24	84.36	79.47
Qwen3-VL-4B	Prompting	59.88	54.60	86.03	77.35	46.24	67.30	64.05	66.42
Qwen3-VL-4B	MedProb	85.69	85.80	90.63	86.02	83.14	74.90	86.49	82.24
Qwen3-VL-8B	Prompting	58.98	58.60	70.36	76.63	46.25	69.58	58.53	68.27
Qwen3-VL-8B	MedProb	86.12	86.20	89.35	84.09	82.34	75.28	85.93	81.86
Llama-3.2-11B-Vision	Prompting	48.94	59.80	68.43	69.88	67.60	68.06	61.66	65.91
Llama-3.2-11B-Vision	MedProb	86.12	86.20	88.21	82.40	83.89	76.04	86.07	81.55
Gemma-27B	Prompting	61.18	62.00	73.12	75.42	69.56	69.58	67.95	69.00
Gemma-27B	MedProb	86.86	87.00	89.17	82.89	83.14	76.04	86.39	81.98
Qwen3-VL-30B	Prompting	63.23	61.20	88.04	80.96	74.53	77.95	75.27	73.37
Qwen3-VL-30B	MedProb	85.70	85.80	91.12	86.75	86.46	81.37	87.76	84.64

Table 1. Performance comparison on medical VQA benchmarks across PATH-VQA, SLAKE, and VQA-RAD. Bold: best per column; MedProb rows highlighted; † trained on PATH-VQA, VQA-RAD, and SLAKE.

Results: Effect of Fine-tuning

Does Fine-tuning Close the Gap?

Fine-tuning the base VLM does not close the probing gap. After in-domain or multi-domain fine-tuning, the probe still outperforms prompting on SLAKE and VQA-RAD. Elicitation, not encoded knowledge, remains the bottleneck.

Figure 4. Probing vs. prompting accuracy after single-domain and multi-domain fine-tuning.

Results: Layer-wise Probing

Where Is the Signal?

Probing accuracy rises with layer depth. Both models start near chance at layer 0; clinically informative signal concentrates in deeper layers.

Figure 5. Layer-wise probing accuracy for Qwen2.5-VL-7B and MedVLThinker-7B.

Results: Answer Option Ordering

Robustness to Option Order

Free-text prompting shows large positional bias (up to ~10 points). MedProb also has positional effects, but is far less sensitive because it reads activations without generating text.

Probing vs prompting under random, correct-first, and correct-last answer orderings.

Figure 6. Probing vs. prompting accuracy under random, correct-first, and correct-last answer orderings.

Results: General vs. Medical

General vs. Medical Models

Across 14 matched pairs, medical adaptation does not consistently improve linear decodability. Six pairs favor the general model; where medical helps, gains are often small (except RL-adapted MediX-R1).

General Model	Medical Model	Δ
Gemma-4B	MedGemma-4B	−4.92
Gemma-27B	MedGemma-27B	−0.95
Open-Flamingo-9B	Med-Flamingo-9B	−1.93
InternVL3-1B	BioMed-InternVL3-1B	+2.36
LLaVA-7B	LLaVA-Med-7B	+0.17
LLaMA3.2-11B-Vision	BioMed-LLaMA3.2-11B	−2.30
Qwen2-VL-2B	BioMed-Qwen2-VL-2B	+0.05
Qwen2.5-VL-3B	MedVLThinker-RL-3B	+2.16
Qwen2.5-VL-7B	MedVLThinker-7B	+1.15
Qwen2.5-VL-32B	MedVLThinker-RL-32B	+1.65
Qwen3-VL-2B	MediX-R1-2B	+11.31
Qwen3-VL-4B	MedMo-4B	−1.65
Qwen3-VL-8B	MedMo-8B	−0.19
Qwen3-VL-30B	MediX-R1-30B	+3.72

Table 2. Probing accuracy delta (Δ) between matched general/medical pairs (averaged across benchmarks). Positive: medical better; negative: general better.

Results: Ablation Controls

Does the Probe Use the Image?

Blanking or shuffling images drops accuracy by 6–18 points vs. the original probe. The probe needs both image and question, not text alone.

Model	Method	SLAKE	VQA-RAD
Llama-3.2-11B-Vision	MedProb	82.4	76.0
	Text-only (image-ablated)	67.2	68.4
	Image-shuffled	66.5	65.8
Qwen3-VL-8B	MedProb	84.1	75.3
	Text-only (image-ablated)	67.7	68.4
	Image-shuffled	65.8	69.1

Table 3. Best-layer probe accuracy (%) with the original image, an image-ablated (blank) control, and an image-shuffled (mismatched) control.