Translating Inference-Time Control to Radiology Vision-Language Models: Activation Steering for Pneumonia Classification on Chest X-rays

The study evaluates the effectiveness of Contrastive Activation Addition (CAA) for enhancing pneumonia classification in three frozen vision-language models (VLMs): MedGemma-4B-IT, NV-Reason-CXR-3B, and CheXOne-3B, using the Kermany pneumonia test set. Notably, NV-Reason-CXR-3B showed significant performance improvement in F1 score from 0.7692 to 0.8727 with image-conditioned steering, while CheXOne-3B demonstrated a smaller increase. This research suggests that activation steering can effectively modify VLM behavior for medical diagnostics without requiring model weight updates, offering a potentially lightweight solution for practitioners in medical AI applications.