Vision-Language Models

Technical guide published on Medium for efficient fine-tuning of VLMs using LoRA and quantization

methods:

Low-Rank Adaptation (LoRA),Quantization

benefit:

Reduces computational cost and memory footprint for custom VLM training

Research reveals VLMs struggle with fine-grained visual classification despite excelling at complex reasoning

New research published on arXiv reveals VLMs' spatial reasoning collapses when visual elements lack text labels, exposing fundamental limitations.

finding:

Models performed dramatically worse identifying filled squares vs. text symbols

Researchers develop novel fine-tuning technique that improves how medical VLMs understand negation in clinical reports

method:

causal tracing to identify neural network layers

application:

medical imaging and clinical reports