The Text-Crutch Conundrum: How VLMs' Spatial Reasoning Depends on Reading, Not Seeing

A startling new study published on arXiv reveals a fundamental weakness in today's most advanced vision-language models (VLMs): their spatial reasoning capabilities appear to depend heavily on text recognition rather than genuine visual understanding. The research, titled "Can Vision-Language Models See Squares? Text-Recognition Mediates Spatial Reasoning Across Three Model Families," demonstrates that when visual elements lack textual identity, VLMs' ability to perform basic spatial tasks collapses dramatically.

The Grid Test: Text vs. Visual Elements

Researchers conducted a deceptively simple experiment using fifteen 15x15 binary grids with varying densities of filled cells (10.7%-41.8%). They presented these grids to three frontier VLMs—Claude Opus, ChatGPT 5.2, and Gemini 3 Thinking—in two distinct formats: one using text symbols (. and #) and another using filled squares without gridlines.

The results were striking. In the text-symbol condition, Claude and ChatGPT achieved approximately 91% cell accuracy and 84% F1 scores, while Gemini achieved 84% accuracy and 63% F1. However, when presented with identical patterns using filled squares instead of text symbols, all three models collapsed to 60-73% accuracy and 29-39% F1 scores.

Critically, both conditions passed through the same visual encoder—the text symbols were rendered as images, not processed as tokenized text. This eliminates the possibility that the models were simply reading text differently than processing images. The performance gap between text and visual conditions ranged from 34 to 54 F1 points across models, indicating a massive disparity in how VLMs handle these fundamentally similar tasks.

Distinct Failure Modes, Common Underlying Deficit

Each model exhibited unique failure patterns when confronted with non-textual visual elements:

• Claude Opus systematically under-counted filled squares
• ChatGPT 5.2 massively over-counted filled squares
• Gemini 3 Thinking hallucinated grid templates that didn't match the input

Despite these different failure modes, all three models shared the same underlying deficit: severely degraded spatial localization capabilities when visual elements lacked textual identity. This suggests that VLMs may be relying on a "text-recognition pathway" for spatial reasoning that dramatically outperforms their native visual processing capabilities.

Implications for Real-World Applications

The findings have significant implications for how we deploy and trust VLMs in practical applications:

Medical Imaging: Given the recent development of fine-tuning techniques for medical VLMs (as noted in the knowledge graph context from February 16, 2026), this research raises concerns about how these models interpret medical images that lack clear textual annotations. If VLMs struggle with simple grid patterns, how reliably can they identify subtle anatomical features or pathology markers?

Autonomous Systems: The knowledge graph context mentions Claude AI's association with autonomous weapons systems and mass domestic surveillance. If VLMs cannot reliably process non-textual visual information, their deployment in critical autonomous systems becomes questionable. Spatial reasoning is fundamental to navigation, object avoidance, and situational awareness.

Scientific Analysis: Many scientific visualizations—from astronomical images to microscopic photography—contain minimal text. VLMs' apparent dependence on textual cues for spatial understanding could limit their utility in analyzing such visual data.

The Architecture Question: Why This Happens

Researchers hypothesize that this performance gap stems from how VLMs are trained and architected. Most VLMs are built by connecting a visual encoder (often adapted from image recognition models) to a large language model. During training, these models see millions of image-text pairs where textual descriptions provide rich semantic context about visual content.

This training approach may inadvertently teach VLMs to rely on text-like patterns in images rather than developing robust visual understanding. When presented with abstract visual patterns lacking clear textual analogs, the models struggle to apply genuine spatial reasoning.

Industry Context and Recent Developments

The study arrives at a critical moment in VLM development. Just one day before this paper's submission (February 16, 2026), researchers announced novel fine-tuning techniques to improve how medical VLMs understand negation in clinical reports. This juxtaposition highlights the field's rapid advancement in textual understanding alongside persistent gaps in visual comprehension.

Anthropic's Claude AI, mentioned in the knowledge graph context, has been associated with ethical AI development, yet this research suggests fundamental limitations in its visual capabilities. The connection to mass domestic surveillance and autonomous weapons systems in the knowledge graph raises important questions about deploying potentially flawed visual systems in high-stakes applications.

Future Research Directions

The authors suggest several promising directions for addressing this limitation:

1. Specialized training on non-textual visual patterns to strengthen native visual pathways
2. Architectural modifications to better integrate visual and linguistic processing
3. Evaluation benchmarks that specifically test spatial reasoning independent of text recognition
4. Multimodal training approaches that emphasize visual understanding as a distinct capability

Conclusion: Beyond the Text Crutch

This research exposes what might be called the "text crutch" problem in contemporary VLMs: their apparent reliance on text-like patterns to perform tasks that should require genuine visual understanding. As VLMs become increasingly integrated into critical systems—from healthcare to autonomous vehicles—addressing this fundamental limitation becomes urgent.

The study serves as a reminder that impressive performance on text-heavy benchmarks doesn't necessarily translate to robust visual intelligence. As the AI community continues to push toward artificial general intelligence, developing models that can truly "see" rather than just "read" visual patterns will be essential.

Source: arXiv:2602.15950v1, "Can Vision-Language Models See Squares? Text-Recognition Mediates Spatial Reasoning Across Three Model Families" (Submitted February 17, 2026)