LLMForge scores 0.89 on a 97-design CAD benchmark, with deepseek-v3" class="entity-chip">DeepSeek-V3.2 and Qwen3-235B-A22B topping the leaderboard. A VLM critic achieves 100% watertight meshes but exposes cylinder geometry as a persistent failure mode.
Key facts
- 97 engineering design problems in the LLMForge benchmark.
- Top 4 models score 0.885–0.890 overall mean under IterTracer.
- 98.97% mesh success rate under IterTracer.
- IterVision yields 100% watertight meshes on leading model.
- Seven foundation models evaluated including DeepSeek-V3.2 and Qwen3-235B-A22B.
A new paper from researchers J. de Curtò, Victoria Guillén, and I. de Zarzà introduces LLMForge, a unified framework for evaluating foundation models on parametric 3D CAD generation from natural language. The benchmark covers 97 engineering design problems across four geometry families: plates with holes and bolt circles, multi-feature boxes, flanged cylinders, and L-brackets.
Seven models were tested: DeepSeek-V3.2, Qwen3-235B-A22B, Llama-3.3-70B, Gemma-3-27B, GLM-4.5, MiniMax-M2.1, and INTELLECT. Under the IterTracer critique regime — which uses a Phong-shaded ray-trace renderer with analytic visual metrics (silhouette IoU, hole visibility, edge clearance, aspect-ratio conformance) — the top four models clustered tightly, with an overall mean score between 0.885 and 0.890 and a 98.97% mesh success rate. This suggests that compact instruction-tuned models can match substantially larger systems, a finding with implications for cost-sensitive industrial deployments.
Key Takeaways
- LLMForge scores 0.89 on 97-design CAD benchmark.
- VLM critic achieves 100% watertight meshes but cylinders remain a failure mode.
VLM Critic Outperforms Analytic Scoring
The second critique regime, IterVision, replaces the analytic scorer with a VLM semantic critic (Qwen2.5-VL-72B) that evaluates rendered views via chain-of-thought visual reasoning. IterVision achieved 100% watertight mesh generation on the leading model, a notable improvement over IterTracer. However, the paper reports that visual and semantic scoring diverge most on rotationally symmetric geometries such as cylinders, indicating a systematic difficulty that may require specialized prompting or dataset augmentation.
Failure Modes and Industrial Implications
The study identifies key failure modes: misaligned hole patterns, incorrect aspect ratios, and missing features on complex flanged cylinders. The authors discuss CAD-oriented prompting strategies and the implications for scalable automated mechanical design. Notably, the benchmark does not yet include freeform surfaces or assemblies, limiting immediate industrial applicability.
The paper is submitted to arXiv on July 6, 2026, and is categorized under Computer Science > Artificial Intelligence.
What to watch
Watch for extensions of the LLMForge benchmark to include freeform surfaces and multi-part assemblies, which would stress current VLM-based critics. Also monitor whether DeepSeek or Qwen release CAD-specific fine-tuned variants in response to the cylinder failure mode.
Source: arxiv.org
