A new mid-training method, function-aware fill-in-the-middle (FIM), improves SWE-Bench by +2.8 to +5.4 on 7B to 14B models. The technique exploits function call structure to boost coding agents while preserving general coding and tool-use abilities.
Key facts
- SWE-Bench improvement: +2.8 to +5.4 across 7B to 14B models.
- 6 checkpoints and a 400K dataset released on Hugging Face.
- Preserves general coding and tool-use abilities.
- Method exploits function call structure, not just token patterns.
- 14B model gained +5.4, 7B model gained +2.8.
Researchers have released a new mid-training method called function-aware fill-in-the-middle (FIM) that targets the function call structure within code to improve coding agent performance. According to @HuggingPapers, the technique yields SWE-Bench improvements of +2.8 to +5.4 across 7B to 14B parameter models. The method preserves general coding and tool-use abilities, a common casualty of narrow fine-tuning. Six checkpoints and a 400K dataset are now available on Hugging Face.
Key Takeaways
- Function-aware FIM mid-training boosts SWE-Bench by +2.8 to +5.4 on 7B-14B models, preserving general abilities.
- Six checkpoints and 400K dataset open-sourced.
How It Works
Standard FIM trains models to predict missing tokens in the middle of code, but function-aware FIM specifically masks and reconstructs entire function call blocks. This aligns training with the actual structure of agentic coding tasks, where models must invoke functions with correct arguments and order. The 400K dataset contains code examples with annotated function call boundaries, enabling the model to learn call semantics rather than just token co-occurrence.
Benchmark Gains
The +2.8 to +5.4 improvement on SWE-Bench is significant for mid-training techniques, which typically yield single-digit gains. The method outperforms naive FIM and other mid-training approaches, especially on tasks requiring multi-step function orchestration. The 7B model variant saw a +2.8 boost, while the 14B model gained +5.4, suggesting larger models better leverage the structured training signal.
Preservation of General Abilities
A critical finding is that function-aware FIM does not degrade general coding benchmarks or tool-use benchmarks, a problem that plagued earlier mid-training methods. The authors report that HumanEval and MBPP scores remain stable, and tool-use accuracy on API-calling tasks is maintained. This makes the technique suitable for production deployment without requiring separate fine-tuning for different tasks.
What to watch
Watch for adoption of function-aware FIM in production coding agents (e.g., GitHub Copilot, Cursor) in Q3 2026. Also track whether the technique generalizes to larger models (70B+) and to other structured code elements like class definitions or API endpoints.
