Benchmark Study: Hierarchical Multi-Agent LLM Architecture Achieves F1 0.921 at 1.4x Cost for Financial Document Extraction

A new study provides the first systematic, production-scale benchmark of multi-agent LLM architectures for financial document processing, offering concrete guidance on the cost-accuracy-latency tradeoffs that have remained largely anecdotal. Published on arXiv, the research evaluates four orchestration patterns across 10,000 SEC filings (10-K, 10-Q, 8-K forms) using five frontier and open-weight LLMs, measuring 25 extraction fields covering governance, executive compensation, and financial metrics.

The findings reveal that while more complex architectures can significantly boost accuracy, they come with substantial cost multipliers—but intelligent hybrid configurations can recover most accuracy gains while minimizing overhead. This comes at a critical time as financial institutions increasingly deploy LLM systems for regulatory compliance and analysis, yet face fundamental architectural decisions with limited empirical guidance.

The Four Architectures Benchmarked

The researchers compared four distinct multi-agent orchestration patterns, each representing a different approach to decomposing and coordinating the information extraction task:

1. Sequential Pipeline: A linear chain where each agent processes the document and passes results to the next. This serves as the cost baseline.
2. Parallel Fan-out with Merge: Multiple agents process different document sections simultaneously, with a merging agent combining results.
3. Hierarchical Supervisor-Worker: A supervisor agent decomposes the task, assigns subtasks to specialized workers, and validates/integrates results.
4. Reflexive Self-Correcting Loop: An iterative architecture where agents critique and refine each other's outputs through multiple verification cycles.

Each architecture was evaluated across five LLMs (though the paper doesn't specify which models) on the same corpus of 10,000 SEC filings, extracting 25 structured fields including board composition details, executive compensation figures, debt covenants, and key financial ratios.

Key Results: Accuracy vs. Cost Tradeoffs

The benchmark measured performance along five axes: field-level F1 score, document-level accuracy, end-to-end latency, cost per document, and token efficiency.

The reflexive architecture achieved the highest field-level F1 score (0.943), representing a 5.8% absolute improvement over the sequential baseline. However, this came at 2.3x the cost and more than doubled the latency. The hierarchical architecture emerged as the most favorable position on the cost-accuracy Pareto frontier, delivering 3.4% absolute F1 improvement over baseline at only 1.4x cost.

How Hybrid Configurations Recover Efficiency

The researchers conducted ablation studies on three optimization techniques that, when combined, enable hybrid configurations approaching reflexive accuracy at near-baseline cost:

• Semantic Caching: Storing and reusing extraction results for similar document sections across filings
• Model Routing: Dynamically selecting the most appropriate LLM (frontier vs. open-weight) for each subtask based on complexity
• Adaptive Retry Strategies: Applying verification cycles only to low-confidence extractions rather than universally

These optimizations together demonstrated that hybrid configurations could recover 89% of the reflexive architecture's accuracy gains while increasing cost to only 1.15x the sequential baseline—effectively delivering most of the accuracy improvement for a fraction of the overhead.

Scaling Analysis: From 1K to 100K Documents Per Day

A particularly valuable contribution for production deployments is the scaling analysis, which reveals non-obvious throughput-accuracy degradation curves as systems scale. The researchers simulated processing volumes from 1,000 to 100,000 documents per day, finding that:

• All architectures experience some accuracy degradation at scale due to increased contention and resource constraints
• The degradation is non-linear and architecture-dependent, with parallel designs showing the steepest declines
• Hierarchical architectures maintain the most stable accuracy-cost profile across scaling ranges
• Capacity planning must account for these degradation curves rather than assuming linear scaling

These findings provide concrete guidance for infrastructure planning and architecture selection based on expected processing volumes.

Implementation Considerations for Financial Environments

The paper emphasizes several considerations specific to regulated financial environments:

1. Auditability: Hierarchical and reflexive architectures naturally produce audit trails through supervisor decisions or critique chains, while parallel architectures may require additional logging.
2. Determinism: Financial applications often require reproducible results, which can be challenging with non-deterministic LLM sampling. The study notes temperature settings significantly impact consistency.
3. Regulatory Compliance: Certain architectures may better support compliance requirements—for example, hierarchical designs allow for explicit validation steps that can be mapped to control frameworks.
4. Model Selection: The benchmark shows that architecture choice interacts significantly with model capabilities, with some patterns benefiting more from frontier models while others work adequately with open-weight alternatives.

gentic.news Analysis

This benchmark arrives precisely when the industry needs it most. As our coverage has documented, multi-agent AI systems have seen a surge of research interest, with a technical framework outlining four architecture patterns and a three-layer governance model for enterprise deployment published just last week. This study provides the empirical validation that was missing from those theoretical frameworks, grounding architectural decisions in concrete cost and accuracy metrics.

The finding that hierarchical architectures occupy the sweet spot on the cost-accuracy Pareto frontier aligns with emerging industry practice but now has numbers to back it. This contradicts some earlier assumptions that more complex reflexive or massively parallel designs would dominate. The 1.4x cost multiplier for a 3.4% F1 improvement represents a quantifiable tradeoff that engineering teams can now evaluate against their specific accuracy requirements and budget constraints.

Notably, this research follows a pattern we've observed in recent arXiv publications: a shift from purely algorithmic innovations to systematic engineering studies of production deployment considerations. Just yesterday, arXiv published research on LLMs de-anonymizing users from public data trails, and earlier this week, a study on whether reasoning models enhance embedding models. This trend toward practical, scaled evaluation reflects the maturation of LLM technology from research curiosity to production infrastructure. The hybrid optimization results—recovering 89% of accuracy gains at 1.15x cost—demonstrate that intelligent system design can dramatically improve efficiency without sacrificing performance, a crucial insight for cost-conscious enterprises.

Frequently Asked Questions

What are the four multi-agent architectures benchmarked in this study?

The study systematically compares four orchestration patterns: (1) Sequential Pipeline (linear chain of agents), (2) Parallel Fan-out with Merge (simultaneous processing with result merging), (3) Hierarchical Supervisor-Worker (task decomposition with validation), and (4) Reflexive Self-Correcting Loop (iterative critique and refinement). Each represents a different approach to coordinating multiple LLM agents for document processing tasks.

Which architecture provides the best balance of accuracy and cost for financial document processing?

The hierarchical supervisor-worker architecture achieves the most favorable position on the cost-accuracy Pareto frontier, delivering a field-level F1 score of 0.921 (3.4% absolute improvement over the sequential baseline) at only 1.4x the cost. While the reflexive architecture achieves higher accuracy (F1 0.943), it comes at 2.3x the cost, making the hierarchical design the recommended choice for most production deployments where cost efficiency matters.

How much accuracy can be recovered through hybrid optimizations like caching and routing?

The ablation studies show that hybrid configurations combining semantic caching, model routing, and adaptive retry strategies can recover 89% of the reflexive architecture's accuracy gains while increasing cost to only 1.15x the sequential baseline. This means practitioners can achieve most of the accuracy improvement of the most complex architecture for a fraction of the overhead through intelligent system design.

What scaling considerations should teams plan for when deploying these architectures?

The scaling analysis from 1K to 100K documents per day reveals non-linear throughput-accuracy degradation curves that vary by architecture. All designs experience some accuracy degradation at scale due to resource contention, with parallel architectures showing the steepest declines. Hierarchical architectures maintain the most stable profile. Capacity planning must account for these degradation curves rather than assuming linear scaling behavior.