Alibaba released MobilePA-Bench, an interactive benchmark for mobile planner agents with 1,705 tasks and 212 tools. The stateful design targets long-horizon planning gaps left by static suites.
Key facts
- 1,705 tasks in MobilePA-Bench
- 212 realistic tools included
- Four evaluation dimensions: tool use, memory, skills, sub-agents
- Released by Alibaba, announced via @HuggingPapers
- Stateful, interactive benchmark design
Alibaba released MobilePA-Bench, an interactive, stateful benchmark for mobile planner agents covering 1,705 tasks and 212 realistic tools According to @HuggingPapers. The benchmark evaluates agents across four dimensions: tool use, memory, skills, and sub-agent collaboration.
Key Takeaways
- Alibaba released MobilePA-Bench, a stateful benchmark with 1,705 tasks and 212 tools for mobile planner agents.
- It evaluates tool use, memory, skills, and sub-agent collaboration, but lacks baseline scores.
Why statefulness matters
Most existing agent benchmarks — think ToolBench or API-Bank — evaluate single-turn tool calls against static API schemas. MobilePA-Bench's stateful design means the agent's actions mutate the environment, and subsequent decisions depend on prior outcomes. This forces the agent to maintain a working memory of what it has already done, rather than treating each call as an isolated query.
The 212-tool count is notable: it's closer to a real device's surface area than the handful of APIs typical in academic benchmarks. The sub-agent collaboration dimension addresses a specific architectural pattern — hierarchical planners that delegate subtasks — which most mobile agent evaluations ignore entirely.
What the benchmark does not disclose
The announcement does not include baseline scores, model rankings, or a comparison against prior mobile benchmarks like MobileAgent or AppAgent. Without those numbers, it's impossible to say whether current models struggle on the memory dimension or whether the 1,705 tasks are actually hard. The paper link in the tweet presumably contains details, but the public announcement is thin on results.
The four dimensions map to known failure modes in production mobile agents: tool selection errors, context loss across long trajectories, skill acquisition from demonstration, and coordination overhead in multi-agent setups. MobilePA-Bench appears designed to surface those specific failures rather than produce a single aggregate score.
What to watch
Watch for the associated paper's baseline results — specifically whether any model exceeds 50% task completion on the memory dimension, and whether Alibaba publishes per-dimension scores that isolate sub-agent collaboration overhead. Also track adoption: if MobilePA-Bench appears in the next round of agent leaderboards within 90 days, it's a serious contender.
