ByteDance's SemComp-Bench reports 37.8% best open-access task completion against 91.8% general reliability. The 54-point gap exposes a structural weakness in video generation models: they look good, but fail at the actual job.
Key facts
- SemComp-Bench: ByteDance video task completion benchmark
- Best open-access OA: 37.8%
- Best open-access GR: 91.8%
- Gap: 54 percentage points between GR and OA
- Source: ByteDance research, via @HuggingPapers
Video generation models have been graded on aesthetics, motion quality, and text-video alignment. ByteDance's SemComp-Bench shifts the question to something harder: did the model actually finish the task shown in the prompt? According to @HuggingPapers, the benchmark's top open-access model scores 37.8% overall accuracy (OA) while hitting 91.8% general reliability (GR). That 54-point delta is the story, not the absolute numbers.
What SemComp-Bench actually measures
SemComp-Bench appears to be the first benchmark to explicitly separate general video quality from task-specific completion. GR captures whether output looks like a plausible video; OA captures whether the model executed the semantic steps required to complete the task. The divergence suggests current models optimize for visual plausibility at the expense of functional correctness—a known failure mode in text-to-video systems that produce beautiful but semantically wrong output.
The benchmark's design implies a hierarchy of evaluation: first, can the model generate coherent video (GR); second, can it complete the task (OA). The gap between them quantifies how much of current capability is surface-level. The source tweet does not specify the number of tasks, video length, or evaluation protocol, so direct comparison to prior benchmarks like VBench or EvalCrafter is not yet possible.
Why the gap matters more than the headline
The 37.8% OA figure is not a failure of one model class—it is a structural observation about the entire video generation paradigm. Models trained on next-token prediction or diffusion objectives learn statistical patterns, not causal task models. They can imitate task trajectories seen in training data but cannot generalize to novel task compositions. This is the same limitation that plagued early language models on multi-step reasoning before chain-of-thought and tool-use training.
The GR/OA split suggests a path forward: if OA becomes the optimization target rather than a byproduct, training objectives will need to reward task completion explicitly. This could mean reinforcement learning with task-level reward models, or architecture changes that separate planning from rendering. The benchmark's existence signals ByteDance is moving evaluation toward functional outcomes, a direction that may pressure other labs to follow. [The source tweet provides the benchmark's headline numbers but does not disclose the evaluation set size, model list, or prompting strategy].
What to watch
Watch for the SemComp-Bench paper or technical report with full evaluation details: task count, video length, and model-specific breakdowns. If ByteDance releases a model trained specifically to optimize OA, the delta between GR and OA will be the metric to track—a narrowing gap would signal genuine task-level progress.
