ARC is the independent benchmark VLA labs run before they ship to customers. We measure the cognitive brain — System 2, after RoboBench (PKU/BAAI) — and the motor policy — System 1, after VLABench (ICCV ′25) — on the same physical rollout, then fuzz the result with the ten perturbation operators of VLATest (FSE ′25). Every score is signed, replayable, and tied to a hardware-of-record you can audit. No simulators. No cherry-picked episodes. One number per model, defensible on a customer call.
We grade the cognitive brain (System 2) and the motor policy (System 1) on the same rollout — RoboBench-style symbolic fidelity above VLABench-style execution success.
/ Dual-system evaluationEvery rollout perturbs the environment along the ten VLATest operators — confounding objects, lighting, camera pose, paraphrased instructions, OOD substitutions. Memorization fails by design.
/ Robustness primaryPlans are rolled out through an MLLM-simulated world with human-annotated state DAGs — Pearson-validated against 20 expert annotators. No BLEU, no LLM-pairwise theatre.
/ Statistical fidelityTwo calibrated physical benches. Hardware ref, scoring rubric, rollout videos, failure-DAG annotations are all public. Any lab can replicate the suite. CC-BY-SA.
/ ReproducibilityModels are re-evaluated on every checkpoint. The leaderboard reflects current capability — not the cherry-picked snapshot you submitted six months ago.
/ Live evaluationEach run emits a typed failure trace — execution / identification / common-sense / mode-specific (per RoboBench §5.3). Auditable, regression-comparable, deployment-grade.
/ Failure taxonomyRetrieve a specified item from a densely populated shelf containing visually similar distractors. Adversarial confounders, paraphrased instructions, drifting lighting between rollouts.
Receive a paraphrased natural-language instruction (5 mutations per rollout, mixed languages) and fold a deformable object accordingly. Probes language grounding and instruction robustness.
Plan and execute a 9-step assembly: open dispensers, portion, plate, garnish, serve. Composite long-horizon — maps directly to VLABench Track-6 (where SR for π0-Fast peaks at 1.6%).
Pour granular media into a vessel while an adversarial human nudges the workspace. Online replanning, force-aware control, recovery from contact failure.
Manipulate fragile transparent items absent from training distribution. Camera-pose, lighting, and object-mesh perturbations sweep the VLATest fuzzing operators.
Two arms — one human, one robotic — collaborate to pass and assemble pieces. Intent inference, timing, dynamic affordance, safety envelope under shared workspace.
Navigate an unmapped office to fetch an object specified only by referring expression ("the blue mug Maria left near the window"). Spatial language + memory-driven exploration.
Intended tool is missing. Reach the goal using a non-canonical substitute (a ruler in place of a spatula). Tests creative reuse and physical-law reasoning.
An MLLM-as-Embodied-Brain Evaluation Benchmark
System-2 cognition. 5 dimensions, 14 capabilities, 25 tasks, 6,092 QA pairs. Plans rolled out through an MLLM-simulated world with human-annotated state DAGs.
▍Frontier MLLMs trail human cognition by 10–35 points across embodied dimensions.
Long-Horizon Language-Conditioned Manipulation
System-1 execution. 100 task categories (60 primitive + 40 composite), 2,000+ 3D objects, MuJoCo. Six tracks measuring skill, generalization, semantic adaptation, long-horizon composition.
▍State-of-the-art VLAs score in the single digits on real generalization.
Testing & Evaluating Vision-Language-Action Models
Robustness fuzzing. 18,604 generated scenes, 78,604 rollouts, 580+ GPU hours, ten testing operators across object configuration, lighting, camera pose, instruction phrasing.
▍Pretrained VLAs lack the robustness necessary for practical deployment.
Internal benches drift. Customers don't trust your numbers. ARC publishes a signed score, on hardware you don't own, that your sales team can put in a deck the next morning.
Pick a candidate model. We run it on your reference workcell SKU under our perturbation envelope and hand back a defensible procurement-grade comparison — not a vendor pitch deck.
Every episode is logged at 30 Hz, signed, and replayable from raw observation. Safety envelopes (force, reach, exclusion zones) are scored as first-class metrics — not an afterthought.
State-of-the-art VLAs score in the single digits on
generalized long-horizon manipulation. ICCV said it.
FSE said it. We just keep score.
VLABench Track-6 SR — π0-Fast: 1.6 / OpenVLA: 0.4 / Octo: 0.0