Find the failures the simulator misses.

SCOUT spends a small target-system budget where a cheap proxy is informative, and uses support-aware mutual information so those labels cover more than one failure mode.

Anjali Parashar, Rachel Luo, Apoorva Sharma, Sushant Veer, Edward Schmerling, Carson Sobolewski, Mingxin Yu, Chuchu Fan, Marco Pavone · MIT LIDS and NVIDIA Research

Try the rule

Failure is the missed-car score on real KITTI: one minus the average overlap between the detector’s boxes and the ground-truth cars. A frame counts as severe when that score is at least 0.45. Proxy only ranks frames by the simulator surrogate. Real only spends the same labels without that surrogate. The map groups frames into islands and colors each island by how severe the frames there are. Marks are the frames that method has labeled, so coverage shows up inside the hotter islands.

Ours
Proxy only
Real only
MI only
Random

real labels 0 / 85

Loading the KITTI acquisition…

0real frames labeled

0severe failures, ours

0severe failures, proxy only

0severe failures, real only

Fifteen clips step through all 85 real labels. The pictures are frames our method labeled.

1. Shortlist

Mutual information proposes frames the labels already collected do not cover. Each method then spends the same number of real KITTI evaluations.

2. Correct the proxy

On the shortlist only, a local control variate estimates the target score from the surrogate and the free simulator. The proxy is a covariate, not the answer.

3. Spend the real label

Each method labels 85 real KITTI frames from the same 10-frame start. The pictures skip ahead through that run, about 15 clips in all.

Supplementary video

Quadruped velocity-tracking on a Unitree Go2. The commands in the video are ones the method predicted as failures or as a non-failure.

The supplementary file docs/assets/video.mp4 is not in this checkout yet.

Hardware demos from the paper: two failure commands and one non-failure, selected from the learned failure predictor.

Paired SIMPLER rollouts

Same grasp, two visual simulators. The gap case is a success in the proxy and a miss on the target.

Proxy and target both grasp the can — Both succeed

Proxy and target both miss the can — Both fail

Proxy grasps the can and the target misses — Proxy succeeds, target fails

Results

Same target budget for every method. Curves are mean ± std from the paper runs in the research repository, including nuPlan.

nuPlan cumulative severity, coverage, and positive samples — Closed-loop driving. Proxy is open-loop playback. Failure is time-to-collision. Budget 170 after 20 initial labels, batches of 5. Ours leads coverage and the fraction of failures.

Positive-sample rate on nuPlan, mean ± std over four seeds.
Threshold γ	Random	BNN-C	BAMS	BNN-CV	Ours
0.30	0.23 ± 0.01	0.25 ± 0.01	0.28 ± 0.01	0.27 ± 0.01	0.33 ± 0.01
0.20	0.22 ± 0.00	0.24 ± 0.02	0.27 ± 0.01	0.26 ± 0.01	0.32 ± 0.01
0.05	0.21 ± 0.01	0.21 ± 0.02	0.25 ± 0.01	0.22 ± 0.02	0.29 ± 0.01

t-SNE of nuPlan scenarios colored by failure and method — t-SNE of the 384-dimensional scenario embedding. Support-aware mutual information keeps new labels off the regions that are already covered, whatever the initial sample looked like.

KITTI cumulative failure, regret, and top-10 percent recall — Real KITTI missed-car rate against the Virtual KITTI clone of the same frame. Keeping the proxy term (β) raises the cumulative failure of the labels that are actually collected and recalls more of the worst frames.

KITTI component ablation from the paper.
	Full	Greedy	β = 0 (n+k)	β = 0	MI	−MI−C	−MI−G
Utility	0.38	0.38	0.36	0.36	0.32	0.40	0.41
P.S. at 0.5	0.24	0.22	0.14	0.17	0.11	0.30	0.32
P.S. at 0.7	0.03	0.01	0.02	0.01	0.02	0.02	0.01
Coverage	0.78	0.58	0.79	0.72	0.91	0.60	0.48

Quadruped cumulative tracking error, coverage, and positive samples — Unitree Go2 velocity tracking. Proxy is a MuJoCo RL policy. Target budget 30 after 10 initial commands, batches of 5. Failure is mean tracking error above 0.7.

Quadruped commands found by our method and by BAMS — Commands in (v_y, ω_z). BAMS concentrates near the threshold. The learned score has two symmetric failure modes, matching the task.

Positive-sample rate by severity. BAMS finds none of the failures at γ = 1.0.
γ	Ours	BAMS
0.7	0.6 ± 0.1	0.5 ± 0.1
0.8	0.3 ± 0.1	0.3 ± 0.2
1.0	0.2 ± 0.1	0.0 ± 0.0