Auditing Demonstration Curation Metrics: Action-Only Scorers Fail on the Structural Defects That Degrade Imitation Policies

Researchers have found that curation metrics for imitation learning policies have limitations in detecting demonstration defects, with some metrics failing to identify structural errors that can degrade policy performance.

Imitation-learning policies inherit the quality of the demonstrations they are trained on, and curation metrics aim to score and filter low-quality demonstrations automatically^[1]. Seven curation metrics were tested for their ability to identify demonstration defects. The study found that action-only metrics are unable to detect structural errors, where a demonstration executes a wrong action at a key moment. Only metrics that examine the state trajectory can detect such errors^[1]. Furthermore, the study discovered that the metric with the highest defect-detection AUROC produces the worst curated policy^[2]. In contrast, a metric with a substantially lower AUROC produces a policy that nearly matches the oracle trained on ground-truth clean data^[2]. Five of the seven metrics exploit episode length as a trivial proxy for the defect label^[2]. The contaminated baseline succeeds on only a small percentage of rollouts, while the two best curation methods close the gap to within a small percentage point gap of the oracle ceiling^[2].