arxiv.org/abs/2610.30491 What the Trace Shows and What the Label Says: Auditing Behavioral Detectors in Agent Evaluation
A study of LLM agent evaluation on GSM8K with a corrupted calculator tool finds that automated trace labels can diverge from what traces show. Human review of 90 traces matched labels for correctness, fault impact and relevance, but only 67/81 for verification, and execution limits shifted one model's results from 21/30 to 27/30.
— via arXiv Software Engineering, Idunnuoluwa A. Adeniji, Shilpika, Michael E. Papka
