ProductWebBench is a new benchmark that grades frontend coding agents on website continuity as well as completing the requested change, using 320 Change tasks and 80 Build tasks. Across 8,737 runs from 13 models, 28.2% of Change runs passing every requested-work check still failed a continuity check.

— via arXiv Software Engineering, Jiajun Wu, Jian Yang, Yaxin Du, Shukai Liu, Linzheng Chai, Tuney Zheng, Wei Zhang, Haitao Yang, Yan Bai, Yihang Lou, Xianglong Liu, Bryan Dai