Large language models, AI coding assistants, agents, and their use in software development. Share substantial technical work, research, tools, and thoughtful discussion. LLM-focused stories belong here; use another Programming forum when a different engineering topic is clearly the main focus.
Anthropic launched OSS Scanner, a free opt-in service that sends maintainers of critical open-source projects security reports generated by its models without human review. Anthropic says it produced over 29,000 candidate vulnerabilities, and maintainers such as wolfSSL and curl have confirmed valid findings, including a curl CVE, though some severity ratings were inflated.
— via InfoQ, Matt Saunders
Researchers show that self-evolving agent harnesses can become unsafe when individually safe updates to memory, prompts, skills and tools interact. They found 43 pairwise and 18 irreducible 3-way failures and propose a typed hypergraph with runtime monitoring that lowers checking costs.
— via arXiv Software Engineering, Zhixiang Zhang, Zesen Liu, Wai Ip Lai, Hongxu chen, Dongdong She
Researchers ran a small FunSearch-style loop with a local 30B model on nine construction problems, testing three operator packages in a full factorial design. The combined operators closed more of the gap to the record, repulsion raised diversity, and a frontier model reached in tens of samples what the local model did not in hundreds; searches plateaued at 85-92% gap closure.
— via arXiv Software Engineering, Roberto I. Ono Filho
Researchers extend the 270M FunctionGemma model to broader Android device control using a synthetic dataset of about 9,500 conversations across fifteen categories. Fine-tuning raises accuracy on the new set from 29.3% to 76.5%, while a combined model scores 82.3% on Google's set, 8.0 points below Google's specialist.
— via arXiv Software Engineering, Ali Rezagholizadeh, Soheila Samiee
Code Comprehensor is a framework that combines a code dependency call graph, LLM-generated hierarchical documentation, and a CLIP-style dual-encoder to retrieve relevant code context for an LLM orchestrator. Tests on OpenCV and a proprietary 5M-LOC repository showed improved text-to-code retrieval and above 90% faithfulness on RAGAS.
— via arXiv Software Engineering, Sujoy Roychowdhury, Joseph H. R. Isaac, Jiangning Gao, Martin Klitte, Diarmuid Corcoran
APO-DD adapts GEPA to automatically optimize LLM prompts for dialogue disentanglement in multi-party chat, tuning both task instructions and chat formatting functions. With Qwen3.5-9B on Ubuntu IRC it averaged 68.7 across seven metrics versus 57.5 for the best manual prompt.
— via arXiv Software Engineering, Naoki Takada, Tomohiro Okatsu, Yin Min Pa Pa, Katsunari Yoshioka, Tatsunori Mori
The paper proposes Adaptive Skill Granularity, which lets coding agents choose at runtime between primitive tool calls, macro skills, or hybrid execution with fallback to micro-level recovery. Evaluations indicate fixed macro execution is brittle under mismatch, while runtime routing and macro-to-micro recovery improve robustness.
— via arXiv Software Engineering, Dingxuan Zhang, Yanhui Jiang, Jun Wang, Hefeng Zhou
The study examines faults in LLM agent long-term memory that change which stored records reach the model's context, even though content and tasks stay fixed. Such corruption lowers LongMemEval correctness by 40.54 points and Memora FAMA by 30.60 points, with recovery observed after enforcement. The authors argue memory control-state integrity should be evaluated alongside retrieval accuracy.
— via arXiv Software Engineering, Bowen Liu, Youting Wang, Bonan Shen, Dingyan Shang
MarkTune is a fine-tuning framework that embeds watermarks directly into language model weights, using GaussMark's detection statistic as a reward while regularizing toward high-quality text or a stronger oracle model. It improves the quality-detectability trade-off, approaches inference-time schemes' detectability, and stays robust under the evaluated edits and open-weight attacks.
— via arXiv Cryptography Security, Yizhou Zhao, Zhiwei Steven Wu, Adam Block
A controlled study on ARC-AGI-3 tested whether frontier models can build and revise their own task harness. Across 27 lineages, performance ranged from zero to near-ceiling, and two revisions raised held-out RHAE by only 2.80 points, with development gains overstating generalization.
— via arXiv Software Engineering, Jens Lehmann, Sahar Vahdati
A study measures the benign-traffic cost of four memory-poisoning defenses for LLM agents, using half of LoCoMo with three repeated runs. The write-time defenses showed no detectable accuracy cost, while the read-time reranker withheld memory from 45% of questions and lowered core accuracy by 4.4 points, a borderline effect.
— via arXiv Cryptography Security, Pritom Bhowmik
ARATest is a property-based testing library that turns prose research claims into executable property checks with three verdicts: holds, violated, or inconclusive. Of 243 claims in 33 artifacts, 54 were faithfully declared; 32 held, eight were violated, and 14 were inconclusive, and the tool caught all 553 seeded faults.
— via arXiv Software Engineering, Yifeng He, Jiachen Liu, Hao Chen
A study of LLM agent evaluation on GSM8K with a corrupted calculator tool finds that automated trace labels can diverge from what traces show. Human review of 90 traces matched labels for correctness, fault impact and relevance, but only 67/81 for verification, and execution limits shifted one model's results from 21/30 to 27/30.
— via arXiv Software Engineering, Idunnuoluwa A. Adeniji, Shilpika, Michael E. Papka
An exploratory study tested nine prompt styles on GPT-5.5 across three IOI accuracy tasks and two NP-hard speed tasks. No single prompt won; the best depended on task type and mean versus median, and outlier runs skewed averages. Against human IOI contestants, the AI beat the average on one task and trailed on two.
— via arXiv Software Engineering, Saad B. Alarifi, Mohammed A. Alam
Researchers built six new safety datasets and trained two probe architectures on 20 base models from five families, finding that probe error follows two simple scaling laws tied to model size and label count. Fitted on the smallest models, the laws predicted the next larger model's error within 0.02 in 72% of cases.
— via arXiv Cryptography Security, Yuchen Wu (University of Washington), Jerry Wei (Anthropic)
SESAA is a framework where an LLM agent begins with no rules and iteratively evolves readable joern queries for static analysis, guided only by a second LLM critic without ground-truth labels. It reaches a mean F1 of 0.742 on the OWASP Benchmark for Python and detected 10 of 19 later CVEs in a pilot.
— via arXiv Software Engineering, Rui Melo, Andre Duarte, Aditya Oke, Shubham Gandhi, Nachiket Kotalwar, Rui Abreu, Lei Li, Carolyn Rose
The AgentPhishing benchmark tests whether persuasive phishing emails, optimized against individual agents via a method called PhishOpt, can fool LLM email agents. Across nine agents, four detected under 10% of phishing and complied with over 45% of malicious requests.
— via arXiv Cryptography Security, Justin Tan, Miao Li, Ding Zhao
Researchers introduce the OpenDS task and OpenDS-World, a benchmark of 200 tasks for testing how well agents carry out multi-step deep search on the live Web. Among 12 frontier models evaluated, the top performer reached 39.0% accuracy, while humans scored 21.1% across 1,000 attempts.
— via arXiv Software Engineering, Xiaomin Yu, Hongliang Chen, Zhe Huang, Shilin Yan, Shao-Rong Su, Xuqiao Ma, Feiyang Wang, Jiayu Chen, Jiwei Fang, Yu Yan, Yini Huang, Chen Chen, Ruoxi Cheng, Yuzhi He, Yuhui Zhang, Xiaoxing Hu, Chen Liu, Jianbo Lin, Yijiang Li, Xiaobin Hu, Menglong Li, Zhuoran Lu, Weishi Wang, Zhongqi Yue, Chengwei Qin
The paper audits the hand-written hard-deny sets and routing region maps behind learned access control on five public ABAC Lab policies. Exhaustive checks show deny clauses transcribing explicit restrictions are unsound at rates of 0.50-0.70, which would withhold 16-78% of granted access if shipped unchecked.
— via arXiv Cryptography Security, Arsalan Jalil Sarghaleh, Mehdi Hazrati
VERA is an open-source, self-hosted dashboard used at a large European bank that runs public benchmarks on LLMs and scores the twelve measurable EU AI Act requirements from COMPL-AI, also tracking evaluation energy use. In a within-subject study, reviewers needed 30 seconds on average versus 56 seconds with raw files, and rated usefulness 4.36 out of 5.
— via arXiv Software Engineering, Robin Qu'eriaux, Solveig Bachellery, L'ea D'el'eris, Saurabhsingh Rajput, Mariam Barry, Tushar Sharma, J'er^ome Lebecq
