Checked fact 16828 Sep 2026Research

The benchmark draws on 26 fully functional reference web applications, from which the researchers extracted 1,975 replay-verified behaviors and built 4,063 tasks.

By ai notis. An AI newsroom with a human editor. Every fact links to its source. How we work

The exact words it rests on

26 applications ... 1,975 replay-verified behaviors ... 4,063 tasks without human intervention

What the source said when we opened it, on 28 Sep 2026.

The source

ProgramDistill: benchmarking coding agents on feature discovery in real applications
arXiv · 2026-09-16

Open the source archived copy

Checked

Checked by the notis newsroom on 28 Sep 2026, against the source above.

In the story

ProgramDistill, a benchmark built from 26 web apps, scores GPT-6 Astra at 49.2 percent and Claude Opus 5 at 28.8. 28 Sep 2026

Cite this fact

Short permanent address

In a sentence

“The benchmark draws on 26 fully functional reference web applications, from which the researchers extracted 1,975 replay-verified behaviors and built 4,063 tasks.” ai notis, checked 28 Sep 2026. Source: ProgramDistill: benchmarking coding agents on feature discovery in real applications (arXiv). https://ainotis.com/c/168

As a reference (APA style)

ai notis. (2026, September 28). The benchmark draws on 26 fully functional reference web applications, from which the researchers extracted 1,975 replay-verified behaviors and built 4,063 tasks [Checked claim 168]. ai notis. https://ainotis.com/c/168

Anyone may quote this address. It does not change; if we correct the story, this page says so.