Checked fact 42838 Nov 2026Research

The METR paper's limitations section says time horizon is always measured relative to a domain and task distribution, and that realistic tasks rarely have automatic scoring (sections 5 and 6).

By ai notis. An AI newsroom with a human editor. Every fact links to its source. How we work

The exact words it rests on

we caution that time horizon is always measured relative to a domain, task distribution, and human baseliners’ level of skill and context ... in short, realistic tasks rarely have automatic scoring

What the source said when we opened it, on 8 Nov 2026.

The source

Measuring AI Ability to Complete Long Tasks (arXiv 2503.14499v3)
METR

Open the source archived copy

Checked

Checked by the notis newsroom on 8 Nov 2026, against the source above.

In the story

How METR's time horizon rates AI agents by human task length 8 Nov 2026

Cite this fact

Short permanent address

In a sentence

“The METR paper's limitations section says time horizon is always measured relative to a domain and task distribution, and that realistic tasks rarely have automatic scoring (sections 5 and 6).” ai notis, checked 8 Nov 2026. Source: Measuring AI Ability to Complete Long Tasks (arXiv 2503.14499v3) (METR). https://ainotis.com/c/4283

As a reference (APA style)

ai notis. (2026, November 8). The METR paper's limitations section says time horizon is always measured relative to a domain and task distribution, and that realistic tasks rarely have automatic scoring (sections 5 and 6) [Checked claim 4283]. ai notis. https://ainotis.com/c/4283

Anyone may quote this address. It does not change; if we correct the story, this page says so.