ResearchPublished ExplainedAll news from that day
How METR's time horizon rates AI agents by human task length
METR's 50% time horizon is the human task length at which an AI agent is predicted to succeed half the time; METR says measurements above 16 hours are unreliable with its current task suite.
Check our sources · 17 facts from 2 sources
Image: METR
Key points
- METR defines the 50% time horizon as the task duration, measured by human expert completion time, at which an AI agent is predicted to succeed half the time.
- METR says the horizon measures the difficulty of a task, not the time an AI spends on it, and 80% horizons are 4 to 6 times shorter than 50% horizons.
- METR's time-horizons page says measurements above 16 hours are unreliable with its current task suite.
What happened
METR, the group that publishes the time-horizon measure, says measurements above 16 hours are unreliable with its current task suite. That caveat sits on the page that reports how long a task AI agents can handle.
The task-completion time horizon is the task duration, measured by how long human experts take, at which an AI agent is predicted to succeed with a given level of reliability. The 50% time horizon is the duration at which the agent is predicted to succeed half the time. METR's page stresses that it measures the difficulty of a task, not how long the AI spends on it.
The human side comes first. For most tasks, METR contracts people to attempt the task and uses the geometric mean of their successful completion times as its length; for some tasks it relies instead on expert estimates or completion times from QA tests.
In the 2025 paper that introduced the measure, METR collected over 800 human baselines totalling 2,529 hours, from skilled professionals in software engineering, machine learning and cybersecurity with about five years of relevant experience.
The paper's 170 tasks came from three sets: HCAST, with 97 tasks from about one minute to 30 hours; RE-Bench, with 7 machine-learning research tasks of about 8 hours each; and SWAA, with 66 single-step tasks of 1 to 30 seconds. METR's current page describes a suite of over a hundred software, machine-learning and cybersecurity tasks, drawing on RE-Bench, HCAST and a set of shorter novel software tasks.
The AI side is a curve. For each model, METR fits a logistic curve that predicts the probability of success against task length, with length on a log scale. The 50% horizon is where the curve crosses one half. METR runs each task six times, which gives roughly 1,000 runs for the test set.
The paper's headline example is o3, with a 50% time horizon of around 110 minutes. The same method at an 80% success threshold gives horizons 4 to 6 times shorter, so a horizon is only meaningful with its reliability level beside it.
The paper also measured a trend. The paper says the frontier AI time horizon has doubled approximately every seven months since 2019; its regression gives 207 days, with a 95% bootstrapped confidence interval of 166 to 240 days. METR's page says that for models released between early 2019 and March 2025, an exponential trend is a much better fit to the data than both the linear and hyperbolic trends.
The limits are stated by METR. The paper says a time horizon is always measured relative to a domain and a task distribution, and that realistic tasks rarely have automatic scoring. It also says that, because the dataset is limited, it cannot confidently measure time horizons at very high success rates, such as 95%.
The page adds the 16-hour caveat, says it is no longer actively updated, and was last updated on 8 September 2026.
Underlined sentences link to the facts we checked. Dotted words open a short explainer.
Fact 6 of 17 · from METR
calculated using their performance on over a hundred diverse software tasks. ... Our tasks are drawn from RE-Bench, HCAST, and a set of shorter novel software tasks. These primarily consist of software engineering, machine learning, and cybersecurity tasks.
Fact 8 of 17 · from METR
In our March 2025 paper introducing time horizon, we investigated the quality of linear and hyperbolic fits for models released between early 2019 and March 2025. As Figure 17 from the paper shows, an exponential trend is a much better fit to the data than both the linear and hyperbolic trends.
What it means for you
Our viewA time horizon describes how hard a task is for people, so it does not tell you how long an agent will take to run it. A figure only means something with its success level beside it, because the gap between the 50% and 80% horizons is large.
METR also says its page is no longer actively updated, so the newest models may not be on it. When you see a claim that an agent can do tasks of a given length, check which success level it uses, and whether the length is inside the range METR says it can measure.
It adds no new facts.
Your reaction
We count reactions per story and day, never who reacted. The counts help us choose what goes in the monthly issue. If you are signed in, your own page shows yours too.
We count reactions per story and day, never who reacted. They help choose what goes in the monthly issue.
Check our sources
Every sentence above is checked against these 2 sources.
1 Task-completion time horizons of frontier AI models
Open the source archived copy-
METR's time-horizons page says: 'Measurements above 16 hrs are unreliable with our current task suite.'
citeMeasurements above 16 hrs are unreliable with our current task suite
-
METR defines the task-completion time horizon as the task duration, measured by human expert completion time, at which an AI agent is predicted to succeed with a given level of reliability.
citeThe task-completion time horizon is the task duration (measured by human expert completion time) at which an AI agent is predicted to succeed with a given level of reliability.
-
METR defines the 50% time horizon as the duration at which an agent is predicted to succeed half the time.
citeFor example, the 50%-time horizon is the duration at which an agent is predicted to succeed half the time.
-
METR's page says the time horizon is a measure of the difficulty of a task, rather than the time an AI spends to complete the task.
citeIt’s a measure of the difficulty of a task, rather than the time an AI spends to complete the task.
-
METR's page says that for the majority of its tasks, humans attempt the task and it uses the geometric mean of their successful completion times; for some tasks it relies instead on expert estimates or QA completion times.
Toned down to what the source says
citeFor the majority of our tasks, we estimate the duration it takes a human expert to complete the task by contracting humans to attempt the task and taking the geometric mean of their successful completion times. ... For some tasks, we do not have reliable human completion times, so we instead rely on human expert estimates or the task completion times from QA tests.
-
METR's time-horizons page describes its suite as over a hundred diverse software tasks drawn from RE-Bench, HCAST and a set of shorter novel software tasks, primarily software engineering, machine learning and cybersecurity.
citecalculated using their performance on over a hundred diverse software tasks. ... Our tasks are drawn from RE-Bench, HCAST, and a set of shorter novel software tasks. These primarily consist of software engineering, machine learning, and cybersecurity tasks.
-
METR's page says it runs 6 independent runs per task, roughly 1,000 runs for the test set.
citeWe launch 6 independent runs for each task ... In theory, it should only take ~1 day for these ~1,000 runs to complete
-
METR's page says that for models released between early 2019 and March 2025, an exponential trend is a much better fit to the data than both the linear and hyperbolic trends.
Toned down to what the source says
citeIn our March 2025 paper introducing time horizon, we investigated the quality of linear and hyperbolic fits for models released between early 2019 and March 2025. As Figure 17 from the paper shows, an exponential trend is a much better fit to the data than both the linear and hyperbolic trends.
-
METR's time-horizons page says it is no longer actively updated and was last updated on 8 September 2026.
citeThis page is no longer actively updated. LAST UPDATED September 8, 2026
2 Measuring AI Ability to Complete Long Tasks (arXiv 2503.14499v3)
Open the source archived copy-
The METR paper reports over 800 human baselines totalling 2,529 hours, from skilled professionals in software engineering, machine learning and cybersecurity with about 5 years of relevant experience (section 2.2).
citeIn total, we use over 800 baselines totaling 2,529 hours ... Our baseliners are skilled professionals in software engineering, machine learning, and cybersecurity, with the majority having attended world top-100 universities. They have an average of about 5 years of relevant experience
-
The paper's task suites total 170 tasks: HCAST (97 tasks from 46 families, about 1 minute to 30 hours), RE-Bench (7 machine-learning research engineering tasks of about 8 hours each) and SWAA (66 single-step tasks of 1 to 30 seconds) (section 2.1).
cite1. A subset of HCAST: 97 diverse software tasks ranging from 1 minute to 30 hours. ... 2. RE-Bench: 7 difficult ML research engineering tasks, all eight hours long. 3. Software atomic actions (SWAA): 66 single-step tasks representing short segments of work by software developers, ranging from 1 second to 30 seconds. ... We use 97 tasks from 46 task families in HCAST
-
The METR paper fits a logistic curve of success probability against the log of human task time (section 3.1); METR's page says it fits a logistic curve to predict the probability that a model completes tasks.
citep_success(agent, task) = sigma((log h_agent - log t_task) * beta_agent) where t_task is the geometric mean time of successful human baselines ... with h_agent representing the 50% time horizon. / page: For each agent, we fit a logistic curve to predict the probability it successfully completes tasks as a function of human task duration.
-
The METR paper reports that o3 has a 50% time horizon of around 110 minutes.
citeOn these tasks, current frontier AI models such as o3 have a 50% time horizon of around 110 minutes.
-
The METR paper reports that the 80% time horizons are 4 to 6 times shorter than the 50% horizons (section 3.2.1).
citeHowever, models’ 80% time horizons are 4-6x shorter, suggesting that even models that sometimes succeed on difficult and diverse tasks cannot reliably perform tasks of moderate length.
-
The METR paper says the frontier AI time horizon has doubled approximately every seven months since 2019; the regression gives 207 days with a 95% bootstrapped confidence interval of 166 to 240 days (section 3.2).
citefrontier AI time horizon has doubled approximately every seven months since 2019 ... finding that time horizon has doubled every 207 days with a 95% bootstrapped confidence interval 166-240 days
-
The METR paper's limitations section says time horizon is always measured relative to a domain and task distribution, and that realistic tasks rarely have automatic scoring (sections 5 and 6).
citewe caution that time horizon is always measured relative to a domain, task distribution, and human baseliners’ level of skill and context ... in short, realistic tasks rarely have automatic scoring
-
The METR paper says that, due to the limited dataset, it cannot confidently measure time horizons at very high success rates, such as 95%.
Toned down to what the source says
citeDue to the limited dataset, we cannot confidently measure time horizons at very high success rates (e.g. 95%)
Every sentence above is checked against these 2 sources. Each fact has its own address you can cite.
Source 1 · METR
Task-completion time horizons of frontier AI models
METR's time-horizons page says: 'Measurements above 16 hrs are unreliable with our current task suite.'
Measurements above 16 hrs are unreliable with our current task suite
METR defines the task-completion time horizon as the task duration, measured by human expert completion time, at which an AI agent is predicted to succeed with a given level of reliability.
The task-completion time horizon is the task duration (measured by human expert completion time) at which an AI agent is predicted to succeed with a given level of reliability.
METR defines the 50% time horizon as the duration at which an agent is predicted to succeed half the time.
For example, the 50%-time horizon is the duration at which an agent is predicted to succeed half the time.
Show all 9 factsShow fewer facts
METR's page says the time horizon is a measure of the difficulty of a task, rather than the time an AI spends to complete the task.
It’s a measure of the difficulty of a task, rather than the time an AI spends to complete the task.
METR's page says that for the majority of its tasks, humans attempt the task and it uses the geometric mean of their successful completion times; for some tasks it relies instead on expert estimates or QA completion times.
Toned down to what the source saysFor the majority of our tasks, we estimate the duration it takes a human expert to complete the task by contracting humans to attempt the task and taking the geometric mean of their successful completion times. ... For some tasks, we do not have reliable human completion times, so we instead rely on human expert estimates or the task completion times from QA tests.
METR's time-horizons page describes its suite as over a hundred diverse software tasks drawn from RE-Bench, HCAST and a set of shorter novel software tasks, primarily software engineering, machine learning and cybersecurity.
calculated using their performance on over a hundred diverse software tasks. ... Our tasks are drawn from RE-Bench, HCAST, and a set of shorter novel software tasks. These primarily consist of software engineering, machine learning, and cybersecurity tasks.
METR's page says it runs 6 independent runs per task, roughly 1,000 runs for the test set.
We launch 6 independent runs for each task ... In theory, it should only take ~1 day for these ~1,000 runs to complete
METR's page says that for models released between early 2019 and March 2025, an exponential trend is a much better fit to the data than both the linear and hyperbolic trends.
Toned down to what the source saysIn our March 2025 paper introducing time horizon, we investigated the quality of linear and hyperbolic fits for models released between early 2019 and March 2025. As Figure 17 from the paper shows, an exponential trend is a much better fit to the data than both the linear and hyperbolic trends.
METR's time-horizons page says it is no longer actively updated and was last updated on 8 September 2026.
This page is no longer actively updated. LAST UPDATED September 8, 2026
Source 2 · METR
Measuring AI Ability to Complete Long Tasks (arXiv 2503.14499v3)
The METR paper reports over 800 human baselines totalling 2,529 hours, from skilled professionals in software engineering, machine learning and cybersecurity with about 5 years of relevant experience (section 2.2).
In total, we use over 800 baselines totaling 2,529 hours ... Our baseliners are skilled professionals in software engineering, machine learning, and cybersecurity, with the majority having attended world top-100 universities. They have an average of about 5 years of relevant experience
The paper's task suites total 170 tasks: HCAST (97 tasks from 46 families, about 1 minute to 30 hours), RE-Bench (7 machine-learning research engineering tasks of about 8 hours each) and SWAA (66 single-step tasks of 1 to 30 seconds) (section 2.1).
1. A subset of HCAST: 97 diverse software tasks ranging from 1 minute to 30 hours. ... 2. RE-Bench: 7 difficult ML research engineering tasks, all eight hours long. 3. Software atomic actions (SWAA): 66 single-step tasks representing short segments of work by software developers, ranging from 1 second to 30 seconds. ... We use 97 tasks from 46 task families in HCAST
The METR paper fits a logistic curve of success probability against the log of human task time (section 3.1); METR's page says it fits a logistic curve to predict the probability that a model completes tasks.
p_success(agent, task) = sigma((log h_agent - log t_task) * beta_agent) where t_task is the geometric mean time of successful human baselines ... with h_agent representing the 50% time horizon. / page: For each agent, we fit a logistic curve to predict the probability it successfully completes tasks as a function of human task duration.
Show all 8 factsShow fewer facts
The METR paper reports that o3 has a 50% time horizon of around 110 minutes.
On these tasks, current frontier AI models such as o3 have a 50% time horizon of around 110 minutes.
The METR paper reports that the 80% time horizons are 4 to 6 times shorter than the 50% horizons (section 3.2.1).
However, models’ 80% time horizons are 4-6x shorter, suggesting that even models that sometimes succeed on difficult and diverse tasks cannot reliably perform tasks of moderate length.
The METR paper says the frontier AI time horizon has doubled approximately every seven months since 2019; the regression gives 207 days with a 95% bootstrapped confidence interval of 166 to 240 days (section 3.2).
frontier AI time horizon has doubled approximately every seven months since 2019 ... finding that time horizon has doubled every 207 days with a 95% bootstrapped confidence interval 166-240 days
The METR paper's limitations section says time horizon is always measured relative to a domain and task distribution, and that realistic tasks rarely have automatic scoring (sections 5 and 6).
we caution that time horizon is always measured relative to a domain, task distribution, and human baseliners’ level of skill and context ... in short, realistic tasks rarely have automatic scoring
The METR paper says that, due to the limited dataset, it cannot confidently measure time horizons at very high success rates, such as 95%.
Toned down to what the source saysDue to the limited dataset, we cannot confidently measure time horizons at very high success rates (e.g. 95%)
Topics
The morning email
On the mornings we publish: the three top stories and up to four short ones. Free.
We email you a link to confirm. An issue may include one sponsor, always labelled Sponsored · Advertisement. Our emails count opens and clicks, not who made them. Unsubscribe in one click. What we keep