About a month ago I started LiveNerf, an open-source project to independently measure whether AI models actually get worse after they're released.
Every time a new model comes out, people seem to agree that it's incredible for the first week or two. Then the complaints start. People think companies might be quietly reducing inference costs, lowering reasoning effort, or changing the model being served. Other people argue that nothing has changed and users are just noticing limitations they hadn't noticed before.
I wanted to see whether we could actually measure it.
We've now collected 34 consecutive days of Opus 5.5 measurements, including two completed 10-day comparison windows against a 10-day baseline. So far, we haven't found statistically significant evidence of degradation.
How LiveNerf works
We started collecting data on September 24, about two and a half days after Opus 5.5 launched. We screened 2,336 questions from GPQA Diamond, MMLU-Pro, competition mathematics, and AIME 2025–26, sampling each four times.
Opus 5.5 got about 93% right on the first attempt. Roughly 97% of questions were either always right or always wrong during screening. Neither group is especially useful for measuring small changes. We wanted questions that the model sometimes solved and sometimes missed.
That left us with a frozen panel of 78 questions. Each day we run those same questions, plus a 12-question control arm, through a pinned Claude Code CLI with fixed prompts, grading, and harness settings. Raw logs and deviations are public.
The first 30 days
Our launch-period baseline was 58.4% accuracy. The first comparison window scored 59.2%, and the second scored 57.1%. Neither met the pre-registered threshold for declaring a change.
| Window | Accuracy | Change ± SE | Median output tokens |
|---|---|---|---|
| Days 1–10 | 58.4% | Baseline | 588 |
| Days 11–20 | 59.2% | +0.7 ± 1.8 pp | 606 |
| Days 21–30 | 57.1% | −1.5 ± 1.8 pp | 594 |
There are noticeable day-to-day swings, but they don't amount to a sustained regression. The realized minimum detectable effect is about 6.6 percentage points per 10-day window. A smaller change could be real and still escape detection.
So the conclusion is fairly narrow: we haven't detected a sustained change in accuracy on this particular panel. That doesn't establish that nothing about the model or the product has changed.
Effort changes are easier to see in tokens
One of the more interesting findings came from our validation experiments. We deliberately reduced reasoning effort to see whether the benchmark could pick up a known change.
| Effort vs high | Accuracy change | Output tokens |
|---|---|---|
| Medium | −4.2 ± 3.9 pp | −26% |
| Low | −8.3 ± 4.5 pp | −62% |
The accuracy estimates are noisy, but the token reductions are substantial. If a provider quietly reduced reasoning effort, output-token distributions might reveal it before benchmark accuracy does. In our actual daily measurements, median output tokens have stayed relatively stable.
Could we detect a model swap?
We also tried substituting Opus 5 for Opus 5.5. The measured accuracy difference was −3.8 ± 6.3 percentage points, with about 23% fewer output tokens. At our preregistered 99% confidence threshold, we couldn't reliably distinguish the two models on accuracy in that validation.
That's a limitation worth being explicit about. If an instrument can't reliably detect a particular same-family substitution, it shouldn't be used to claim that substitution hasn't happened. Our daily windows have more samples, but we haven't demonstrated that they're sufficient to detect a change of that size either.
The benchmark questions have problems too
During our item audit, we found eight answer keys that looked incorrect and around 30 ambiguous questions among the 80 reviewed, including two subsequently excluded items.
It's a surprisingly large fraction, although it makes some sense when you consider how we selected the panel. We deliberately chose questions the model sometimes gets right and sometimes gets wrong. Some are genuinely difficult. Others are difficult to grade consistently.
We didn't quietly remove those questions after collecting data. We documented the issues and kept the original panel frozen, with a sensitivity analysis for checking their effect. Otherwise we'd risk changing the measurement halfway through the experiment.
What the results do and don't say
Based on the evidence we've collected so far, I don't think there's good support from LiveNerf for the claim that Opus 5.5 has been degraded since launch.
That doesn't mean people reporting problems are necessarily imagining them. The model is only one part of the product. System prompts, context management, tool behavior, and inference settings could change someone's experience without producing an obvious change on our benchmark. And our panel doesn't directly measure coding agents, long conversations, or every other capability people depend on.
It's also possible that some perceived regressions are just ordinary variation. That's the whole reason I wanted to collect a baseline instead of relying on anecdotes.
Where it goes from here
We originally planned to run for 30 days, but have extended the series to 45. As of October 27, we've collected 34 consecutive days without missing a scheduled day. The third comparison window is still in progress.
I'd like to expand LiveNerf beyond a single model and a single kind of benchmark, especially toward more specific capabilities. Aggregate accuracy can remain stable while something important to users changes.
More broadly, it seems strange that we have so much infrastructure for comparing models when they launch, but relatively little independent infrastructure for verifying that they continue performing the same way afterward.
I didn't build LiveNerf because I was certain AI companies were nerfing their models. I built it because I thought the question was worth investigating. So far, the results haven't supported that hypothesis for Opus 5.5 within the limits of what we're measuring.
We'll keep measuring.
Project, raw results, and methodology: github.com/ninjahawk/livenerf. I'd be interested in criticism of the statistical design or suggestions for better longitudinal evaluations.