How to Detect Claude Code Quality Regressions in Your Logs

When developers suspect a coding model has gotten worse, the conversation usually stalls without evidence. Someone says the model feels lazier, someone else says their sessions are fine, the vendor says nothing changed, and the thread dies without anyone producing evidence either side could check. Issue #42796 on the claude-code repo, filed in early April by the user stellaraccident, data-mines 6,852 Claude Code session files, 17,871 thinking blocks, and 234,760 tool calls spanning January 30 through April 1. It turns "the model feels worse" into a set of measured behavioral shifts with dates attached. Whether or not you buy every inference in it, the methodology provides a working template for detecting model quality regressions from session logs and makes a strong empirical case that extended thinking depth drives the other changes.

What the issue actually claims

The report, produced by having Claude analyze months of session JSONL data from four systems-programming projects, ties a quality regression to two overlapping events. First, estimated thinking depth started falling in mid-February: median thinking content dropped from roughly 2,200 characters in the January 30 to February 8 baseline to around 720 by late February, a 67% reduction, before any visibility change happened. Second, a thinking-redaction rollout, identified by a redact-thinking-2026-02-12 header, made thinking content invisible in a staged deployment through early March: 1.5% of blocks redacted on March 5, 24.7% on March 7, 58.4% on March 8, and effectively 100% from March 12 onward.

The date the team independently reported a quality regression was March 8, the same day redaction crossed 50%. The data still show behavioral changes.

Estimating thinking depth after the content disappears

Once thinking blocks are redacted, the report found you can still estimate how long they were. The signature field on thinking blocks correlates with thinking content length at 0.971 Pearson, measured across 7,146 paired samples where both were present. That correlation, established during the visible period, becomes a proxy afterward. Using it, the report estimates median thinking depth at roughly 600 characters in the fully redacted period, down about 73% from the January baseline, and shows the decline began weeks before redaction started.

For LLM observability efforts, a correlated field can preserve a signal after a provider removes the direct one. A streaming SSE proxy confirmed the direct signal was gone, with zero thinking_delta events in current API responses, leaving the signature proxy as the remaining window into reasoning depth.

The behavioral fingerprint of shallow thinking

The issue also computes behavioral metrics independently from tool-call logs, and they shifted in lockstep with the estimated thinking decline. These are metrics anyone running Claude Code at scale can compute from ~/.claude/projects/ today.

These metrics point to the same underlying change. In the good period, the logs show a research-first workflow: read the target file, related files, usages, headers, and tests before making a precise edit. In the degraded period, the model reads the immediate file and edits, and one in three edits landed on a file the model had not read in its recent tool history. The report catalogs familiar consequences: edits that violate file-level conventions, duplicated logic already present elsewhere in the file, and new declarations spliced between a documentation comment and the function it documents because the model did not know where the comment block ended.

The doubling of full-file Write usage supports the same conclusion. Rewriting a whole file requires less reasoning than a surgical edit, and a model short on thinking budget gravitates toward actions that require less reasoning to justify. The report tracks the word "simplest" in model output as a proxy for that mentality and documents a two-hour window where the model used it six times while producing code its own later self-corrections described as "lazy and wrong," "rushed," and "sloppy."

Users also see self-corrections like "oh wait" and "let me reconsider" surfacing in output rather than being resolved internally. That rate more than tripled, reaching 26.6 per thousand tool calls by the late period, with worst-case sessions producing more than 20 reversals in a single response.

Hooks as canary metrics

The team had built a stop hook, stop-phrase-guard.sh, to programmatically catch ownership-dodging, premature stopping, and permission-seeking behavior. Before March 8 it fired zero times. After March 8 it fired 173 times in 17 days, about ten per day.

The hook provided a machine-readable, timestamped signal when behavior changed. Most teams write hooks to enforce policy; this one also measured quality regressions. The report also tracked frustration indicators in user prompts, up 68% from 5.8% to 9.8%, and the frequency of corrections needed for ownership-dodging, up 117%. Prompts per session fell 22%, likely because users abandon sessions faster when the model stops being trustworthy.

Teams need to instrument before they need the baseline. The metrics that mattered here are derivable from session logs: read-to-edit ratio, edits-without-prior-read rate, Write-versus-Edit share, self-correction phrases per thousand tool calls, and hook violation counts. None require access to anything the vendor controls. The issue is credible because the team had months of clean baseline data when the shift happened.

Why extended thinking is load-bearing

The report argues that thinking tokens are structurally required for complex, long-session engineering work, and its proposed mechanism fits the data. Extended thinking lets the model plan which files to read and in what order, recall project-specific conventions, catch mistakes before emitting them, and decide whether to keep working or stop. Compress that budget and the model takes actions that require less reasoning: editing without reading, stopping without finishing, and taking superficial workarounds instead of fixing the code generator.

This team ran 50+ concurrent agent sessions doing systems programming in C, MLIR, and GPU drivers, with 30-plus-minute autonomous runs, and merged 191,000 lines across two PRs in one weekend during the good period. At that intensity, a shift from 6.6 reads per edit to 2.0 compounds across hundreds of tool calls per session. A casual user asking for a script tweak might never notice the same regression that makes this workflow, in the filer's words, unusable.

What the report asks for, and why it generalizes

The issue closes with four requests to Anthropic: transparency about thinking allocation when it changes, a paid tier with guaranteed deep thinking for users who need it, a thinking_tokens field in API usage responses when content stays redacted, and vendor-side monitoring of canary metrics like hook violation rates as leading indicators of regressions. The issue itself has since been closed on GitHub.

The requests apply beyond this dispute and beyond Anthropic. Hosted model providers ship silent server-side changes into workflows where reasoning depth determines whether a PR is mergeable, and users currently discover those changes without evidence. Exposing thinking_tokens would have reduced this months-long investigation to a dashboard line. Until providers expose that signal, users need to keep session logs, establish tool-usage baselines while things are good, and count what their hooks catch.