What Changed — The Problem with AI-Generated Changelogs
You've been there: your Claude Code agent auto-generates changelog entries for every merged PR, and the output reads like a compressed diff. Lines like "Updated retry_handler.py to add exponential backoff logic in the _should_retry method and modified the RetryConfig dataclass to include a new jitter field with default value 0.1" — technically correct, but useless to anyone who isn't reading the code.
One developer hit this wall hard. They run an autonomous coding agent that ships small changes daily, and their first month of auto-generated changelogs was unreadable. Subscribers to their public release feed started asking what half the entries meant. The core realization: "summarize the diff" and "write a changelog" are not the same task.
What It Means For You — The Two-Pass Fix
The developer's solution was a two-pass prompt structure, plus a third verification pass. Here's how to implement it in Claude Code.
Pass 1: Extract the user-facing effect
Instead of asking Claude to summarize the diff, force a translation step. Use this prompt:
You are not summarizing code. You are answering: "If I didn't read this
diff, what would I notice differently about this product tomorrow?"
Diff:
<diff>
If there is no user-facing effect (internal refactor, test-only change,
CI config), respond with exactly: NO_USER_IMPACT
The NO_USER_IMPACT escape hatch is the single highest-leverage line. Without it, the agent will fabricate a user-facing angle for pure refactors just to satisfy your request. With it, ~35% of merged PRs correctly get skipped.
Pass 2: Turn the effect into a changelog line
Once you have the user-facing effect, phrase it properly:
Turn this user-facing effect into one changelog line, in the style of
[Stripe / Linear / your product's changelog].
Rules:
- Lead with the verb (Fixed / Added / Improved / Removed)
- No file names, function names, or internal module references
- No implementation detail ("using X algorithm", "via Y library")
- Under 20 words
- If the effect is genuinely minor, it's fine to be blunt about that
Effect: <output from pass 1>
Splitting "what happened" from "how to phrase it" matters. When done in one prompt, the model anchors on the diff's vocabulary and technical terms leak through. The intermediate representation breaks that anchoring.
Pass 3: Blind verifier
Even with two passes, about 1 in 8 entries was still bad. The fix was a third pass run by a separate agent call with no access to the original diff:
Read only this changelog line: "<line>"
Would this make sense to someone who has never seen the codebase?
Fail it if it: references internal names, is vague to the point of
being meaningless, or assumes context the reader doesn't have.
Respond PASS or FAIL with a one-sentence reason.
Running the verifier "blind" is deliberate — if it can see the diff, it grades against the code again instead of against a naive reader's understanding. Anything that fails goes back through Pass 2 once with the verifier's reason as feedback. If it fails twice, drop the entry entirely.
Try It Now — Grouping and Ordering
Individual lines being good wasn't enough. A day with six merged PRs produced six flat bullets in arbitrary order. The final fix was a batching pass that runs once per release:
You have N changelog lines for today's release. Group them under
these headers, in this order, and omit any header with no entries:
Security
Fixed
Improved
Added
Removed
Within each group, order by how much a typical user would care,
most important first. Do not edit the wording of any line, only
group and reorder them.
This one change did more for perceived quality than either wording fix. A five-line changelog with a "Security" header at the top reads as trustworthy in a way six flat bullets never did.
What Didn't Work
Two approaches failed: giving the agent the PR title and description as extra context (it just copied the title's phrasing, including internal terms), and generating entries at PR-open time instead of merge time (PRs change enough that entries were stale ~25% of the time).
Results After a Month
- ~35% of merged PRs correctly marked
NO_USER_IMPACT - Verifier fail rate dropped from ~13% to ~3% after tuning Pass 2's rules
- Zero reader complaints since switching, versus multiple per week before
Lessons for Claude Code Users
- Treat "summarize the diff" and "write a changelog" as different prompts. One prompt can't do both well.
- Always include an explicit "nothing to report" escape hatch. It prevents fabrication.
- A diff-blind verifier catches different errors than a diff-aware one. Use it for anything user-facing.
- Force an intermediate representation to prevent vocabulary leaks.
- A dropped entry is cheaper than a confusing one. Optimize for never publishing something bad.
Source: dev.to
