On September 2, 2026, GitHub published an engineering account of changes to Copilot's efficiency. The useful lesson is not a cheaper model or a new subscription: a shorter tool response can make a coding task more expensive if the agent must recover missing information.
That creates a practical question for teams buying or building coding agents: what should count as a saving? Our answer is the cost of an accepted fix, with failed attempts and human repair still on the bill. Here is a measurement plan you can apply before rolling out output compression or changing your agent's model.
What the Copilot evidence actually shows
In GitHub's account, output reduction sometimes caused agents to reread information or repeat commands. Its eventual approach preserved source-like output and compressed repetitive logs selectively. These are findings from GitHub's tested workloads, not proof that every compression tool is counterproductive.
The distinction matters: a smaller intermediate message is an observation, not an outcome. A team can celebrate that observation while paying for another search, another model turn, or an engineer finishing the patch.
Do not turn the announcement into a universal configuration recommendation. Turn it into a hypothesis to test against your own repository.
Define the unit before calculating the price
For this experiment, define an accepted fix as a patch that satisfies the task's acceptance criteria, passes the relevant checks, and survives review without a material repair. That is our proposed operational definition, not a vendor metric.
Keep agent-only acceptance separate from human-assisted completion. Otherwise an agent that routinely hands unfinished work to a developer can look as effective as one that produces reviewable patches.
Use two views of the same cohort:
- Execution cost per accepted fix: all attributable model and tool-execution spending, including failed attempts, divided by accepted fixes.
- Human effort per accepted fix: review and repair time across the same cohort, divided by accepted fixes.
Keep these separate initially. Converting engineering time into money introduces an internal accounting assumption; record that assumption if you combine them later. A lower API bill and more review work is a tradeoff, not an unqualified win.
If no fixes are accepted, report that result directly. A zero-success batch does not have a meaningful finite cost per accepted fix.
A cheap attempt can be the expensive choice
Consider a deliberately invented example, not a benchmark or a claim about any product.
| Same task cohort | Setup A | Setup B |
|---|---|---|
| Total execution spending | $40 | $30 |
| Accepted fixes | 20 | 10 |
| Execution cost per accepted fix | $2 | $3 |
Setup B spends less overall but costs more per accepted fix. The table still cannot tell you which setup to deploy: perhaps B failed the difficult tasks your team most needs solved, or A required substantially more review.
That is why the denominator needs an explanation. Publish the task mix, success rate, and human effort next to the ratio. Do not silently exclude timeouts, abandoned runs, or expensive failures.
For subscription products, distinguish attributable usage from the actual invoice. Lower usage may create capacity without reducing a fixed subscription charge. Do not describe estimated marginal savings as cash saved unless your billing arrangement supports that conclusion.
Run a paired test, not a model beauty contest
Start with representative tasks from your own backlog: a localized bug, a cross-file change, a dependency adjustment, and a test repair are possible categories. Use sanitized fixtures where necessary. Write acceptance criteria before seeing either agent's patch.
Anthropic's evaluation guidance distinguishes what an agent says it achieved from the resulting state of its environment. For coding work, inspect the patch and execute the checks; a confident completion message is not the grader.
Our proposed experiment is deliberately narrow:
- Freeze the starting point. Give both setups the same repository revision, task instructions, permissions, and available tools.
- Change one factor. Compare output handling or model selection, not a bundle of unrelated changes.
- Reset between trials. Do not let one setup inherit the other's edited files or warmed task state unintentionally.
- Use the same stopping rules. Decide the retry, time, and spending limits before running the comparison.
- Retain every outcome. Record success, failure, timeout, and human intervention, alongside spending and elapsed time.
Repeat trials when feasible and report variation. Do not declare a winner from one unusually good patch. For small samples, show the actual counts and treat the result as exploratory.
Log the detour, not just the tokens
A useful run record contains the task identifier, setup version, acceptance outcome, execution cost, elapsed time, human review minutes, and a short failure category.
For compression experiments, add a recovery event whenever the agent retrieves an original log or repeats a command to regain omitted information. Inspect these events rather than assuming every reread is bad; verification can be legitimate.
Anthropic's tool-design guidance recommends measuring task runtime, tool calls, token consumption, and errors alongside accuracy. It also describes giving agents concise and detailed response options. That supports testing a recoverable interface, not treating brevity as sufficient evidence of quality.
Our rollout rule would be simple: accept an optimization only if the cost improvement survives the full task boundary without an unacceptable decline in acceptance or increase in human repair. Set those tolerances for your workflow before reviewing results.
Look at difficult categories separately. A favorable average can conceal a regression in the cross-file changes that justify using an agent in the first place.
Limitations: this is a test plan, not a savings claim
We have not run this comparison on TeqVolt's repositories, and we are not asserting that one model, compressor, or coding product wins. GitHub's findings are workload-specific; the example above is arithmetic, not measured performance.
Passing tests is also an incomplete definition of software quality. Weak tests can accept a bad fix, and a good patch may still need security or design review. Keep that review in the acceptance process rather than using the metric to eliminate it.
Finally, record the setup and evaluation date. A later model or harness update can invalidate the comparison even if the product name stays unchanged.
The Bottom Line
Buy completed, reviewable work—not an attractive token counter. Before deploying a cost optimization, test whether it reduces the bill for accepted fixes while preserving quality and keeping human repair visible. If it only makes the intermediate messages smaller, you have not yet demonstrated a saving.
