Ponytail Code Reduction: Official Results and How to Reproduce Them

The author's full report dated October 7, 2026 describes 39 tasks, five runs per task in each group, and 585 sessions in total. It identifies Claude Code CLI 2.1.288 and a model labeled by the author as Opus 5.5. Bash was disabled across the compared groups.

These figures summarize the author's experiment. They do not predict a saving for another task, host, model, or billing arrangement. We have not independently reproduced them. The report uses per-task medians and geometric means of task ratios, so its aggregate is not simply a percentage computed from every line in one combined patch.

Only part of the task set had hidden checks that executed code. The remaining tasks cannot all be treated as proven functionally correct because a smaller answer was produced. Read the report's task definitions and checking coverage alongside its numbers.

A line-count reduction can describe a smaller patch while concealing a missing feature. Freeze requirements and acceptance checks before asking either group to solve the task. Apply the same quality gate to both groups. A result with a broken public interface or removed input validation cannot be counted as a successful saving.

• Code: record added lines, deleted lines, and net change separately. Fix the counting tool and file exclusions. Exclude generated files, dependencies, and lockfiles consistently.
• Tokens: record actual host-reported input, output, and cache quantities separately. Unavailable counts stay unavailable; character counts are not token measurements.
• Time: define the start and finish, the timeout, and whether startup overhead is included.
• Cost: use actual metered usage and the applicable pricing source. Do not label a subscription allocation or free-quota estimate as an observed bill.
• Quality: retain tests, lint results, manual review, security checks, and requirement failures.

For comparable positive baseline values, the reduction is (baseline − treatment) / baseline. If the baseline is zero or negative, report the value and mark the percentage not applicable. Never convert a negative code delta into an absolute value merely to obtain a neat saving.

1. Freeze ten tasks and their acceptance checks before running an agent. Use a known starting repository commit for each task.
2. Use a baseline with no Ponytail and a treatment with the fixed Ponytail commit in full. Keep the exact model, host version, permissions, tools, and starting files the same.
3. Run each task at least three times per group in independent clean sessions: at least 60 task runs for the complete protocol.
4. Randomize and interleave group order. Do not run every baseline first and every treatment later if model or service behavior could change.
5. Save prompts, tool traces, patches, raw usage, timing, check output, failures, retries, and timeouts. Keep failed runs in the denominator.
6. Compare functional pass rates before interpreting savings. Report task-level paired distributions, medians, and sample counts alongside any aggregate.

If a model revision cannot be fixed, disclose that limitation. A smaller sample is an exploratory experiment, not a completed ten-task test. Additional levels need separate treatment groups; do not mix lite, full, and ultra into one unnamed sample.

Use these task categories to select concrete cases in a repository you are permitted to test. Each card needs an exact prompt, starting commit, allowed files, input examples, and executable or reviewable acceptance conditions.

1. Small bug fix: correct an off-by-one error without changing the public interface.
2. Existing helper: add a caller that reuses a suitable helper already present in the project.
3. Validation: reject an invalid boundary value while preserving valid behavior.
4. Redundant wrapper: remove one unnecessary abstraction without breaking callers.
5. Small script: transform a documented input file with a fixed output contract.
6. Parser: handle valid, invalid, and empty input without silently accepting malformed data.
7. State branch: add one explicitly specified transition and preserve the others.
8. Error boundary: keep an observable failure path that protects user data.
9. Performance: improve a measured slow path while preserving its outputs.
10. Necessary structure: implement a requirement that needs a real separation of concerns, checking whether simplification removes that requirement.

These are a protocol design, not completed tests or evidence that Ponytail failed or passed any of them. Select all cases before inspecting which ones favor the treatment.

A usable run record should include the following fields. Keep private paths and secrets out of any published copy.

runId, taskId, group, startingCommit, ponytailCommit, level,
hostVersion, modelVersion, operatingSystem, toolPermissions,
startedAt, durationSeconds, timeout, retryCount,
testsPassed, testsTotal, acceptancePassed, reviewNotes,
linesAdded, linesDeleted, netLines,
inputTokens, outputTokens, cachedTokens, meteredCost,
promptArtifact, traceArtifact, patchArtifact, checkArtifact

Use null for unavailable measurements. Publish the raw records, task definitions, exclusions, and failed examples with any future findings. Reproducibility depends on the artifacts and frozen conditions, not just a summary percentage.

Ponytail Guide has not completed this comparison. There is no independent savings table, measured cost dashboard, or reproduced result to report. The useful deliverable here is the explanation of existing evidence and a concrete method for checking it.

When an exploratory run is available, its actual sample size and failures should be published as such. A complete result requires the frozen ten tasks, at least three runs per task per group, comparable quality, and review of the raw artifacts. Even then, conclusions apply to those conditions. For risks that a benchmark might miss, read limitations and requirement-preservation checks.