Recursive Self-Improvement in Codex

A coding agent can finish a feature and still leave the system that produced it unchanged. Here you will take a repeated failure in software delivery, encode one repair in a Codex skill, and test whether that repair helps on later tasks.

The worked case is a navigation product. A team asks for temporary road closures. The agent adds closed-road markers to the map; the route search still chooses the closed edge and voice guidance still tells the driver to take it. That is a product failure. If the same integration miss appears across tasks, it is also a candidate harness failure. We will test a route-constraint verification skill, then try it on a different restriction such as vehicle height. The same protocol works for permissions, payment flows, migrations, refactors, and CI repair.

You need: a Git repository, the Codex CLI, one executable acceptance check or a named human reviewer, and permission to use two disposable worktrees. Commands that say npm run ... are examples: replace them with your repository's real commands. The evaluation script and fixtures live outside the agent's writable checkout. This guide is the implementation manual; the shorter story explains why the two loops matter.

1. Turn the user need into a frozen task

Start with behavior a user can observe. “Show closure pins” is too narrow for a navigation feature. A route planner consumes the same road restriction in search, displayed route geometry, ETA, and spoken maneuvers. Name those surfaces in the brief. Freeze it before either H0 or H1 run. H0 is the current instructions and skills; H1 is exactly H0 plus one candidate skill.

Put the task pool and evaluator in a controller directory outside both worktrees. Select a few development tasks to diagnose the pattern, separate validation tasks to choose the candidate, and untouched audit tasks to check transfer. Include a control task where the skill should stay inactive. For a first pilot, a small pool teaches you the mechanics; it does not establish a general performance claim.

The next fixture format is deliberately small. Your adapter must drive the real product (API, service, simulator, or browser), save its returned routeEdgeIds and maneuvers, and keep its invocation identical for H0 and H1. A hand-written result JSON does not test the product. Keep the fixture and evaluator unavailable for editing from the run worktrees; a read-only mount or a separate evaluator job works.

That evaluator catches an unsafe route and an unsafe voice instruction separately. It does not prove the adapter used real closure data or that ETA is correct: add integration checks for those claims before using them as acceptance criteria. A failed adapter or unavailable simulator is blocked, never a pass. Adapt the fields to your domain while keeping the principle: the oracle observes the product, not Codex's prose.

2. Capture a failure trace before writing a rule

Run current H0 on actual work. Save codex exec --json events, the final report, the diff, and the external evaluator result. The JSONL contains tool and usage events; -o saves the final message. That final message is a claim, not acceptance. In the navigation case, inspect which hop failed: restriction ingestion, graph-edge eligibility, route construction, guidance generation, or test coverage. A single bad implementation does not justify a permanent skill. Look for the same mechanism across multiple tasks, including a case where the agent could have checked it with available tools.

Write a one-line hypothesis that can be falsified: “When route behavior changes, a skill that traces restriction data through both route geometry and guidance reduces accepted misses without hurting unrelated tasks.” This is more useful than “be thorough.”

Set LAB to a directory outside APP in your real run; the sample paths are placeholders. Put the frozen brief at $LAB/tasks/NAV-41.md and do this diagnostic run in a disposable checkout. Save the baseline test output first. If git status --short prints changes, preserve them and create a clean worktree at a recorded commit. The --sandbox workspace-write flag permits edits in the worktree; it does not protect a task fixture placed inside that same worktree.

3. Make one versioned harness change

Put stable repository facts and commands in AGENTS.md. Codex reads a chain of instruction files from the Git root toward the working directory; nearer instructions can override broader ones. Put this narrow procedure in .agents/skills/, with required name and description frontmatter. Its trigger matters: a skill that fires on every task can inflate cost and distract the agent. Keep H0 unchanged and store the H1 file separately until testing.

Do not add a new model, permission, hook, MCP server, and skill together. If H1 wins, you need to know which change caused it. If the failure is missing tool access rather than a verification habit, test a tool or environment repair as a separate candidate. A skill cannot enforce a security boundary; keep evaluator files and promotion rights under controller ownership.

4. Replay matched tasks in isolated Codex runs

For each validation task, create two fresh worktrees from that task's same application commit. H0 receives the current harness; H1 receives only the candidate skill. Keep the exact task brief, model, Codex version, dependencies, sandbox, allowed services, timeout, and evaluator commit fixed. Run each in a fresh codex exec process. Do not resume a prior chat: its context may carry the answer or the failed attempt.

Below is the worktree skeleton for one pair. It assumes APP points to a clean Git repo, BASE is a commit, and LAB is outside the repo. Save the H1 skill shown above at $LAB/candidate/verify-route-constraints/SKILL.md; keep it absent from H0. Run the two Codex commands serially if they share external services or state. The task brief is injected as text; the agent does not need write access to the controller directory.

The example tests a newly added skill absent from the base commit. If your repository already has that skill, define H0/H1 at explicit harness commits instead of copying over it. Pin the model through your Codex configuration or CLI option supported by your installed version, and record the resolved value. Do not use --ephemeral if you rely on Codex's saved rollout history; the explicit event files above remain your portable trace. Never put secrets in shared trace files.

After each run, the controller drives the app with its adapter at 09:30 and 12:00, saves two product response files, and calls node "$LAB/evaluate-route.mjs" "$LAB/fixtures/NAV-41.json" "$LAB/runs/NAV-41-H0.route.json" plus the matching control fixture and result (then repeats for H1). The 12:00 fixture assumes E17 is the shortest open edge, so its required inclusion is meaningful. Run broader repository tests and inspect the diff as well. Do not let the worktree write $LAB/fixtures or $LAB/evaluate-route.mjs. Repeat on validation and control tasks, then on untouched audit tasks after candidate selection.

5. Compare per-task evidence and decide

Log one pair per task: split, start commit, harness fingerprint, resolved model and CLI version, tool permissions, environment/lockfile, evaluator version, raw trace, independent acceptance, usage or cost, human rework minutes, and any critical regression. Compare paired wins, losses, and ties. An aggregate pass rate can conceal that H1 fixed one task and broke another. Report blocked and unreviewed runs separately. A candidate with one dramatic win on a task used to design it has not demonstrated transfer.

Predeclare a rule before seeing H1: for example, validation wins must exceed losses; audit must have no net loss; critical safety or privacy regressions reject immediately; extra rework must remain under a stated limit. Small sets are directional. For a consequential change, use more tasks, repeat runs to expose variance, and get a fresh domain review. A human promotes the exact H1 commit only after reading failed cases. Keep the H0 commit for rollback and evaluate the next harness change against the promoted version, not against a moving mixture.

Replace placeholders and null usage with observed values from the run artifacts; keep fail, blocked, and unreviewed distinct. This example pair is illustrative, not a measured claim about Codex.

Adapt the method to any software task

Change the oracle, not the experimental controls. For a payment feature, check ledger entries, retries, and duplicate charges. For an authorization bug, test owner and unauthorized identities. For a database migration, verify old and new reads, rollback, and production-shaped data. For a refactor, assert invariant behavior and measure performance. For CI repair, evaluate a cold checkout in the target runner. In each case: user outcome → independent acceptance → repeated trace diagnosis → one harness change → matched replay → fresh audit → human decision.

If you code by directing agents rather than writing tests: ask a developer or a separate reviewer to own the acceptance checklist and run the same user journey on both worktrees. Save screenshots, API responses, and failure descriptions with task IDs. Do not ask the same agent that built H1 to judge its own result. A human checklist is slower than an automated evaluator, but it is still usable when the observations are concrete and recorded.

Troubleshooting: If Codex ignores the skill, check its path and frontmatter, the working directory and instruction chain, and test an explicit $verify-route-constraints invocation. If results vary, pin versions and repeat the pair. If tests are green but behavior is wrong, fix the external acceptance oracle before testing another candidate. A schema-constrained final report can improve log parsing, but it cannot certify the delivered code.