The 16 held candidates and their grading record
The extraction record names every company, quotes every passage and carries every URL. It is private, and it stays private, because naming a company on this public surface is a claim about that company and this house makes that claim only through an admitted row. What each candidate carries here is the hash of the passage it rests on, which is the citation identifier and names nobody.
Gate the merge on a human review of agent-produced work
Do not merge an agent-authored change until a human review of it is recorded.
Checkable artifact: The pull request, showing a review event dated before the merge.
held · needs 2 admitted companies · 8 admitted today · Named operating systems · 5 captured passage(s)
Oleg's answer, held: Work going live without human review. Human review happens in production. Sdlc is fully executed by agents who play different roles including qa
Why held: The answer moves review into production, while this candidate requires a human review before merge.
At one person the reviewer is the author, so the exercise proves only that a review step happened. A second pair of eyes is the half one person cannot supply. The degradation is stated on the practice.
Attribute the agent in the record of the change
Name the agent as a co-author of the work it produced, in the commit or in the byline.
Checkable artifact: The commit trailer, or the author list on the published thing.
held · needs 2 admitted companies · 8 admitted today · Engineer authored cadence · 2 captured passage(s)
Human grading: not recorded.
One of its two supporters does not clear the admission bar, so this candidate falls below the threshold the moment the publication rule is applied. It is also the only candidate filed under its dimension.
Do the steps that must not vary in code the model never touches
In your agent pipeline, make the invariant steps plain code and leave the model only what is left.
Checkable artifact: The workflow definition, showing which steps call a model and which do not.
held · needs 2 admitted companies · 8 admitted today · Reproducible mechanism · 4 captured passage(s)
Oleg's answer, held: Our agents are not running on any sdk or self crafted loops, they are proper Claude code or codex harnesses. So enforcement is important but only can be done by the means available to steer the harnes
Why held: The answer rejects the candidate's self-crafted-loop premise and its observable fails R16.
Publish a named limitation of the approach you are recommending
In the same document that promotes an approach, name a specific thing it does not do or has not been checked for.
Checkable artifact: The limitation sentence, in the same published document as the recommendation.
held · needs 2 admitted companies · 8 admitted today · Adverse evidence · 3 captured passage(s)
Oleg's answer, held: a per-item human step that stops scaling
Why held: The answer names the reviewers' per-item limitation. This candidate asks for the act of publishing a limitation.
The loosest grouping in the set. The three supporting limitations are of different kinds: a capacity bound, an evaluation that cannot diagnose, and a checker with no checks of its own. They are grouped because the human action is one action and it produces one artifact. Split by kind, each supporter stands alone and the candidate dies.
Enumerate your system's failure modes by name
List the ways your own system fails, individually, before anyone asks.
Checkable artifact: The published list, one named mode per line.
held · needs 2 admitted companies · 8 admitted today · Adverse evidence · 2 captured passage(s)
Oleg's answer, held: a written list of failure modes with their costs
Why held: The submitted observable does not appear in both preserved blind descriptions, so R16 refuses publication.
Give each task its own disposable full environment
Boot a complete working copy of the project per task and tear it down when the task ends, so the agent never works in your live checkout.
Checkable artifact: The boot and teardown definition committed in the repository: container file, worktree script, sandbox configuration.
held · needs 2 admitted companies · 8 admitted today · Reproducible mechanism · 3 captured passage(s)
Oleg's answer, held: isolated environment per agent run
Why held: The submitted observable does not appear in both preserved blind descriptions, so R16 refuses publication.
Put the evidence a human would use to verify inside the agent's reach
Give the agent the logs, metrics, errors and rendered screens you would look at yourself, so it can check its own work.
Checkable artifact: The tool configuration granting those sources, plus a session transcript in which the agent cites one of them.
held · needs 2 admitted companies · 8 admitted today · Reproducible mechanism · 3 captured passage(s)
Oleg's answer, held: the agent wired into the same instruments an engineer uses
Why held: The submitted observable does not appear in both preserved blind descriptions, so R16 refuses publication.
Write the knowledge the agent needs into the place it actually loads
Take what you know and the agent does not, write it into the file the agent loads, and stop relying on it to find the information elsewhere.
Checkable artifact: The committed context or instruction file, and the agent's run using it.
held · needs 2 admitted companies · 8 admitted today · Reproducible mechanism · 2 captured passage(s)
Oleg's answer, held: written-up knowledge placed where the agent will read it
Why held: The submitted observable does not appear in both preserved blind descriptions, so R16 refuses publication.
Harvest what a human corrected back into the agent's instructions
When you correct the agent, write the correction into its instructions or into a check, so the same correction is never made twice.
Checkable artifact: The diff to the instruction file or the check, traceable to the correction that caused it.
held · needs 2 admitted companies · 8 admitted today · Human and societal consequences · 2 captured passage(s)
Oleg's answer, held: a human correction fed back into the agent's instructions or tooling
Why held: The submitted observable does not appear in both preserved blind descriptions, so R16 refuses publication.
Kept separate from p10 on the artifact test: p10 produces a context file, p11 produces a dated change to one whose provenance is the failure that prompted it. Merging them loses the observable that separates writing documentation from maintaining it.
Build a set of cases with known-correct answers and gate changes on it
Write cases whose right answer you already know, run the agent against them, and ship no context or configuration change that fails them.
Checkable artifact: The committed case set with its expected answers, and a results table per configuration.
held · needs 2 admitted companies · 8 admitted today · Reproducible mechanism · 3 captured passage(s)
Oleg's answer, held: an eval suite of cases with known answers
Why held: The submitted observable does not appear in both preserved blind descriptions, so R16 refuses publication.
Make each run emit a machine-readable record of itself, and measure from that record
Have the agent's run write its own trace or usage record, and compute any number you publish from that record. An estimate does not count.
Checkable artifact: The stored run records, and a published figure traceable to them.
held · needs 2 admitted companies · 8 admitted today · Reproducible mechanism · 3 captured passage(s)
Oleg's answer, held: traces and token usage
Why held: The submitted observable does not appear in both preserved blind descriptions, so R16 refuses publication.
Publish a production figure with its window, its population, and where it can be re-queried
Never publish a bare number about your own work. State the period it covers, the population it is drawn from, and the source someone could re-run.
Checkable artifact: The published figure carrying those three fields.
held · needs 2 admitted companies · 8 admitted today · Measured production results · 5 captured passage(s)
Oleg's answer, held: a usage number stated with its measurement window
Why held: The answer names the window but omits the candidate's population and re-query source, and it fails R16.
When you republish a figure, restate the earlier one beside it
Every time you update a number you have published before, put the previous value next to the new one, so the movement is visible without opening the old post.
Checkable artifact: The new publication carrying both values.
held · needs 2 admitted companies · 8 admitted today · Measured production results · 2 captured passage(s)
Oleg's answer, held: merged pull requests attributed to the agent
Why held: The answer names agent attribution. This candidate asks for the previous value beside the new value.
Cut personal data out of what the agent can reach, and say so
Decide what personal data the agent must never see, enforce it in the access path, and state the boundary publicly.
Checkable artifact: The access rule or exclusion in the configuration, and the published statement of it.
held · needs 2 admitted companies · 8 admitted today · Reproducible mechanism · 2 captured passage(s)
Oleg's answer, held: Ok
Why held: The answer records assent but supplies no observable, so R16 cannot publish it.
Keep the durable logic in your own harness so the model underneath can be swapped
Put the parts that outlive any model, the verification, the ordering and the recorded output, in code you own, behind a seam where the model is replaceable.
Checkable artifact: The harness, with the model named in one place, and a record of a swap that happened.
held · needs 2 admitted companies · 8 admitted today · Adverse evidence · 2 captured passage(s)
Oleg's answer, held: We keeping instructions and using flagship vendor harnesses
Why held: The answer chooses flagship vendor harnesses, while this candidate requires durable logic in a replaceable harness.
Express a repeated change as an automated pass that opens its own pull requests
When the same edit is needed in many places, write the pass that makes it and let it raise the pull requests, so you edit neither by hand nor file by file.
Checkable artifact: The transformation or scheduled job, and the pull requests it opened.
held · needs 2 admitted companies · 8 admitted today · Named operating systems · 2 captured passage(s)
Oleg's answer, held: pull requests
Why held: The phrase is common across unrelated candidates and does not resolve the reviewers' action mismatch.