Solumbe

Problem

A coding agent writes the diff and then tells you it is fine. The model that misread the request is the one reviewing the work, so it misreads it the same way twice. Before the edit it guesses which files matter from memory or a stale rules file. After it, nothing independent says whether the diff matches what was asked, or what it touched on the way.

Constraints

• No model in the verdict. The same checkout and the same request give the same answer on any machine.
• Nothing leaves the machine by default: no server, no account, no embeddings, no upload.
• It runs beside every agent host instead of replacing one: Claude Code, Claude Desktop, Codex, Cursor, VS Code, Gemini CLI, and Kimi Code.
• Indexing never writes into the repository it inspects. The code map lives in a per-user cache.
• A pass is not a merge approval. Hosted CI, code owners, and a human still decide.

Architecture

One Node.js package with two entry points and one runtime dependency, the TypeScript compiler, which it uses to read code. Below the entry points the core is a pure function of repository state: read it top to bottom, in the order a change moves through it.

Key decisions

1. Static analysis for the verdict

   Instead of asking a model to review the diff

   A model cannot judge its own output. One that misread the requirement misreads it again at temperature zero, and two runs agreeing proves nothing about the first. A deterministic check fails the same way every time, where you can see it.

2. Local-first, with no server

   Instead of hosted code search and embeddings

   No account, no upload, and the same query always gives the same packet. It also means nothing hosted can change a verdict: every hosted feature is opt-in and has a local equivalent that works fully.

3. Gating the exact staged tree

   Instead of scoring whatever is in the working directory

   A verdict is only evidence if someone else can recompute it. The receipt is bound to the base, the parent, and the staged tree, so the number in a pull request traces back to one exact change.

4. Declaring scope before the edit

   Instead of reading the diff afterwards and hoping

   The files a request should touch are recorded first, so the gate can name any file that was never declared. Growing the scope takes an amendment with a reason, and the reviewer reads that reason.

5. Grading risk weights against git history

   Instead of hand-tuned weights, or calling a calibrated model

   A risk level that has never been compared to an outcome is a heuristic wearing a number. History already holds the outcome: which commits were later repaired. Measuring it needs no network, no key, and no labelling budget.

6. Model routing as advice

   Instead of letting a model into the gate

   Some things live in the sentence and not the code, so a model read helps pick a tier. But an unreachable or unkeyed model should cost a tier, never a verdict, and only the host can actually switch models.

7. Delegating to gitleaks, Semgrep, and osv-scanner

   Instead of growing a scanner of my own

   The gate reads paths, names, and a short list of credential formats. It is not a vulnerability database and should not turn into one.

What broke, and the fix

1. Medium-risk changes were safer than low-risk ones.

   Cause

   A lockfile bump added two points, which pushed hundreds of otherwise-low commits into medium. Over 30 days on one repository, low, medium, and high changes were later repaired 26.9%, 17.3%, and 53.2% of the time.

   Fix

   Dependency manifests got a flag of their own, scored at zero and still shown to the reviewer. The same bands now read 10.1%, 42.1%, and 52.6%.

2. The first calibration said nearly every commit was later repaired.

   Cause

   It joined a fix to any earlier commit that touched the same file. At 90 days that called 91.4% of commits repaired, which separates nothing.

   Fix

   Blame the exact lines a fix changes and join on line overlap. The rate fell to 30.7% and the flags started to discriminate.

3. Fix commits were being graded as feature requests.

   Cause

   The rule that recognises a fix by its subject missed hot-fix, bug, patch, fixes, and fixed, so 119 fix commits across two repositories were invisible as repairs.

   Fix

   One rule shared by both graders, and saved runs can be regraded with it.

Outcome

• Published on npm and the MCP Registry: 14 tools over one stdio server, and one command installs it into seven agent hosts.
• On a 618-commit replay of one production API, changes the gate warned on were later repaired 52.2% of the time, against 32.0% for changes it passed. The intervals do not overlap.
• The same replay shows the limit. The gate raised an alarm on 56.0% of changes, and 47.7% of those alarms were on changes nothing later repaired.
• None of that shows the gate prevents a repair. A trial with a control arm is running to measure exactly that, and I will not quote a number until both arms hold enough changes.