What is measured
Each rule (hardcoded secrets, SQL concatenation, duplicate code, function length, and so on) is scored on two axes, separately for each language it supports:- Precision - of the findings a rule reports, how many are real. Low precision means noise: the rule flags things that are fine.
- Recall - of the real issues present, how many the rule catches. Low recall means blind spots: the rule misses things.
How it works
The benchmark lives in the analysis engine itself, versioned alongside the rules it measures, so a rule change and its expected-result change are always one commit.- A labelled corpus. Every rule has a set of hand-written fixture files - realistic code, not toy snippets - split into positive fixtures (the rule should fire, with the exact line labelled) and negative fixtures (the rule must stay silent). Negative fixtures deliberately encode the tricky look-alikes: placeholder secrets, commented-out debug prints, parameterised SQL, test-file credentials, uniform data tables that resemble duplication.
- A deterministic run. The harness analyses every fixture with the shipped
balancedpreset - never a machine-specific config - so the numbers are reproducible. A finding matches a label when the rule id matches and the line is within one line of the expected line. - Precision and recall per rule, per language, computed from true positives, false positives, and false negatives.
Enforcement and advisory groups
Every rule carries a group in the benchmark data: enforcement for the high-confidence rules describing a defect worth stopping a push over, advisory for signals better suited to review - TODOs, magic numbers,console.log.
This grouping is a confidence classification, and it is not the same thing as what
blocks a push. Being precise about that matters, because an earlier version of this
page said enforcement rules were “the rules that feed the quality gate” and advisory
rules “never block anything”, and neither half was quite true.
What actually decides a block is the gate, which has nine rules of its own:
- The health-score threshold (
minHealthScore). Every finding costs points, advisory ones included, so any rule can contribute to a failing score. - Eight specific caps - secrets, complexity, file length, smells per file, security smells, suppressions, duplicate blocks and naming violations. Each has its own configurable limit and is off unless you set it.
async-void, for instance - reaches the gate only through
the score like any other. The group tells you how much to trust a finding; the
gate configuration tells you what stops a push. Each
rule’s gateRule, where it has one, is in the
machine-readable copy.
The regression tripwire
The measured result is saved as a checked-in baseline. Iris Code’s own test suite re-runs the benchmark on every change and fails the build if any rule’s precision or recall drops below its baseline, or if a previously measured rule becomes unmeasured. Improvements never slip in silently either - regenerating the baseline is an explicit, reviewable step, so a rule getting better shows up in a diff. Because that test runs in the same suite gating every release, no release can ship an accuracy regression unnoticed.What “rule count” means
Four different numbers all get called “the rule count”, and conflating them is the single most common mistake made about this page - including by AI agents summarising the site, one of which counted entries in the configuration schema and reported a different figure again. Each has a name here:
Eight analysers cover ten supported languages. Vue and Svelte
<script> blocks are
analysed by the JavaScript and TypeScript analysers according to their lang attribute,
so they carry those languages’ rules rather than rules of their own. That is why the
table has eight columns while the product supports ten languages - both numbers are
correct.
A machine-readable copy of everything below, including these definitions, is published at
iriscode.co/benchmark/v1/rules.json.
It is generated from the same two files the benchmark itself reads, so it cannot disagree
with this page.
Current results: every measured rule
Precision is how often a finding is correct (the opposite of a false alarm); recall is how many of the real issues the rule catches. Both groups are published, and so is the sample each percentage came from. Read the Cases column before reading any percentage. It is the number of labelled fixtures behind that row, and most rules sit between one and fifteen. At those sizes a single false alarm moves precision by tens of points, so every cell below 100% carries its own counts. There are two:parse-error in JavaScript is one correct finding and one
false alarm, and sql-concatenation in JavaScript is two correct findings and one false
alarm. Neither has ever missed a planted case.
That cuts both ways - a row reading 100% on a single case is a much weaker claim than the
same figure on fifteen. A — means the rule was not measured in that language, never
that it scored zero. Per-language true positive, false positive and false negative counts
are in the machine-readable copy if
you want the arithmetic rather than the ratio.
Generated from the benchmark corpus in @iris-code/core 0.4.2. Every cell is precision / recall for that rule’s own implementation in that language; — means not measured, and the reasons are listed under the table.
37 rules measured across 8 analysers, covering 10 supported languages. 11 enforcement, 26 advisory. 168 measured rule-language pairs over 347 fixture files (239 positive, 178 negative). Unmeasured rules: 0.
Where a cell is empty, that rule has no fixtures for that language. Three distinct
reasons sit behind an empty cell, and they are not the same thing:
- Not applicable. The language has no such construct.
any-usageandmissing-return-typeare TypeScript-only by definition. - Not implemented. The check does not exist for that language yet. Unused-code detection is deliberately absent for Ruby, C# and Rust, because each resolves at runtime or through mechanisms static analysis cannot see - so an “unused” verdict is not one we can stand behind.
- Implemented but not measured. The rule runs, but no fixtures pin it in that language. These are reported loudly by the harness rather than hidden, and the count is published above.
Known false positives
Two remain, both published deliberately. A tool that hides its weak spots is not one to believe about its strong ones.sql-concatenation(JavaScript: two correct findings, one false alarm, no misses - 66% precision). SQL keywords in ordinary prose joined by+can read as a query built by concatenation. The TypeScript implementation measures clean at 100%; the JavaScript one does not, and the honest fix needs to know a string is SQL bound for a query call rather than a tighter pattern. Tightening the regex would trade a rare, harmless false alarm for the risk of missing a real injection on a rule that feeds the quality gate, and recall matters more here than precision. Recall stays at 100%: it has never missed a planted injection on this corpus.parse-error(JavaScript: one correct finding, one false alarm, no misses - 50% precision, the lowest figure on this page). A regex literal such as/{/can unbalance the delimiter scan that decides whether a file parses. Fixing it properly needs regex-literal-aware lexing rather than a heuristic, because this rule decides whether a file is analysed at all.
What is not published
The corpus itself. It is 347 files of deliberately vulnerable code and credential-shaped strings, so publishing it would mean distributing a library of exploit fixtures to make a point about openness. The results and the method are here; the fixtures stay in the engine. Two rules are shared across languages on purpose rather than duplicated per language. Catch-all exception handling is one rule for both C# and Java: the same defect with the same fix, so one severity setting and one suppression comment cover a codebase that uses both. The one sub-100% figure - SQL concatenation, where a sentence of prose containing a SQL keyword can occasionally read like a query - is a known, documented limitation kept honest rather than tuned away at the risk of missing a real injection. It affects the JavaScript implementation only; the figure shown is the average across all eight measured languages.Known limitations
We would rather state these plainly than imply the benchmark is the last word:- A corpus is not the universe of real code. It covers each rule’s common shapes and known false-positive traps, not every construct in every codebase.
- The analysers are pattern-based. Some false positives are structural - a SQL keyword inside a prose string, a regex-like arithmetic expression - and are documented rather than papered over.
- Line matching is within one line, so the benchmark does not penalise harmless off-by-one shifts from formatting.