What is measured
Each rule (hardcoded secrets, SQL concatenation, duplicate code, function length, and so on) is scored on two axes, separately for each language it supports:- Precision - of the findings a rule reports, how many are real. Low precision means noise: the rule flags things that are fine.
- Recall - of the real issues present, how many the rule catches. Low recall means blind spots: the rule misses things.
minHealthScore threshold and optional category thresholds. An advisory finding can reduce the health score enough to fail that check. The async-void rule belongs to the enforcement group, but affects the gate through the score because it has no separate category threshold.
How it works
The benchmark lives in the analysis engine itself, versioned alongside the rules it measures, so a rule change and its expected-result change are always one commit.- A labelled collection of examples. Each rule has test files, called fixtures, with a known expected result. Positive examples contain a problem the rule should find; negative examples should produce no finding. Negative examples include fake secrets, debug prints inside comments and data tables that resemble repeated code. These test whether the rule can distinguish a problem from a harmless look-alike.
- A deterministic run. The harness analyses every fixture with the shipped
balancedpreset - never a machine-specific config - so the numbers are reproducible. A finding matches a label when the rule id matches and the line is within one line of the expected line. - Precision and recall per rule, per language. These are calculated from correct findings, false alarms and missed problems.
Enforcement and advisory groups
Every rule carries a group in the benchmark data: enforcement for the high-confidence rules describing a defect worth stopping a push over, advisory for signals better suited to review - TODOs, magic numbers,console.log.
The group describes how much confidence to place in a finding. Your gate settings separately decide whether it stops a change.
The gate can check a minimum health score and specific thresholds, such as the number of secrets or duplicate blocks allowed. Advisory findings can contribute to the score, so they can still affect whether the gate passes. A dedicated threshold applies only when you configure it.
See gate configuration for the full set of thresholds. The machine-readable rule list records each rule’s dedicated gateRule, where it has one.
The regression tripwire
The measured result is saved as a checked-in baseline. Iris Code’s own test suite re-runs the benchmark on every change and fails the build if any rule’s precision or recall drops below its baseline, or if a previously measured rule becomes unmeasured. Improvements never slip in silently either - regenerating the baseline is an explicit, reviewable step, so a rule getting better shows up in a diff. The release checks block regressions on these measured cases. They cannot prove that every possible input will be detected correctly.What “rule count” means
A rule is one check, such as looking for a hardcoded secret. The same rule can run in several languages, so counting rules, languages and tested combinations produces different totals. These names explain which total each figure represents:
Eight analysers cover ten supported languages. Vue and Svelte
<script> blocks are
analysed by the JavaScript and TypeScript analysers according to their lang attribute,
so they carry those languages’ rules rather than rules of their own. That is why the
table has eight columns while the product supports ten languages - both numbers are
correct.
A machine-readable copy of everything below, including these definitions, is published at
iriscode.co/benchmark/v1/rules.json.
It is generated from the same two files the benchmark itself reads, so it cannot disagree
with this page.
Current results: every measured rule
Precision is how often a finding is correct (the opposite of a false alarm); recall is how many of the real issues the rule catches. Both groups are published, and so is the sample each percentage came from. Read the Cases column before reading any percentage. It is the number of labelled fixtures behind that row, and most rules sit between one and fifteen. At those sizes a single false alarm moves precision by tens of points, so every cell below 100% carries its own counts. The imperfect cell issql-concatenation in JavaScript: two correct findings and one false alarm. It caught every planted SQL-concatenation case in that sample.
That cuts both ways - a row reading 100% on a single case is a much weaker claim than the
same figure on fifteen. A - means the rule was not measured in that language, never
that it scored zero. Per-language true positive, false positive and false negative counts
are in the machine-readable copy if
you want the arithmetic rather than the ratio.
Generated from the benchmark corpus in @iris-code/core 0.7.2. Every cell is precision / recall for that rule’s own implementation in that language; - means not measured, and the reasons are listed under the table.
38 rules measured across 8 analysers, covering 10 supported languages. 11 enforcement, 27 advisory. 174 measured rule-language pairs over 361 fixture files (245 positive, 186 negative). Unmeasured rules: 0.
Where a cell is empty, that rule has no fixtures for that language. Three distinct
reasons sit behind an empty cell, and they are not the same thing:
- Not applicable. The language has no such construct.
any-usageandmissing-return-typeare TypeScript-only by definition. - Not implemented. The check does not exist for that language yet. Unused-code detection is deliberately absent for Ruby, C# and Rust, because each resolves at runtime or through mechanisms static analysis cannot see - so an “unused” verdict is not one we can stand behind.
- Implemented but not measured. The rule runs, but no fixtures pin it in that language. These are reported loudly by the harness rather than hidden, and the count is published above.
Known false positives
One rule still reports a finding that is not real. It is listed here with its measured numbers, because a benchmark that omits its weak cells gives you no way to judge the strong ones.-
sql-concatenationin JavaScript: 2 correct, 1 false alarm, 0 missed, 66% precision. SQL keywords in ordinary prose joined with+can read as a query being assembled by concatenation. The TypeScript implementation measures 100% precision on the same corpus. Tightening the pattern would trade a rare and harmless false alarm for the risk of missing a real injection, and this rule feeds the gate, so recall matters more than precision here. Recall has stayed at 100%: the rule has never missed a planted injection on this corpus. A proper fix needs to know that a string is SQL bound for a query call, which requires parsing rather than a narrower pattern.
parse-error was listed here until core 0.4.6, at 50% precision in JavaScript. A regex
literal such as /{/ unbalanced the delimiter scan, so a file that parses correctly was
reported as broken. The lexer is now regex-literal aware, and the rule measures 100%
precision and 100% recall in all four languages that implement it.
What is not published
The corpus itself. It is 361 files of deliberately vulnerable code and credential-shaped strings, so publishing it would mean distributing a library of exploit fixtures to make a point about openness. The results and the method are here; the fixtures stay in the engine. Two rules are shared across languages on purpose rather than duplicated per language. Catch-all exception handling is one rule for both C# and Java: the same defect with the same fix, so one severity setting and one suppression comment cover a codebase that uses both. The JavaScript SQL-concatenation check can mistake a sentence containing a SQL keyword for a query. Its precision cell records that false alarm directly. It is a result for JavaScript, not an average across languages.Known limitations
We would rather state these plainly than imply the benchmark is the last word:- A corpus is not the universe of real code. It covers each rule’s common shapes and known false-positive traps, not every construct in every codebase.
- The analysers are pattern-based. Some false positives are structural - a SQL keyword inside a prose string, a regex-like arithmetic expression - and are documented rather than papered over.
- Line matching is within one line, so the benchmark does not penalise harmless off-by-one shifts from formatting.
Tell us when a finding is wrong
Every finding in the sidebar and Issues view has a small 👍 / 👎 control. If a rule flags something incorrectly - or nails something useful - rate it. The feedback is count-only: Iris Code records the rule id, the language, and your verdict, and nothing else. No source code, file paths, or line contents ever leave your machine, and the feedback respects the analytics setting in your account. Confirmed false positives become permanent negative fixtures in this corpus, so reporting one directly hardens the rule.Refactor suggestion evidence
Suggestions are separate from the findings benchmark above. Built coverage describes implemented detectors; it does not establish semantic correctness or behaviour preservation. Real-source review status: complete. Recorded observations cover 220 distinct plans. The completed corpus covers 20 plans for each of ten languages and 20 supplementary React plans. Per-observation inspection scope is recorded in the ledger; declaration seams do not certify a behavioural rewrite, and zero emissions do not prove detector accuracy.
The omission threshold is more than one definite nonsense suggestion in ten observations for that suggestion and language. Structural edits still need review, tests and verification.
Language coverage lists every implemented, inapplicable and omitted suggestion.