What is measured
Each rule (hardcoded secrets, SQL concatenation, duplicate code, function length, and so on) is scored on two axes, separately for each language it supports:- Precision - of the findings a rule reports, how many are real. Low precision means noise: the rule cries wolf.
- Recall - of the real issues present, how many the rule catches. Low recall means blind spots: the rule misses things.
How it works
The benchmark lives in the analysis engine itself, versioned alongside the rules it measures, so a rule change and its expected-result change are always one commit.- A labelled corpus. Every rule has a set of hand-written fixture files - realistic code, not toy snippets - split into positive fixtures (the rule should fire, with the exact line labelled) and negative fixtures (the rule must stay silent). Negative fixtures deliberately encode the tricky look-alikes: placeholder secrets, commented-out debug prints, parameterised SQL, test-file credentials, uniform data tables that resemble duplication.
- A deterministic run. The harness analyses every fixture with the shipped
balancedpreset - never a machine-specific config - so the numbers are reproducible. A finding matches a label when the rule id matches and the line is within one line of the expected line. - Precision and recall per rule, per language, computed from true positives, false positives, and false negatives.
Enforcement vs advisory rules
Every rule is classified as enforcement or advisory:- Enforcement rules are the high-confidence ones that feed the quality gate and can block a push or build - hardcoded secrets, SQL concatenation, disabled TLS verification, weak hashing, duplicate code, function and file length, complexity.
- Advisory rules are useful signals better suited to review than to blocking
- TODOs, magic numbers,
console.log, and similar.
- TODOs, magic numbers,
The regression tripwire
The measured result is saved as a checked-in baseline. Iris Code’s own test suite re-runs the benchmark on every change and fails the build if any rule’s precision or recall drops below its baseline, or if a previously measured rule becomes unmeasured. Improvements never slip in silently either - regenerating the baseline is an explicit, reviewable step, so a rule getting better shows up in a diff. Because that test runs in the same suite gating every release, no release can ship an accuracy regression unnoticed.Current results: enforcement rules
These are the rules that can actually block a push, build, or CI check - where accuracy matters most. The figures are measured against the labelled corpus and regenerated on every release. Precision is how often a finding is correct (the opposite of a false alarm); recall is how many of the real issues the rule catches.
Figures are aggregated across languages for readability; the per-language
breakdown, and the advisory-tier rules (TODOs, magic numbers, and the like,
which never block anything), are available on request. The one sub-100%
figure - SQL concatenation, where a sentence of prose containing a SQL keyword
can occasionally read like a query - is a known, documented limitation kept
honest rather than tuned away at the risk of missing a real injection.
Known limitations
We would rather state these plainly than imply the benchmark is the last word:- A corpus is not the universe of real code. It covers each rule’s common shapes and known false-positive traps, not every construct in every codebase.
- The analysers are pattern-based. Some false positives are structural - a SQL keyword inside a prose string, a regex-like arithmetic expression - and are documented rather than papered over.
- Line matching is within one line, so the benchmark does not penalise harmless off-by-one shifts from formatting.