RTL Review Lab

An LLM reviews Verilog work products against a 9-rule checklist; every number comes from planted defects with known ground truth. Code: github.com/pavanyadava007/rtl-review-lab

Browse the held-out files

Pick a file: the yellow line is where the defect was planted. Green findings match the planted defect, red ones are false positives. LLM run shown: . The first file is a spec constant the LLM caught and lint cannot see.

Results

Measured 2026-09-29 on one NVIDIA L4 (Qwen3-Coder 30B and Qwen2.5-Coder 7B through Ollama, temperature 0.3). Regenerate everything with make report; raw per-review output is in results/raw.jsonl.

What the numbers say

• Lint alone (Verilator -Wall + Yosys) is precise but blind to half the checklist: 0.56 recall at 0.95 precision on the held-out modules. It never sees a missing reset, a spec constant or an off-by-one, because those need the specification.
• The LLM alone, with a plain checklist prompt, is not usable: 0.48 recall at 0.34 precision. It flags correct code, misses every width defect and files "findings" that say the code is fine.
• Giving it the lint output (checklist+lint) raises recall to 0.77 at 0.68 precision.
• Making it walk through every rule before listing findings (walkthrough) lifts precision to 0.93 with no false findings on any clean file, at 0.70 recall. The cost: it reports a missing reset 0.60 of the time instead of 0.93.
• Hybrid (trust lint for what lint can see, ask the LLM only for the rest) with the walkthrough prompt: 0.70 recall at 0.95 precision, the same precision as lint with 14 points more recall. Hybrid with the plain checklist prompt has the highest recall, 0.80, at 0.70 precision. Which one to run is a team decision about how much noise reviewers accept.
• A second verify pass did not help: it removed correct findings along with wrong ones (0.66 recall, 0.95 precision).
• Spec conformance is the open problem. Constants and boundaries that differ from the specification header were found at most one time in three by the variants above.
• spectable (added after the first held-out run, built on dev only): a deterministic extractor lists every line with a constant or comparison, and the model must give a spec-vs-code verdict per line before its findings. Held-out, one seed: spec constants 0.44 (walkthrough 0.22) and boundaries 1 of 3 (walkthrough 0), but missing resets drop to 0.20 (from 0.60): the model moves its attention, it does not get better overall (0.65 recall, 0.84 precision). The same trade-off showed on dev.
• One focused pass per rule group (post hoc, defined after seeing the numbers above, one seed, so optimistic): lint for lint rules, walkthrough for R02/R04, spectable for R07/R08 gives 0.77 recall at 0.92 precision. This is the next thing to test properly on fresh modules, not a result to quote.
• Model size matters. Qwen2.5-Coder 7B with the walkthrough prompt: 0.65 recall at 0.43 precision and 2.0 false findings per clean file (30B: 0.93 precision, none). The 7B finds a missing reset one time in ten.

Scoring checks the rule and the location, not the wording of the explanation. A manual read of the 9 hits on review-only rules in one held-out walkthrough run (seed 0): 8 explain the defect correctly, 1 names the right register and rule but gives a wrong reason. A reviewer still has to read the finding.

Three things I got wrong first and fixed before the final runs: the model's answer to the same prompt changed between Ollama's cached and uncached evaluation, so every LLM number is averaged over three seeds; the mutant generator once named a planted signal unused_bits, and Verilator's default --unused-regexp is *unused*, so lint was told to ignore it (renamed, affected files re-run); and a random testbench retriggered timers too often to expose some spec defects, so validation now uses three stimulus settings.

Held-out test modules (11 modules, planted defects and clean files)

Recall per rule (held-out; * = rule lint tools cannot see)

Dev modules (prompt iteration, 6 modules, 1 seed)

Review time (NVIDIA L4, Ollama, all runs)