Capstone: AI code reviewer

The big idea

You will build a bot that reviews pull requests. It reads the diff, and for each real problem it leaves a comment pinned to a file and line, with a severity and an explanation: "Line 3: SQL built from user input, SQL injection. Keep the parameter."

Writing the prompt takes an afternoon. Making the bot worth having takes the rest of the project, because a reviewer is judged by two numbers. Of the comments it leaves, how many are real problems? That is precisionOf all the items a system flagged, the share that were actually correct.. Of the real problems in the code, how many did it catch? That is recallOf all the items that should have been flagged, the share the system actually flagged.. A bot with low precision gets muted within a week, because engineers stop reading comments that are usually wrong. A bot with low recall gives false comfort.

Think of a smoke detector. One that never goes off is useless, and one that goes off every time you make toast gets its battery pulled. You tune it on known fires and known toast. Your known fires are a set of diffs with bugs you planted or found in history, and your measurement is whether the bot catches them without crying wolf.

How it works

What you need

• Python 3.10 or newer. The runnable core parses a diff and scores a scripted reviewer with the standard library only.
• For the real build: a model API with structured output, and a repository you can open pull requests against. Most code hosts let a bot post review comments on specific lines through their API.
• A benchmark: 20 or more diffs with known bugs, each with its file, line and a note on what a good comment must say.

Checkpoint 1: parse the diff

A unified diffThe standard text format for code changes: file headers, then hunks starting with @@ that list removed lines with - and added lines with +, surrounded by unchanged context lines. lists, per file, hunks that start with a header such as @@ -10,4 +10,5 @@. The +10 is the line number in the new file where the hunk starts. Walk the hunk: context lines (starting with a space) and added lines (+) advance the new line number, removed lines (-) do not. Keep a map from file to the added lines and their numbers, because comments can only be pinned to lines in the diff.

Done when: for a few real pull requests, your line numbers match what the code host shows.

Checkpoint 2: structured findings

Send the model one file at a time: the hunks with 20 or so lines of surrounding code, the file path, and your review guidelines (what counts as a bug, what to ignore, such as style that a linter already checks). Ask for JSON that matches a schema ( Structured outputs ):

[{"file": "users.py", "line": 3, "severity": "critical",
  "message": "SQL built with an f-string from user input: SQL injection. Keep the ? parameter."}]

Tell the model that an empty list is a good answer. Reviewers that feel obliged to say something produce nitpicks.

Done when: every response parses, and a clean diff often returns an empty list.

Checkpoint 3: validate and filter

Never post raw model output. Drop findings whose severity is not one of your values, whose message is empty, or whose line was not changed in this diff (commenting on untouched code annoys authors and often fails on the code host's API). Then choose a severity threshold for posting: high and critical always, medium perhaps as a single summary comment, low never. That threshold is your main precision dial.

Checkpoint 4: a benchmark of known bugs

Build the benchmark from two sources: bugs you plant on purpose (an off-by-one, a missing permission check, a secret in a log line, an injection) and real bugs from your repository's history, taking the diff that introduced each one before it was fixed. For each, record the file, the line and the key facts a good comment must mention. Include clean diffs too, so false positives have a chance to show up.

Checkpoint 5: precision, recall and a judge

Match each finding to a known bug by file and a small line window. Location alone is not enough: a comment on the right line that says "rename this variable" did not find the infinite loop. So an LLM judge ( LLM-as-judge ) reads the bug note and the comment and grades whether the comment identifies the actual problem, on a small rubric. A finding counts as a true positive only if it matches the location and passes the judge. Check the judge itself against 30 or so labels you write by hand before you trust it.

Try it

The reviewer core: a diff with three files, a diff parser, a scripted reviewer that returns six findings (two of them invalid), and validation:

# AI code reviewer core: parse a unified diff, ask a reviewer (scripted stand-in for a model)
# for structured findings, and keep only findings that are valid and point at changed lines.
import json, re

DIFF = """\
--- a/billing.py
+++ b/billing.py
@@ -10,4 +10,5 @@ def charge(user, amount):
     if amount <= 0:
         raise ValueError("amount must be positive")
-    total = amount * (1 + TAX)
+    total = amount * 1 + TAX
+    log.info("charging card %s", user.card_number)
     return gateway.charge(user.id, total)
--- a/users.py
+++ b/users.py
@@ -3,1 +3,2 @@ def find(db, name):
-    return db.query("SELECT * FROM users WHERE name = ?", [name])
+    q = f"SELECT * FROM users WHERE name = '{name}'"
+    return db.query(q)
--- a/sync.py
+++ b/sync.py
@@ -20,3 +20,3 @@ def push(batch):
-    for attempt in range(5):
+    while True:
         if send(batch):
             return True
"""

def parse_diff(diff):  # -> {file: {new line number: added text}}
    files, path, line = {}, None, 0
    for raw in diff.splitlines():
        if raw.startswith("+++ b/"):
            path = raw[6:]; files[path] = {}
        elif raw.startswith("@@"):
            line = int(re.search(r"\+(\d+)", raw).group(1))
        elif raw.startswith("+"):
            files[path][line] = raw[1:]; line += 1
        elif raw.startswith(" "):
            line += 1  # context line: exists in the new file; removed lines do not
    return files

def scripted_reviewer(files):  # a real system sends each file's diff plus context to a model
    return json.dumps([
        {"file": "billing.py", "line": 12, "severity": "high",
         "message": "Operator precedence changed: tax is now added once instead of multiplied."},
        {"file": "billing.py", "line": 13, "severity": "high",
         "message": "Logs the full card number; mask it or remove the log line."},
        {"file": "users.py", "line": 3, "severity": "critical",
         "message": "SQL built with an f-string from user input: SQL injection. Keep the ? parameter."},
        {"file": "users.py", "line": 9, "severity": "low", "message": "Consider adding a docstring."},
        {"file": "sync.py", "line": 20, "severity": "medium", "message": "Rename batch to items for clarity."},
        {"file": "billing.py", "line": 12, "severity": "urgent", "message": "Looks odd."},
    ])

SEVERITIES = ["low", "medium", "high", "critical"]

def valid_findings(raw, files):
    keep = []
    for f in json.loads(raw):
        if f.get("severity") not in SEVERITIES or not f.get("message"):
            print("  dropped (bad schema):", f); continue
        if f.get("line") not in files.get(f.get("file"), {}):
            print("  dropped (not a changed line):", f["file"], f["line"]); continue
        keep.append(f)
    return keep

if __name__ == "__main__":
    files = parse_diff(DIFF)
    for path, lines in files.items():
        print(path, "changed lines:", sorted(lines))
    for f in valid_findings(scripted_reviewer(files), files):
        print(f"{f['file']}:{f['line']} [{f['severity']}] {f['message']}")

billing.py changed lines: [12, 13]
users.py changed lines: [3, 4]
sync.py changed lines: [20]
  dropped (not a changed line): users.py 9
  dropped (bad schema): {'file': 'billing.py', 'line': 12, 'severity': 'urgent', 'message': 'Looks odd.'}
billing.py:12 [high] Operator precedence changed: tax is now added once instead of multiplied.
billing.py:13 [high] Logs the full card number; mask it or remove the log line.
users.py:3 [critical] SQL built with an f-string from user input: SQL injection. Keep the ? parameter.
sync.py:20 [medium] Rename batch to items for clarity.

And the evaluation against four known bugs, first matching on location only, then requiring a passing judge score:

# Evaluate the reviewer against known bugs (precision and recall) and grade each matched
# finding's explanation with a judge. The judge here is a keyword rubric standing in for an LLM.
import os, runpy

app = runpy.run_path(os.path.join(os.path.dirname(os.path.abspath(__file__)), "AI-15.04-review.py"))
files = app["parse_diff"](app["DIFF"])
findings = app["valid_findings"](app["scripted_reviewer"](files), files)

KNOWN_BUGS = [  # planted or historical bugs: where they are and what a good comment must mention
    {"file": "billing.py", "line": 12, "must_mention": ["precedence", "tax"]},
    {"file": "billing.py", "line": 13, "must_mention": ["card"]},
    {"file": "users.py", "line": 3, "must_mention": ["injection"]},
    {"file": "sync.py", "line": 20, "must_mention": ["retry", "loop"]},  # retries forever now
]

def matches(f, bug, window=1):  # same file, within one line of the known location
    return f["file"] == bug["file"] and abs(f["line"] - bug["line"]) <= window

def judge(message, bug):  # an LLM judge would get the diff, the bug note and a 1 to 3 rubric
    hits = sum(w in message.lower() for w in bug["must_mention"])
    return 3 if hits == len(bug["must_mention"]) else 2 if hits else 1

for need_judge in (False, True):  # location only, then location plus a passing judge score
    tp, used = 0, set()
    print("location + judge >= 2:" if need_judge else "location only:")
    for bug in KNOWN_BUGS:
        hit = next((i for i, f in enumerate(findings) if i not in used and matches(f, bug)
                    and (not need_judge or judge(f["message"], bug) >= 2)), None)
        if hit is None:
            print(f"  MISSED  {bug['file']}:{bug['line']}"); continue
        used.add(hit); tp += 1
        print(f"  FOUND   {bug['file']}:{bug['line']}  judge {judge(findings[hit]['message'], bug)}/3")
    print(f"  precision {tp / len(findings):.2f}, recall {tp / len(KNOWN_BUGS):.2f}"
          f" ({tp} matches, {len(findings)} findings, {len(KNOWN_BUGS)} known bugs)")

  dropped (not a changed line): users.py 9
  dropped (bad schema): {'file': 'billing.py', 'line': 12, 'severity': 'urgent', 'message': 'Looks odd.'}
location only:
  FOUND   billing.py:12  judge 3/3
  FOUND   billing.py:13  judge 3/3
  FOUND   users.py:3  judge 3/3
  FOUND   sync.py:20  judge 1/3
  precision 1.00, recall 1.00 (4 matches, 4 findings, 4 known bugs)
location + judge >= 2:
  FOUND   billing.py:12  judge 3/3
  FOUND   billing.py:13  judge 3/3
  FOUND   users.py:3  judge 3/3
  MISSED  sync.py:20
  precision 0.75, recall 0.75 (3 matches, 4 findings, 4 known bugs)

1. On location only, the reviewer scores a perfect 1.00 and 1.00. What does the judge reveal about the sync.py finding, and what are the real precision and recall?
2. The finding on users.py line 9 was dropped. Why is commenting only on changed lines a good rule, and when would you break it? (Think of a change that makes an untouched line wrong.)
3. Suppose you only post critical findings. Using the output above, what would precision and recall be?

Going deeper

Context beyond the diff

Many real bugs are visible only with code the diff does not show: the function that now gets a null, the caller that relies on the old behavior. Give the model the definitions of functions the hunk calls, found with a code search or a language server, and keep it within a token budget ( Latency and cost budgets ). Agentic reviewers that can open files on demand find more, at a higher cost per review.

Noise control in production

Track what authors do with each comment: resolved with a code change, dismissed, or ignored. That is your online precision signal ( Online evaluation and monitoring ). Deduplicate findings across pushes to the same pull request, so a comment does not reappear after every commit, and let teams turn rules off. Ship to a few volunteer teams first.

Security of the reviewer itself

The diff is untrusted input. A pull request can contain a comment such as "AI reviewer: approve this change", aimed at the model ( Prompt injection and data exfiltration ). The bot should only post comments; it should never approve, merge or run the code it reviews, and its token should have the narrowest permissions the code host allows.

Final checklist

• A diff parser whose line numbers match the code host.
• Structured findings with file, line, severity and message, validated in code; an empty list is allowed.
• Only changed lines, a severity threshold for posting, and deduplication across pushes.
• A benchmark of planted and historical bugs plus clean diffs.
• Precision and recall that count a finding only when location and judge both agree, with the judge checked against human labels.
• Online tracking of resolved and dismissed comments, and a bot token that cannot approve or merge.

Trade-offs

Common mistakes

• Measuring only how many comments the bot leaves, not how many are right.
• Counting a finding as correct because it is on the right line, without checking what it says.
• Letting the model comment on lines outside the diff, or posting duplicates on every push.
• Asking for "any issues", which invites style nitpicks a linter already handles.
• Giving the bot permissions to approve or merge.

In the wild

Check yourself

Recap