Let an Open-Weight Model Audit Your Code, but Do Not Give It the Keys

Let an Open-Weight Model Audit Your Code, but Do Not Give It the Keys

Revised
Published
~ 7 min read

#ai#automation#security#software engineering

A nightly job checks out the default branch, looks for exploitable paths and proposes small fixes. By morning, useful findings are ready for review.

This is a sensible use of a capable open-weight model such as Kimi K3, provided the model is treated as an untrusted analyst. It should not hold repository credentials, reach production, publish vulnerability details or merge patches. A deterministic system must enforce what it can read, run and change.

Why run this now?

Agents can already inspect public code, retry exploits and vary inputs with little supervision. Anthropic’s SCONE-bench research demonstrated working exploits against historically vulnerable smart contracts and found two novel vulnerabilities in a controlled experiment. Smart contracts are unusually testable, so these results should not be generalised to every application.

OpenAI’s EVMbench found that agents performed better at exploiting a known target than at finding every vulnerability or patching one without breaking behaviour. That is a useful warning for defenders: a model may find a real bug yet still stop early, invent findings or produce an unsafe fix.

Kimi K3 is capable, but it is not an autonomous expert. A UK AISI and US CAISI assessment found that it reached step 17 on average in a simulated 32-step corporate attack, compared with 28.5 for the strongest US models tested. That is enough capability to justify preparation, not alarmism.

The defensive approach is familiar: use threat modelling, static analysis, fuzzing and least privilege to remove weaknesses before an incident. The model is another fallible reviewer, useful mainly because it can connect evidence across files and spend time on paths that a normal pull-request review may miss.

Abstract illustration of many probing paths meeting a smaller, hardened software boundary Remove reachable attack paths rather than assuming every probe will be detected.

Split the audit into five stages

  1. Select evidence. Check out an exact commit and collect the threat model, dependency graph, recent diffs and deterministic scanner results.
  2. Discover. Give the model read-only access to that snapshot and require structured findings.
  3. Validate. Reproduce each promising finding in a disposable sandbox with synthetic data.
  4. Remediate. Generate a minimal patch and regression test for a validated issue.
  5. Publish. Let a separate broker enforce policy and open a pull request or private advisory with a short-lived credential.

Do not let the audited repository define the workflow that holds publication authority. Run the audit from a centrally owned security repository or external orchestrator, with pinned runner images, tools, model versions and policy.

This separation limits excessive agency. A stronger system prompt cannot compensate for an agent that can read attacker-controlled text, run arbitrary commands and push a branch.

Keep inference separate from execution

Moonshot publishes Kimi K3’s weights, which lets an organisation pin the model and keep source code in an approved environment. K3 has 2.8 trillion parameters, although only 104 billion are active per token, so self-hosting it is still a substantial infrastructure decision. A smaller model may be the better choice if it passes a repository-specific evaluation.

If a hosted endpoint is cheaper, check that sending source code and possible zero-days to the provider meets the organisation’s data and retention rules. Open weight describes access to the model artefact, not where an API request is processed.

In either case:

  • the model server receives text and returns text or structured tool requests
  • the audit harness owns the filesystem and implements tools
  • the model server has no Git credential, shell, Docker socket or cloud identity
  • model weights, runner images and dependencies are pinned by digest
  • every model or harness update must pass the same seeded vulnerability tests

Do not load arbitrary model repository code into the audit runner. Build and verify the weights separately, then serve them from an immutable image or volume.

Enforce restrictions in the runner

The following is configuration for an audit harness, not a security boundary by itself. The runner and publication broker must enforce every field.

scope:
    repository: acme/payments-api
    base_ref: main
    commit: "resolved-before-the-run"
    deny_write:
        - ".github/**"
        - "CODEOWNERS"
        - "SECURITY.md"
        - "AGENTS.md"
        - "**/*.lock"
        - "migrations/**"

runtime:
    disposable: true
    timeout_minutes: 45
    cpu_limit: 4
    memory_limit_mb: 16384
    network: none
    secrets: none
    production_routes: none

tools:
    read_files: true
    search_code: true
    write_workspace: true
    run_allowlist:
        - "semgrep --config=/opt/audit-rules/semgrep.yml --json"
        - "npm test -- --runInBand"
        - "npm run lint"
    docker_socket: false
    package_install: false

patch:
    max_files: 8
    max_changed_lines: 300
    forbid_binary_files: true
    forbid_symlinks: true
    forbid_submodules: true
    require_regression_test: true

publish:
    mode: brokered
    auto_merge: false
    public_vulnerability_details: false
    human_security_review: required

An unrestricted shell is difficult to contain. Build a per-project runner image with dependencies installed. If tests need a database, use an isolated fixture containing synthetic data. Fetch dependencies through an allow-listed mirror in a separate preparation stage, then disable the network before running repository code.

Workflow files are not the only sensitive paths. Agent instructions, code-owner rules, lock files and migrations can change the trust boundary or conceal a risky patch. Report proposed changes to these files for manual handling.

Assume the repository is hostile input

The default branch is more trusted than a fork, but it is still input. Comments, fixtures, documentation, package scripts and agent instruction files can redirect a model. Dependencies can attack the runner when tests start.

Use an ephemeral virtual machine or microVM. Mount source read-only during discovery, then copy it to disposable storage for validation and patching. Block metadata services, internal DNS, package registries, the Docker socket and the wider network unless a test has a documented need.

Never mount .env files, home directories, SSH agents or package-manager credentials. Treat model output as hostile too: paths, patches and suggested commands remain data until the broker parses and approves them.

Abstract illustration of a bright analysis chamber isolated inside several layered security boundaries The sandbox contains repository code, dependencies and model mistakes as well as the attempted exploit.

Require reproducible findings

Plausible warnings quickly become noise. Require each finding to contain:

  • the affected file, exact commit and violated trust boundary
  • attacker prerequisites and a concrete path to impact
  • a safe reproduction or failing regression test
  • a minimal patch with residual risk stated
  • separate confidence and severity values
  • the model, weights digest, harness version, tools and commands used

Run deterministic tools first. Feed CodeQL, Semgrep, dependency, secret and infrastructure scan results to the model for triage and cross-file reasoning. Put recurring machine findings in SARIF-backed code scanning rather than opening a new pull request every night.

Measure accepted findings, false positives, duplicates, broken patches and review time. Seed the evaluation with faults that resemble the application, such as a missing tenant filter, SSRF through a webhook tester or an over-permissive IAM policy. Re-run them whenever the model, prompt, tools or sandbox changes.

Keep publication deterministic

The publication broker only needs to check that:

  • the finding refers to the exact base commit
  • the patch applies cleanly and touches allowed paths within size limits
  • it introduces no binary, symlink, submodule, workflow or policy file
  • the regression test fails before the patch and passes afterwards
  • the standard CI suite passes in a fresh, secret-free environment
  • no equivalent finding or pull request is already open

Only then should the broker request a short-lived token from a narrowly scoped GitHub App. Protect the default branch with human review, code-owner approval for sensitive paths and required status checks, with no bypass for the bot.

For a public repository, do not publish an unpatched high-severity vulnerability in a pull request. Use a draft private security advisory and its temporary private fork.

Use more than one cadence

A full nightly audit repeatedly inspects stable code and produces duplicate findings. A better schedule is:

  • deterministic tools and a narrow model review on each pull-request diff
  • a nightly audit of recently changed trust boundaries
  • a weekly rotation through the wider codebase, plus a run after material architecture changes

Cap tokens, tool calls and wall-clock time, and record which areas were covered. A run that exhausts its budget is incomplete, not a clean bill of health.

Start with one repository, representative seeded vulnerabilities and a report-only run. Allow patch artefacts only after the findings are useful, then add brokered pull requests after testing the policy against malicious output. Keep merge authority with named reviewers.

Sources

all posts →