Is AI-generated code safe? Risks, a review checklist and the checks to automate

AI-generated code is as safe as the review and checks it passes through: on its own it often contains security flaws, and Veracode's 2025 testing found vulnerabilities in 45% of samples. This guide covers the most common security and quality problems, hallucinated packages, licensing, a reviewer's checklist and the automated checks every pull request should run.

Is AI-generated code safe to use in production?

AI-generated code is safe to use in production when it passes the same review, tests and security checks as any other code. Unreviewed, it is risky: models write code that works on the happy path and often leaves out the defensive parts a security reviewer looks for.

The best-known measurement comes from Veracode, which sells application security testing. Its 2025 GenAI Code Security Report tested over 100 models on tasks in Java, Python, C# and JavaScript and found that 45% of code samples failed security tests and introduced OWASP Top 10 vulnerabilities. Java was the worst at a 72% failure rate, and models failed to defend against cross-site scripting in 86% of relevant samples. Larger and newer models did not do better on security.

Treat those numbers as a description of unguided output on test tasks, not a defect rate for your codebase. The practical lesson holds either way: AI speeds up writing code, so your review and automated checks have to keep up with the extra volume.

What security problems show up most often in AI code?

The same handful of flaws appear again and again, because models reproduce common patterns from public code, including insecure ones.

Missing authorization deserves extra attention. Static analysis tools catch injection patterns well, but they can't know that user A shouldn't read user B's invoice. That check needs a human who knows the data model, or a test that tries it.

Why do AI models write insecure code?

Models write insecure code for predictable reasons, and knowing them tells you where to look:

• They learn from public code, which includes years of tutorials and answers written for clarity, not security.
• They optimize for "it works". A prompt asks for a feature; nothing in it asks for input validation, rate limiting or ownership checks.
• They lack your threat model. The model doesn't know which fields are sensitive, who your users are or which services are internet-facing.
• They produce plausible code. Clean, confident code gets less scrutiny from reviewers, which is the opposite of what it needs.

The fix for the third point is context. Put your security rules in the instructions your agent reads every session, for example in CLAUDE.md or AGENTS.md:

## Security rules- All database access goes through the query builder; never build SQL strings- Every API handler must call `requireOwnership()` or `requireRole()` before reading data- Secrets come from `process.env` via `src/config.ts`; never write keys into code or tests- Ask before adding any dependency, and state its weekly downloads and last release date

What quality risks come with AI-generated code?

Beyond security, AI code tends to fail in ways that pass a quick glance and cost you later. The common ones:

• Duplication. The model writes a new helper instead of finding the one you already have.
• Swallowed errors. A try/catch that logs and continues, hiding failures that should stop the request.
• Unrequested scope. Extra abstractions, options or refactors nobody asked for, which make the diff harder to review.
• Wrong edge cases. Off-by-one errors, time zone handling, empty lists, concurrent writes.
• Tests that prove nothing. Tests written from the implementation pass even when the logic is wrong. Our guide on how to write unit tests with AI covers how to avoid them.
• Confident comments and docs that describe what the code was meant to do, not what it does.

Small diffs are the best defense. A 60-line change gets a real review; a 1,500-line change gets skimmed. For more patterns like these, see our list of AI coding mistakes.

What are hallucinated packages, and why do they matter?

A hallucinated package is a dependency name a model invents because it sounds right. Attackers can register those names on npm or PyPI with malicious code, so a developer who installs what the model suggested runs the attacker's code. The technique is sometimes called "slopsquatting".

The scale is measurable. A study of 16 code-generating models and 576,000 code samples, published on arXiv and accepted at USENIX Security 2025, found that on average at least 5.2% of packages suggested by commercial models and 21.7% by open-source models did not exist, with 205,474 unique invented names.

Before installing any dependency an AI suggested:

1. Confirm the package exists on the official registry under exactly that name.
2. Check that it is the package you meant: repository link, maintainer, download count, last release.
3. Prefer packages already in your lockfile or your organization's approved list.
4. Run a dependency scan after installing.

A review checklist for AI-generated code

Use this checklist on every AI-assisted pull request. It focuses on the places AI output most often looks right and isn't.

• Scope: the diff does what the task asked and nothing else.
• Inputs: every external input is validated; queries are parameterized; no user input reaches a shell.
• Authorization: every data access checks that this user may perform this action on this record.
• Secrets: no keys, tokens or passwords in code, tests, fixtures or logs.
• Dependencies: each new package exists, is maintained, is needed and has an acceptable license.
• Errors: failures are handled or surfaced, not swallowed; users don't see stack traces.
• Defaults: debug flags, CORS, TLS and permissions are safe in production config.
• Tests: new behavior has tests with independent expected values, including a failure case.
• Reuse: no new helper that duplicates an existing one.
• Understanding: you can explain every line. If you can't, ask the model to explain, then verify.

The last item is the one that matters most. Code you don't understand is code you can't maintain or defend in an incident review. Our guide to AI code review covers how to use a second model as an extra reviewer without replacing the human one.

Which automated checks should run on every pull request?

Run four kinds of automated checks on every pull request: secret scanning, static analysis, dependency scanning and your test suite. They catch the mechanical problems so human reviewers can focus on logic and authorization.

Locally, three short commands scan the git history for secrets, run static analysis without an account, and check dependencies for known vulnerabilities:

gitleaks git -v .semgrep scanosv-scanner scan -r .

The gitleaks README also shows how to run it as a pre-commit hook, which stops secrets before they reach the repository. The Semgrep CLI docs recommend installing with pipx install semgrep. Put the same checks in CI so they run whether or not anyone remembered locally; our guide to Claude Code in GitHub Actions shows how an AI review step fits into the same pipeline.

Claude Code also ships a /security-review command that reviews the changes on your branch for risks such as injection, auth issues and data exposure. Use it as an extra pass, not a replacement for the scanners above.

Is AI-generated code a licensing or copyright risk?

AI-generated code carries two legal questions: whether it reproduces someone else's licensed code, and whether you can claim copyright in it. Both are manageable with a few habits.

On reproduction, verbatim matches are uncommon but possible. GitHub's code referencing docs say Copilot checks suggestions against public code and that matches occur in less than one percent of suggestions; depending on your organization's policy, matching suggestions are blocked or shown with the source and license. If a large block of generated code looks like a well-known library, search for it before committing.

On ownership, the US Copyright Office published a report on the copyrightability of AI output in January 2025; protection depends on human authorship, and the human contribution in your code (design, selection, editing) is what counts. Your tool's terms decide who holds rights to output between you and the vendor. For business-critical code, have counsel review your tools' terms.

Dependencies remain the bigger everyday licensing risk. An agent that adds a GPL-licensed package to a proprietary product creates a real obligation. Add a license check to your dependency review.

How do you run coding agents safely?

Coding agents add risks that autocomplete never had: they run commands, install packages, read files and fetch web pages. Malicious instructions hidden in a README, an issue or a web page can steer an agent, which is called prompt injection.

Practical limits:

• Least privilege. Allow the commands an agent needs (npm test, git commit) and deny the dangerous ones (git push to main, package installs, curl to arbitrary hosts).
• No production credentials. Agents work with development keys and test data only.
• Isolation for unattended runs. An agent running without supervision belongs in a container or separate machine, not on a laptop holding your SSH keys and browser sessions.
• Review at the pull request. Agents push branches and open pull requests; humans merge.

Our guide to prompt injection in coding agents covers attack patterns and layered defenses. If you want agents to work unattended without touching your own machine, Orkio runs them on an isolated cloud box, separate from other customers, where they push branches and open pull requests for you to review.

How to put this into practice this week

Five steps take most teams from "we hope it's fine" to a defensible process:

1. Write a one-page policy: approved tools, no secrets or customer data in prompts, human review for every merge.
2. Add security rules to your agent instructions (CLAUDE.md or AGENTS.md) with your actual helpers and conventions.
3. Turn on secret scanning, SAST and dependency scanning in CI and make them required checks.
4. Adopt the review checklist above in your pull request template.
5. Lock down agent permissions: deny installs and pushes to protected branches, and isolate unattended runs.

Frequently asked questions