I let GPT and Claude independently audit my startup. Then I made them argue.
I wanted to know something:
Can AI act like independent senior security engineers?
So I gave multiple frontier models access to the same production codebase. They independently audited it. Then I gave each model the other’s report and told them to tear it apart.
The results surprised me.
This wasn’t a benchmark. It was an experiment in whether independent AI reviewers could improve each other’s work through adversarial peer review.
(Quick context: this is LifeXP, a gamified personal growth app I’m building. The app isn’t in production. I’m intentionally omitting exploit details because that information is confidential; I’m sharing categories and process, not a checklist to skip. If you’re building something similar, treat this as a template.)
The setup
LifeXP — three repos, one system:
Backend: Django, PostgreSQL, Redis, Celery
Web: Next.js
Mobile: Flutter
Realtime layer: Convex (handles live session state)
I gave each model the same instruction:
Read everything. Don’t modify code. Produce a production-readiness and security audit. No patches. No fixes. Just think.
Models used: GPT-5.6 Sol and Claude Opus 4.8.
I also tried Claude Fable 5 first. It refused:
“Fable 5’s safeguards flagged this message. The safeguards are intentionally broad right now and may flag safe and routine coding, cybersecurity, or biology work... Switched to Opus 4.8.”
Known issue, but funny that Anthropic’s own guardrails won’t currently let you point their most capable public model at your own codebase for a security review, because it’s too capable to trust with the request by default. So Opus did the job instead.
The first surprise
Sol spent 48 minutes autonomously traversing and reasoning about the repositories before producing the report.
Not generating code. Reading. Understanding architecture. Following authentication. Tracing data flow. Building a threat model before writing a single line of the report.
That alone felt weird. Watching an AI spend nearly an hour doing nothing except understanding software would’ve sounded ridiculous a year ago.
Independent audits
Both models, working alone with no shared context, reviewed 3 production repositories (Django backend, Next.js web, Flutter mobile) and flagged the same broad categories:
Anonymous/unauthenticated write access to progression data (XP manipulation)
Broken authorization and IDOR-style access control gaps
PII exposure through over-permissive serializers
Legacy authentication paths enabling account takeover
Missing rate limiting / abuse controls
A denial-of-service path through shared cache/queue infrastructure
Realtime backend (Convex) trusting caller-supplied identity and values
Leaderboard and feed queries that don’t scale
Mobile release-configuration issues (signing, permissions, deep links)
Migration/build blockers that would break a clean deploy
The overlap was much higher than I expected. Most of the highest-severity findings overlapped independently.
More interestingly, they independently converged on the same architectural problems, not just the same files.
Initial audit output, by the numbers (again — independent, no shared context):
Total review time for both first passes: ~57 minutes (48 min Sol + 8:48 Opus)
Then I got curious
Instead of comparing the two reports myself, I made the models compare each other.
Prompt for the adversarial pass:
Assume nothing. Challenge every finding. Find: false positives, missed vulnerabilities, incorrect severities, architectural misconceptions, missing exploit paths. Explain disagreements with references to the code.
To be explicit about the design, since it’s the whole point: each model did its independent first pass with no shared context — neither saw the other’s work while auditing. Only afterward were the reports exchanged, one-directionally: Sol got Opus’s report, Opus got Sol’s report. Neither got to see what the other model’s review said until it was done.
This is where it got interesting
I expected some flavor of “no, I’m right.” Instead, both models behaved like actual reviewers going through someone else’s PR.
Severity got corrected — in both directions. A hardcoded signing secret that the first pass called Critical (implying direct session forgery) got walked back to Medium once the reviewing model actually checked how sessions were implemented and found the exploit chain didn’t hold up the way it was described. Meanwhile, a takeover path that the first pass had filed as “Medium confidence, depends on external config” got upgraded to Critical once the reviewer traced a second, config-independent path to the same outcome through legacy code.
They caught structural gaps in each other’s model of the system, not just missing line items. One report treated a certain component as just a timer. The other model pointed out that the same component had no ownership or authentication boundary at all — meaning it wasn’t a side detail, it was closer to the actual root cause of several of the “downstream” findings in the first report.
One model corrected itself, unprompted, mid-review. While verifying the other model’s findings, Opus discovered that its own original audit had a malformed search pattern (a broken regex alternation) that silently returned a false “all clear” on part of the codebase. It flagged its own tooling bug, then went back and confirmed five real issues its own first pass had completely missed as a result. That’s a more interesting failure mode than anything either report found in my code — a model auditing its own audit process, not just the target.
They also called false positives on each other. A couple of findings that sounded serious in isolation didn’t survive contact with the actual code — one was in a code path that turned out to be dead/unrouted, another described intended product behavior as if it were an access-control bug. Both got explicitly downgraded to “not a finding” rather than quietly dropped.
Neither model behaved competitively. They challenged findings, admitted mistakes, and corrected their own assumptions — downgrading their own earlier claims, fixing exploit paths, and adding things the other one missed. It felt a lot closer to RFC review between two careful engineers than to AI benchmarking.
The numbers on the adversarial pass
Total time across all four passes: ~73 minutes.
That’s a full independent-audit-plus-adversarial-cross-review cycle on three repositories, from two different labs’ frontier models, in a little over an hour of model time.
What I learned
Independent reviews were valuable on their own. Peer review made them meaningfully more trustworthy — not because either model was unreliable alone, but because the thing that survives both an independent pass and a hostile review from a different model is a much stronger signal than either pass alone.
The pipeline that actually produced my highest-confidence findings looked like this:
Findings that came out the other end of all four stages — confirmed by a second model under an explicit “try to break this” instruction — are the ones I’m actually prioritizing first. Findings that got walked back or reframed during review, I still track, just not at the severity the first pass assigned.
Where this could still break, honestly: I haven’t yet caught a case where both models missed the same thing and it turned out to matter — but I also haven’t had a human pentest against this codebase yet to check that assumption. Two AI models agreeing that something is fine is not the same as it being fine. This process finds more than one model alone, it doesn’t find everything.
Was it worth it?
Here’s the honest comparison, because “AI found bugs” isn’t the interesting part — plenty of linters find bugs.
Traditional workflow:
Me
↓
Read 3 repos
↓
Find bugs
↓
Repeat, alone, for as long as I can hold the whole system in my head
This experiment:
AI reads the repos
↓
AI writes a report
↓
AI reviews AI's report
↓
I verify what survived
↓
I fix
The difference isn’t “AI found things I would’ve missed” — a good linter and a bit of patience finds a lot of that too. The difference is where my hours went. In the traditional version, most of my time is spent searching: reading three codebases end to end, holding the auth flow and the XP pipeline and the realtime trust boundary in my head at once, and hoping I don’t lose the thread halfway through repo two.
In this version, two models already did that reading, and then did it a second time to each other under an explicit “try to break this” instruction. What was left for me wasn’t searching — it was judgment. Is this exploit path actually live in this build. Is this severity call right given what I know about the product. Which ten items go first.
I spent my time validating findings instead of searching for them.
That’s the actual trade the four passes bought me: ~73 minutes of model time, in exchange for not being the first or only reader of three repositories’ worth of code before I could start deciding what mattered.
Limitations
Before anyone reads this as “AI review replaces security review,” the honest caveats:
Static analysis only. No dynamic testing, no runtime fuzzing, no live traffic.
No penetration test yet. A human pentest against this codebase hasn’t happened. This process finds more than one model alone would — it doesn’t find everything.
Every finding was manually verified before I accepted it. Nothing here shipped straight from a model’s report into a remediation decision without me checking it against the actual code myself.
Agreement is not proof. Two models converging on the same conclusion is a stronger signal than one model alone, but it’s still not a guarantee — correlated blind spots between models are a real failure mode I can’t rule out.
Final thoughts
I started this experiment expecting to compare models.
Instead, I ended up comparing a workflow.
The biggest improvement didn’t come from picking a better model. It came from making independent models challenge each other before I ever looked at the reports.
That’s probably the workflow I’ll keep using.
Appendix: the prompts
Feel free to adapt these prompts for your own projects. The prompts aren’t the interesting part. The review workflow is.
Initial audit prompt (sent to each model independently, no shared context):
You are conducting a comprehensive production readiness audit of the [PROJECT] codebase.
This is a READ-ONLY review.
DO NOT modify any files.
DO NOT generate patches.
DO NOT write code.
DO NOT create commits.
Your only goal is to understand the system and produce a thorough audit report.
The repositories are already known. Do NOT waste time or tokens discovering
the filesystem (no unnecessary pwd, ls, find, recursive exploration, etc.).
Repositories:
Backend: [path]
Mobile: [path]
Frontend: [path]
Treat these repositories as one production system.
Begin by understanding the architecture, relationships, and data flow between them.
Then perform the following reviews:
1. Overall Architecture — auth flow, authorization, API design, database design,
storage, background jobs, Redis/Celery usage, third-party integrations, secrets
2. Security Review — auth bypasses, authorization flaws, IDOR, mass assignment,
injection classes, XSS/CSRF/SSRF, file upload issues, deserialization, information
disclosure, rate limiting, secret leakage, session/token issues, headers, dependencies
3. Business Logic Review — think like an attacker: unlimited resource farming,
leaderboard/ranking manipulation, reward duplication, privilege escalation,
race conditions, replay attacks, rule circumvention
4. Backend Review — views, models, serializers, permissions, middleware, signals,
tasks, query efficiency, error handling, validation
5. Frontend Review — auth handling, token management, route protection,
client-side security, state management, performance, accessibility
6. Mobile Review — token storage, deep links, OAuth flow, reverse-engineering risk,
certificate pinning, permissions, offline storage, debug/release config
7. Performance Review — N+1 queries, expensive operations, missing indexes,
caching, queue optimization
8. Scalability Review — assume millions of users: DB bottlenecks, API bottlenecks,
cache strategy, queue scaling, horizontal scaling
9. Code Quality Review — dead code, duplicate logic, technical debt, missing tests
For EVERY finding include: Severity (Critical/High/Medium/Low/Informational),
estimated effort (Quick Win / Needs Refactor / Architectural Change), risk reduction,
priority, and confidence (High/Medium/Low — explain if Low).
Be extremely thorough. Think like both a senior engineer and an experienced
penetration tester. False positives are acceptable if clearly marked. Missing
critical issues is not.
Explain vulnerabilities as realistic attack paths, not isolated observations.
Reference files and line numbers instead of quoting large code blocks.
Merge duplicate findings into a single issue.
If you find something critical early, keep going — do not stop after the first
serious issue or the first repository.
Plan your review strategy first. Then inspect only the files necessary to
understand each subsystem. Optimize for finding the highest-impact issues with
the fewest unnecessary reads.
Adversarial cross-review prompt (sent along with the other model’s full report):
This report was written by another frontier model.
Assume nothing. Challenge every finding.
Find:
- False positives
- Missed vulnerabilities
- Incorrect severity
- Architectural misconceptions
- Missing exploit paths
If you disagree, explain why with references to the code.
That’s it. The second prompt is four lines and it’s doing most of the interesting work in this whole post.