What is an AI security audit? A buyer's guide
An AI security audit tests a product the way AI models now attack it. What it covers, how it sits beside a pentest and a scanner, and what to ask a provider.
An AI security audit tests a software product against the techniques AI models now use to find and exploit weaknesses: mapping every endpoint, trying every object ID and chaining small gaps into real access, at machine speed. It ends with findings ranked by what they would cost the business and with the fixes, and a good one leaves behind written rules the product can be tested against on every release after it.
This guide is for the CTO, CISO or founder deciding whether they need one, and how to tell a real audit from a renamed scan. It covers what changed in 2026, what an audit should test, how it sits beside a penetration test and a scanner, and what to ask before signing anything.
The short version
- →An AI security audit assumes the attacker has an AI model and an ordinary account, and tests the product from there.
- →It should reach six areas: identity and access, spend and resource abuse, data exposure, automation and fraud, availability, and AI-written and agentic code.
- →It sits beside a penetration test and a scanner. Each answers a different question, and most teams with something to protect use more than one.
- →The deliverable that lasts is the rule set: plain-English rules the product has to hold, each with a check that can run again.
- →Ask any provider what access they need, how they rank findings, who writes the fixes and what you keep afterwards.
Why the audit changed in 2026
Three public findings this year moved the cost of an attack. In April Anthropic reported that Claude Mythos Preview found a 27-year-old flaw in OpenBSD in a model run that cost under $50. The same month Mozilla fixed 271 Firefox bugs in two weeks with one model. In September NIST and Anthropic measured GLM-5.3, a model anyone can download, close behind Mythos at building exploits. Finding and exploiting a weakness used to take an expert weeks; it now takes a model run. An audit built on the old cost tests for an attacker who has moved on.
What an AI security audit should cover
MATT groups the attack patterns it tests into six categories. Whoever runs your audit, these are the areas to check it reaches. Each example rule is one plain-English sentence with a check behind it.
| Area | What gets tested | Example rule |
|---|---|---|
| Identity & access | Who can read or act on whose data: sessions, roles, password resets | A user must never be able to read or act on another user’s object. |
| Spend & resource abuse | Metered features, credits and anything that costs money per call | A metered operation must be reachable only through the flow that meters it. |
| Data exposure | What responses, exports and error pages reveal | A response must contain only the fields the screen displays. |
| Automation & fraud | Sign-up, reset and checkout flows a script can work at scale | A recovery or signup flow must not reveal whether an account exists. |
| Availability | Requests that cost the server far more than they cost the sender | Every input must have a documented upper bound, enforced server-side. |
| AI-written & agentic code | Generated handlers, agents, their tools and prompt injection | Content an outsider controls must never become an instruction the agent acts on. |
Each category has worked examples in the attack library, and the rules shows how a finding becomes a rule.
How it sits beside a pentest and a scanner
These are different tools for different questions.
| Approach | Best at | Runs | What you keep |
|---|---|---|---|
| Penetration test | An expert’s judgement on a fixed scope; often required by compliance | A set window, often yearly | A report of what was found in that window |
| Automated scanner | Known vulnerabilities and misconfigurations, at volume | Continuously or on a schedule | Matches against known signatures |
| AI security audit | Logic and access flaws a model would chain, tested as a real user | Once, as a deep first pass | Ranked findings, the fixes and a written rule set |
| Continuous assurance | Keeping each written rule true as the product changes | On every release | Each rule’s current status: holding or broken |
A penetration test is still worth doing. A skilled tester finds things no rule yet covers, PCI DSS requires one and SOC 2 auditors and enterprise reviews often ask for one. A good audit makes the next test more useful by handing the testers the rules to check, and our pentest scope builder turns those rules into a scope document any testing firm can quote from.
Questions to ask a provider
- →What access do you need? A first audit can start read-only: a test login and read access to the code, with production data, secrets and customer records out of scope.
- →Do you test as a real user, with more than one account? The most damaging flaws, such as one customer reading another's records, only show up across two accounts or two tenants.
- →How are findings ranked? Ask for the business outcome of each one, a bill, a headline or an outage, beside its severity score.
- →Who writes the fixes? Fixes written and reviewed with your engineers close a finding faster than a report your team has to interpret alone.
- →Do you cover our AI features and agents? Prompt injection, tool permissions and generated code fail in their own ways and need their own tests.
- →What do we keep afterwards? Ask for the rules in plain English, each with the check behind it, so the next release can be tested against them.
- →What happens after the audit? Ask whether the rules are re-tested as the product changes, and how you would see one break.
What you should have at the end
- →A findings report ranked by bill, headline or outage, with the business outcome named for each finding.
- →Fixes in your code, written with and approved by your engineers.
- →A rule set: one plain-English sentence per rule, each with an executable check.
- →A control map showing which SOC 2 and ISO 27001 controls each rule supports, for audits and vendor questionnaires.
After the audit: keeping the rules true
A product changes every week, so an audit describes one day. Continuous assurance re-proves each rule against a staging or production deployment on every release, with read-only access, and reports each one as holding or broken. Your pipeline and your release decisions stay with your team. This is the practice we call ExploitOps: declare the rules, observe what changes, prove each rule still holds, fix what breaks and remember it on every release after.
How MATT runs one
It starts with a 15-minute ExploitOps discovery call with a co-founder, to understand your product and whether there is a real need. If there is, the research team scopes a paid ExploitOps Review with you. The audit starts read-only, with nothing installed and no customer data leaving your environment, and every fix in the first engagement is written with your team. Assurance then runs on your release cycle.
Questions
What is an AI security audit?
A test of a software product against the techniques AI models now use to find and exploit weaknesses, run as a real user with ordinary accounts. It ends with ranked findings, the fixes and a written set of rules the product can be re-tested against.
How is an AI security audit different from a penetration test?
A penetration test is an expert's view of a fixed scope in a set window, and compliance frameworks often require one. An AI security audit tests at the speed and breadth of an attacker with an AI model and leaves behind rules that can be re-tested on every release. Most teams benefit from both.
Does an AI security audit need production data?
A first audit can run with a test login and read access to the code, with production data, secrets and customer records out of scope.
What does an AI security audit cover?
Six areas: identity and access, spend and resource abuse, data exposure, automation and fraud, availability, and AI-written and agentic code, including prompt injection and agent permissions.
What happens after an AI security audit?
The rules it produced are re-proven against the live product on every release and reported as holding or broken, so the audit's answer stays current as the product changes.
Do we still need a vulnerability scanner?
Yes. A scanner matches known vulnerabilities and misconfigurations at volume; an audit looks for the logic and access flaws a model would chain together. They answer different questions.