AI/LLM Security Assessments

AI penetration testing, also called AI/LLM penetration testing, is a security assessment of an AI system or model and the infrastructure that supports it, such as the application it's embedded in, its retrieval pipeline, its agent permissions, or the APIs it calls. The goal is broad vulnerability coverage: finding as many exploitable weaknesses as possible, rather than achieving one specific objective undetected.

A GenAI architecture security review, also known as an AI model security audit, evaluates how an organization has implemented generative AI from a governance and process standpoint by assessing architecture, policies, and controls rather than actively testing the system for exploitable vulnerabilities. It typically covers model and tool deployment, policy review, AI usage discovery, and interviews with the teams responsible for AI systems. It results in recommendations for extending existing security processes to cover generative AI.

Credible AI security assessments typically align to the OWASP Top 10 for LLM Applications (prompt injection, sensitive data disclosure, system prompt leakage, excessive agency), the OWASP Top 10 for Machine Learning (model manipulation, data poisoning, model theft), the OWASP Top 10 for Agentic Applications (goal hijacking, tool misuse, privilege abuse), and MITRE ATLAS, a catalog of adversary tactics and techniques specific to AI systems.

Ask what the engagement is primarily designed to evaluate. Is the vendor looking for exploitable technical vulnerabilities in the software, APIs, infrastructure, permissions, and integrations supporting the AI system? Are they testing how the AI system itself behaves under adversarial pressure? Or are they simulating a real-world attacker pursuing an objective across the broader environment, where AI may simply be one part of the attack path?

Those answers help distinguish AI penetration testing from AI red teaming and traditional red teaming, regardless of what the engagement is called in the proposal.

Most engagements move through the same general phases: scoping and threat modeling to define which models, agents, and integrations are in play; automated and human-led testing across the model, application, agent, and infrastructure layers; validation of any findings that chain across layers to confirm real business impact; and reporting with prioritized, actionable remediation guidance. The exact steps flex based on whether source code is available, whether agents or retrieval-augmented generation are in scope, and whether testing is a one-time engagement or an ongoing program.

Testing is scoped and coordinated with the client's team ahead of time, with automated multi-turn testing rate-limited and, where available, run against staging or test environments rather than live production. Engagements are also often phased to match how a client's AI system is actually built, for example testing the model layer first and adding agent or retrieval-augmented generation testing once those components are deployed, rather than testing components that don't yet exist.

Beyond a single engagement, some organizations work with Bishop Fox on an ongoing basis, with a security team continuously triaging findings as their generative AI footprint grows. This tends to be more effective than one-time testing alone, since AI deployments change quickly and new findings surface as new features, such as agents, retrieval-augmented generation pipelines, or third-party integrations, come online.

Test across the full stack rather than just the chat interface, since the model, agent, application, and infrastructure layers each carry distinct risks. Align testing and controls to recognized frameworks, such as the OWASP Top 10s for LLM, machine learning, and agentic applications, and MITRE ATLAS. Treat retrieval-augmented generation sources, including documents, tickets, and knowledge bases, as untrusted input, since indirect prompt injection through those channels is a common bypass, and review source code where possible, since it materially improves what testing can find.

It depends on what you need to validate. If you need to identify exploitable technical vulnerabilities in the software, APIs, infrastructure, permissions, or integrations supporting an AI system, AI penetration testing is the right fit. If you need to understand whether the AI system can be manipulated into bypassing safeguards, exposing sensitive information, misusing tools, or otherwise behaving outside its intended boundaries, AI red teaming is appropriate.

Many AI systems benefit from both. Penetration testing evaluates the security of the technology supporting the AI system, while AI red teaming tests AI-specific behaviors and failure modes that conventional security testing may not uncover.

A GenAI architecture security review (commonly known as an AI model security audit) is largely a static evaluation: reviewing an AI system's architecture, configuration, training and deployment pipeline, access controls, and data handling against a governance or compliance standard, without necessarily trying to break it. AI penetration testing takes the opposite approach: it's active and adversarial, attempting real prompt injection, jailbreaks, instruction-hierarchy bypasses, and agent or tool misuse against the live system to confirm whether a theoretical weakness can be exploited. Most organizations benefit from both: an audit-style review to confirm the system is architected and governed soundly, and penetration testing to confirm it holds up against a real attacker.

Automated tooling is effective at covering known jailbreak patterns and documented injection techniques at scale, and it's necessary for tasks like multi-turn jailbreak testing that aren't practical to run by hand. On its own, it doesn't chain findings across layers, such as connecting a model-level weakness to an agent permission issue to an exposed cloud credential, and it doesn't judge real business impact. That combination of automated coverage and human-led judgment is what separates thorough AI/LLM security testing from a scan.

The risks that show up most consistently are direct and indirect prompt injection, sensitive data disclosure (user, tenant, or internal), guardrail bypass through multi-turn, encoded, or adversarial inputs, unsafe downstream actions triggered by AI-generated output, agent or tool misuse, and insecure storage of prompts, responses, and model provider credentials. Attackers exploit these by manipulating a model directly through crafted prompts, or indirectly through malicious content planted in a document, ticket, or knowledge base that gets pulled into the model through retrieval-augmented generation, then chaining the result into a downstream action, such as an agent with code-execution ability being manipulated into revealing its own cloud credentials. Mitigation starts with testing for these conditions specifically, since generic application testing won't surface prompt-based attacks, then remediating at the layer where the issue occurs.

Timelines depend on scope, objectives, and complexity, including how many AI-enabled applications are involved, whether agents or retrieval-augmented generation pipelines are in scope, and whether source code is available for review. A narrowly scoped assessment of a single chatbot takes less time than a full-stack evaluation spanning the model, application, agent layer, and underlying infrastructure.

Clients receive a report documenting each finding, its exploitation path, business impact, and recommended remediation, including any cross-layer attack chains that show how issues at the model, agent, application, or infrastructure layer combine into a larger risk. Reporting is typically paired with a findings walkthrough to align security and engineering teams, and retesting to confirm that remediation efforts are effective.

AI/LLM security testing isn't a compliance certification on its own, but it generates the kind of evidence auditors and regulators look for: documented testing against recognized frameworks, identified risks, and demonstrated remediation. For organizations navigating emerging AI-specific regulation, such as the EU AI Act, or existing data protection and security frameworks that now extend to AI systems, testing helps demonstrate that AI-specific risks are being assessed and managed as part of a broader security program.

Point-in-time testing captures what's true the day the assessment runs, but AI-enabled applications change quickly between cycles as new agents, retrieval-augmented generation sources, and third-party integrations get added, each introducing risk that a one-time report can't account for. Rather than relying on a monitoring tool alone, organizations get more durable coverage from a continuous security testing service that revisits AI systems as they evolve, catching new exposures as they're introduced instead of waiting for the next scheduled assessment.