Upskilling security professionals for generative AI

If you've spent any part of your career in security, you’re familiar with new technology arriving faster than the controls needed to secure it. Generative AI is the latest, and in some ways the most disorienting, example of this. 

While authoring my short course GenAI Security Foundations, I worked out something that I want to share here: The fundamentals we already know—such as injection attacks, supply chain risk, broken access control, privilege escalation, and the disciplines of audit and forensics—all are still relevant. They apply directly to AI systems.

What's different is the genAI architecture, and that's what we need to understand to be able to secure it.

Check out my course here. It’s aimed squarely at security professionals who understand security, but have little experience of genAI other than as a user.

The most useful generative AI security concept to understand is the context window.

A large language model has no persistent memory. Instead, imagine a whiteboard that holds everything the model can currently see: the prompts and responses so far, instructions from whoever built the system, and any documents that have been loaded in. Every time the model responds to a prompt from a human or an agent, it reads the whole whiteboard, generates an answer, and then stops.

The context window is the primary attack surface. If an attacker can influence what lands on that whiteboard, the attacker can influence what the model does. 

Interestingly, like most systems we secure, not everything on the whiteboard carries equal trust. There's a hierarchy: Training baked into the model like firmware is the most trustworthy, a system prompt sitting above the conversation comes next, and human or agent user input falls at the bottom, carrying the same level of untrustworthiness as any user-supplied data we've ever had to sanitize.

Add to this architecture Retrieval Augmented Generation (RAG), where organizations add their own document libraries to affect a model's answers, and agents, where a model can call tools to send emails, access the web, query databases, or execute code, and we have the full architectural picture of what you need to secure.

Once we have that picture, a lot of the attack surface maps neatly onto things that existing security professionals already understand. 

For example, prompt injection is unsanitized input finding its way onto the whiteboard. Conceptually, it’s close to SQL injection. Indirect prompt injection, where the malicious instruction is planted in a document or webpage for the model to retrieve later, behaves a lot like stored cross-site scripting. 

RAG poisoning is best thought of as a supply chain attack. The attacker is not compromising the model—it’s compromising the data it trusts.

And a compromised agent, with broad tool access, is functionally like a privileged service account that's been taken over. We know how to limit risks like this by limiting privileges to only what’s needed and adding separation of duties for high-risk actions. That separation could be between an agent and a human, or two agents who have different instructions or use different models.

There are three intrinsic behaviors of generative AI that don't look like anything we’ve come across before.

First, there’s opaque reasoning. Unlike traditional software, a model's decision-making isn't logged or reconstructable, which has implications for incident response and compliance. 

Data and instructions are no longer separate. Traditional architecture keeps these apart: A database doesn't execute what it stores, and a processor doesn’t read and execute random bits of memory.

A language model, however, can not reliably tell the difference between code and data intermingled in a prompt, which is what makes indirect prompt injection so dangerous.

The same attack, and the same defense, can behave differently each time they happen. That can affect assurance activities and really changes how we need to think about red teaming genAI systems. A simple one-time result might not provide the assurance we need. It also means we can never fully enumerate the attack surface.

Your existing ways of thinking as a security professional are a valuable foundation when it comes to securing genAI. But staying relevant will mean learning at a pace faster than most of us have had to before. 

GenAI capability, and the attack techniques targeting it, are moving faster than any technology shift I've seen in fifteen years in this field. The half-life of any technology we know feels like it’s decreasing more quickly than ever.

As an existing security professional, the knowledge and skills you have built up to today still matter. You just need to add new ones quickly. My advice is to start with the architecture, map what you already know onto it, and keep learning, ideally faster than you ever have before. 

At the end of the day, the most successful organizations are not the ones with the most experienced security teams—they're the ones whose teams learn the fastest. Hopefully my new course will be a good starting point for you and your teams.