Prefer a PDF? Download the deck

Workshop Agenda

• Intro (10 min)
• Part 1: AI Governance Fundamentals (20 min)
• Part 2: Tabletop Exercise (30 minutes)
• Part 3: Wrap-Up & Q&A (5 minutes)

AI Governance Fundamentals

What is AI Governance?

• “Bureaucracy and approval gates”
• “Something only regulated industries need”
• “A 50-page policy no one reads”

Goverwhat?

• Most organizations lack basic AI oversight
• No inventory, no review process, no accountability
• Failures aren’t edge cases: they’re systemic and recurring
• No governance = no way to control exposure, liability, or impact

Common Organizational Pitfalls

Core AI Governance Components

• Develop comprehensive AI policies
• Assess and monitor AI risks
• Ensure accountability and documentation.
• Align with regulatory frameworks.

AI Lifecycle: CRISP-ML(Q)

Governance ≠ MLOps

Governance

• Determines the what and why for responsible AI.
• Ensures ethical, legal, and compliant AI use.
• Directs MLOps to build trustworthy AI systems.

MLOps

• MLOps focuses on how to deploy AI efficiently.
• Streamlines AI lifecycle for speed and reliability.

Governance ensures that MLOps is doing the right things, not just efficiently. They’re complementary.

Frameworks Overview

• NIST AI Risk Management Framework (RMF)
• EU AI Act
• GDPR
• ISO/IEC 42001 (AI Management Systems)
• US AI Governance Landscape

NIST AI RMF Overview

• Purpose: Voluntary, risk-based framework for trustworthy AI
• Key Functions: GOVERN, MAP, MEASURE, MANAGE AI risks
• Focuses on trustworthiness characteristics for reliable AI
• Addresses unique AI risks, like data and complexity issues

EU AI Act: Key Details

• World’s first binding legal framework for AI
• Focuses on AI and data protection, complementing GDPR
• Categorizes AI systems into four risk tiers
• High-risk AI requires strict rules and assessments
• Mandates robust risk management and data governance

GDPR & AI: Key Principles

• Applies to AI processing EU residents’ data.
• Reinforces strong data protection principles.
• Mandates lawfulness, fairness, and transparency.
• Requires data minimization and accuracy.
• Emphasizes accountability and human oversight.

ISO 42001: AI Governance Standard

• First international AI management system standard
• Ensures ethical and transparent AI development
• Covers entire AI system lifecycle
• Enhances credibility and mitigates AI risks
• Builds trust and provides competitive advantage

2026 US AI Governance Landscape

• Federal actions formalize NIST AI RMF for trustworthy AI.
• States rapidly adopt AI, establishing governance structures.
• State initiatives focus on impact assessments and responsible AI use.

AI Governance Ownership

• Senior leadership sets AI governance direction and accountability.
• Cross-functional teams ensure enterprise-wide AI policy integration.
• Providers and deployers have distinct compliance responsibilities.
• Organizations are accountable for third-party AI risks.

AI Approval Workflows: Key Steps

• Executive leadership drives AI governance & accountability.
• Cross-functional teams identify & assess AI risks early.
• Rigorous testing verifies technical quality.
• Human oversight & secure design principles are crucial.
• Continuous monitoring ensures ongoing improvement & compliance.

AI Governance: Key Takeaways

• AI governance requires enterprise-wide accountability.
• Adopt a proactive, risk-based AI approach.
• Leverage frameworks for continuous AI adaptation.

Checklist: Is Your AI Governable?

• Develop a comprehensive model inventory
• Include Model Cards, Data Cards, and AI-BOMs
• Conduct thorough risk analysis for AI systems
• Gather forensic evidence for incidents
• Implement robust logging and response mechanisms

Data Provenance

• AI data provenance documents the data lifecycle.
• Key aspects are origin, processing, and training.
• It ensures transparency, accountability, and compliance.
• Provenance manages risks like bias and privacy.
• It is critical for trustworthiness and human oversight.

AI Bill of Materials

• Do any of our models come from near-peer adversaries?
• Is any of our AI built on legacy models that are no longer supported?
• Do we have the legal right to use these models and datasets in our industry?
• Does any of our AI ship with software vulnerabilities?
• Where’s the complete inventory of all our models and datasets?

Governance Board Setup

• Diverse, cross-functional experts should form the internal AI Board.
• Establish a review cadence for AI policies.
• The board needs authority to make and overturn decisions.
• Senior leadership drives accountability for AI systems.

Cultural Enablement

• Governance does not equal bureaucracy.
• Effective communication fosters good AI governance.

Culture eats AI governance for breakfast. Make it easy to do the right thing.

MITRE AI Maturity Model

Thank you

Looking back at 2025, it wasn’t just our first year; it was a year of establishing the “Lab” in Fire Mountain Labs through intensive research, global collaboration, and community engagement.

Advancing the Science of AI Security

Our commitment to technical excellence took us to the most prestigious research venues in the world. We spent 2025 tackling the most complex challenges in the field:

• Operational Risk: We presented on the operational risk modeling of mission-critical AI systems at the NATO IST-210 Research Symposium.
• Threat Modeling: At CAMLIS, we shared our research on offensive security concepts and applications for AI systems, engaging with the community on how to red-team machine learning systems effectively.
• Systems Theory: At BSides Las Vegas, we explored a systems-theoretic approach to securing military AI using STPA-Sec, looking at how entire systems fail under pressure.
• RAG & LLM Defense: At SPIE DCS, where our CTO Josh served as Conference Chair, we published foundational papers on adversarial threat vectors in Retrieval-Augmented Generation (RAG) systems.

The Evolution: From Security to Holistic Maturity

As 2025 progressed, our work revealed a deeper truth: you cannot have sustainable AI security without organizational maturity. While we began with a sharp focus on the “break and fix” of AI models, we realized our partners needed a roadmap for the long haul.

We have expanded our practice to treat AI security as a core component of a broader AI Maturity framework. By grounding our research and advisory services in respected benchmarks like the MITRE AI Maturity Model, we are moving beyond reactive patching toward building resilient organizations.

From Shadow AI to Governance (AI Enablement)

One of our partners was facing “Shadow AI” sprawl across their engineering teams. They had the capability but lacked the guardrails. We helped them move from scattered experimentation to a structured AI Enablement program. By identifying their specific business objectives and designing a customized AI security service offering, we didn’t just lock down their models; we gave their teams a secure path to innovate.

Drawing Down the Risk (AI Risk Assessment)

For a client in a high-stakes regulated industry, the fear of “prompt injection” and model evasion was stalling their entire GenAI roadmap. We addressed this by conducting a comprehensive AI Risk Assessment that went beyond a standard pentest to map their entire AI supply chain, from training data to model endpoints, identifying high-impact vulnerabilities within their specific RAG architecture. Today, we translate that roadmap into action by helping clients balance defense and growth; we conduct rigorous security audits to measurably draw down risk while simultaneously driving AI Enablement strategies that unlock business value and establish a “paved path” for secure innovation.

From Analyst to Architect (AI Maturity Workshop)

While many organizations focus on the technology, we’ve seen that one of the biggest bottlenecks to AI is the skills gap. To solve this, we designed a series of AI Maturity Workshops designed for SOC analysts, auditors, and assessors.

In one session, we worked with a team of seasoned SOC analysts who were working to integrate new AI-driven technologies into their existing practice. Rather than just a standard tool training, we led them through a hands-on AI Governance in Action tabletop exercise. We simulated real-world AI incidents and helped them apply the NIST AI RMF and MITRE ATLAS frameworks to their specific environment. Our workshops have enabled our client to transition from being “AI-skeptics” to “AI-literate” defenders, equipped with a practical playbook to triage, investigate, and mitigate AI-specific threats.

But maturity isn’t just about frameworks; it’s about people. This year, we solidified this by partnering with AI CERTs® to become an Authorized Training Partner. By combining our deep security expertise with global certification standards, we are empowering the workforce to manage AI risk as a professional discipline, not an afterthought.

Community & Conversation

Research only matters if it reaches the people building the systems. Throughout 2025, we prioritized sharing our findings:

• Education: We led hands-on workshops on AI Governance at BSides and sponsored talks with the Cloud Security Alliance (CSA).
• Thought Leadership: Our AI Maturity Webinar series and the SD ISSA “Hot AI Summer” panel allowed us to define what a “mature” AI-enabled enterprise actually looks like.
• Interviews: We shared our vision on the Profectory Voice of Growth and Help Net Security podcasts, and sat down with the Federal News Network to discuss securing AI systems.

The Road Ahead

2025 was about the spark, establishing Fire Mountain Labs as a leader in AI assurance and organizational maturity. As we move forward, our mission remains the same: ensuring that the AI revolution is built on a foundation of security, not just speed.

Thank you to everyone who joined us in 2025. The fire is just getting started.

AI systems are probabilistic rather than deterministic. They do not just have bugs. They have unique weaknesses and failure modes like data poisoning, model theft, and prompt injection that standard firewalls and endpoint protection simply cannot see. To stay ahead of these threats, organizations must move from a purely defensive posture to a deliberate offensive strategy.

This approach is often visualized as an “Inverted Pyramid” which starts with broad assessments and narrows down to realistic, full-scope attacks.

Here is how to break down the three essential layers of offensive AI security.

1. Vulnerability Assessment: The “Health Check”

The Goal: Breadth and Visibility.

The Approach: Automated scanning and inventory.

Before you can secure your AI, you need to know what you have. A Vulnerability Assessment builds your AI Bill of Materials (AI BOM). It catalogs your models, training datasets, and third-party libraries while scanning them for known misconfigurations or outdated components.

Think of this as good hygiene. It does not necessarily prove an attacker can break in, but it highlights the unlocked doors and open windows that make it easy for them to try.

• What it catches: Unencrypted datasets, exposed API keys, outdated ML libraries, and shadow AI usage.

2. AI Penetration Testing: The “Stress Test”

The Goal: Depth on specific targets.

The Approach: Active exploitation in a controlled environment.

While vulnerability assessments look for theoretical issues, penetration testing proves they are real. In an AI context, this involves actively trying to break specific components, such as a chatbot’s safety filter or a model’s inference API, using specialized tools and tactics.

Testers use fuzzing (sending random data) and adversarial inputs to see if they can trick the model into misbehaving.

• What it catches: Prompt injection (jailbreaking), model theft (extracting the model via queries), and bias or hallucinations in output.

3. Red Team Engagements: The “War Game”

The Goal: Full-spectrum realism.

The Approach: Simulated adversarial attack on the entire system.

This is the pinnacle of offensive security. A Red Team engagement is not just checking a box. It is a simulation of a real-world adversary. These experts do not just look at the code. They look at the people, the process, and the technology together.

They might try to steal your training data by socially engineering a data scientist or use physical access to poison a model. The goal is to verify if your blue team (defenders) can detect and stop a sophisticated attack in progress.

• What it catches: Systemic process failures, blind spots in monitoring, and complex kill chains that automated tools miss.

The Strategy: Build, Attack, Defend

Offensive security does not exist in a vacuum. It is part of a continuous loop of improvement. We call this the Build-Attack-Defend triangle.

• Yellow Team (Builders): Develops the AI.
• Red Team (Attackers): Exposes the flaws.
• Blue Team (Defenders): Detects and blocks the attacks.

At Fire Mountain Labs, we’ve been hard at work analyzing the new adversarial threat surfaces introduced by RAG’s reliance on dynamic, mutable data sources. Today, I’m excited to share our latest research, “Adversarial Threat Vectors & Risk Mitigation for RAG Systems,” which dives deep into how bad actors can exploit every stage of a RAG pipeline and, just as importantly, how defenders can stop them.

Why RAG Needs a Security‑First Mindset

Unlike LLM deployments, RAG architecture depends on ingesting, indexing, and retrieving documents in real time. This flexibility comes with a trade‑off:

external data = external risk.

An attacker who sneaks malicious content into your vector store could cause an LLM to hallucinate false facts, leak sensitive information, or even take unwanted actions if your system is hooked into downstream workflows. Conversely, poorly sanitized user queries might trick the model into spilling internal prompts or proprietary IP.

As RAG adoption rises, these adversarial vectors are no longer “interesting research topics”: they’re table stakes for any organization putting RAG in production. If you care about data integrity, regulatory compliance, or simply keeping your users safe, it’s time to build risk controls specifically for RAG.

Spotlight on Key Adversarial Vectors

Data‑Poisoning Attacks

• Document Ingestion Poisoning: Attackers can slip malicious payloads into your ingestion pipeline: PDFs, Word docs, or even scraped web pages that contaminate the vector store. Over time, poisoned vectors skew retrieval results and lead your LLM to generate misleading or harmful outputs.
• Retriever/Index Poisoning: By crafting adversarial embeddings (for example, inserting subtly malformed text or hidden code), an attacker can pollute similarity searches and either suppress relevant documents or surface malicious ones.

Prompt Injection & Information Leakage

In a RAG setup, every user query is combined with retrieved context before hitting the LLM. A cleverly constructed prompt can hijack this process:

• Force the model to reveal PII or internal system prompts.
• Coerce the LLM into performing unauthorized tasks (e.g., “Summarize internal API keys.”).
• Stage membership inference attacks by probing the embedding space.

Configuration & Supply‑Chain Exploits

• Unvetted data sources like public Git repositories, shared network folders, or downstream APIs can introduce rogue content.
• Weak access controls on your retriever endpoint (e.g., no rate‑limiting or API throttling) open the door for automated probing and extraction of proprietary embeddings.

Building the RAG Defensive Stack

We framed our mitigation strategy around a layered “AI Security Pyramid of Pain.” Each tier corresponds to controls that progressively increase the cost and complexity for adversaries, forcing them to work harder to bypass your defenses.

Learn more about The AI Security Pyramid of Pain.

Data Integrity (Base Layer)

• Input Validation & Sanitization: Rigorously scan every document before ingestion. Block malformatted files, strip suspicious metadata, and verify source authenticity.
• Access Controls: Enforce strict IAM policies around document ingestion APIs and retriever endpoints. Treat your vector store like any other sensitive database.

AI System Performance Monitoring

• Embedding‑Space Audits: Continuously calculate embedding distributions and flag sudden shifts in similarity clusters. Visualize drift over time to spot mass poisoning.
• Quality Checks: Routinely sample LLM outputs against golden queries to detect anomalous behavior.

Adversarial Tool Awareness & Defense

• Awareness of Available Tools: Recognize that attackers can leverage publicly available exploits and test suites (e.g., klipper, OpenAI Red Team tools) to probe your RAG pipeline.
• Defense via Simulation: Use the same tools to run regular “attack drills” against your ingestion and retrieval components. By understanding exactly how adversaries operate, you can identify weaknesses before they’re exploited.

Adversarial Inputs Hardening

• Adversarial Training: Augment your LLM’s fine‑tuning dataset with simulated poisoning and prompt‑injection examples. Teach the model to “ignore” or gracefully refuse suspicious inputs.
• Prompt Sanitization: Pre‑process user queries to strip or neutralize common injection patterns (e.g., overriding system prompts).

Data Provenance & Traceability

• Immutable Logs: Record every ingestion event, including source metadata (timestamp, uploader ID, file hash). If something goes wrong, you can backtrack exactly when and how poisoned content entered the system.
• Digital Signatures: Require cryptographic signing for any high‑value document (e.g., internal policy PDFs). Reject unsigned or expired signatures.

Tactics, Techniques, and Procedures

• Red‑Team Exercises: Run routine penetration tests that simulate real‑world adversaries: insert poisoned docs, launch membership inference probes, and pressure‑test your retrieval pipeline.
• Incident Response Playbooks: Define clear runbooks for “when poisoning is detected” or “when a prompt‑injection leak is confirmed.” Include forensic steps, stakeholder notifications, and recovery procedures.

Looking Forward

Securing RAG is not a “one and done” exercise. As attackers innovate, RAG architectures will continue evolving: new embedding algorithms, hybrid retrieval‑generation pipelines, and tighter regulatory requirements. Building a layered defense today buys you the flexibility to adapt tomorrow.

If you’re architecting a RAG system, or already have one in production, I encourage you to dive into our full paper, where we detail each threat vector, quantify inherent and residual risk, and present proof‑of‑concept detections.

The industry is obsessed with AI security. Every vendor, consultant, and CISO is racing to bolt on security solutions, patch vulnerabilities, and deploy red teams. But here’s the uncomfortable truth:

AI security alone is insufficient right now. Not because it’s unimportant, but because we’ve skipped the most crucial step: AI Safety.

We’re spending valuable resources protecting systems that were never designed to be safe in the first place. Let me emphasize this point: Securing unsafe systems is fundamentally misguided.

Why does this matter to you?

Because when safety is treated as an afterthought by the companies building AI, it’s your organization, your brand, reputation, and bottom line, that suffers. The vendor’s risk becomes your risk.

First, Build the Foundation

AI Safety is about designing controls to prevent harmful outcomes at their source. It begins with verifiable evidence that your AI system’s controls are in place, effective, and protected throughout its lifecycle. That’s the necessary foundation. Before hiring a red team or worrying about adversarial attacks, ask:

• Does your AI system have robust alignment mechanisms in place: design features that ensure it pursues only intended goals?
• Have you rigorously defined what constitutes a harmful outcome in your specific context?
• Are there comprehensive guardrails to prevent harmful decisions?

No? Be careful, you’re entering territory where security investments become superficial and wasteful, like installing an expensive alarm system on a house with no walls.

Security’s True Purpose

You can’t secure away bad design. AI Security protects the controls that manage AI risk. When your AI system is properly aligned and safeguarded, security ensures those safety mechanisms cannot be bypassed or defeated. Attempting to secure an inherently unsafe system is merely addressing symptoms while ignoring the disease.

• Without foundational safety mechanisms, security measures protect nothing of value.
• When your AI system is properly aligned and safeguarded, then security ensures those safety mechanisms cannot be bypassed or defeated.
• Attempting to secure an inherently unsafe system is merely addressing symptoms while ignoring the disease.

When Security Can’t Catch Safety Failures

Here’s an example we are all too familiar with: even in highly secure environments, large language models have produced harmful or misleading outputs, without any external breach.

In many cases, the systems were deployed behind firewalls, with access controls and monitoring in place. Yet when users interacted with the models, they received hallucinated content, sensitive internal references, or confidently wrong responses, sometimes even formatted as if they were authoritative company documents.

No one broke in. No code was compromised. The problem wasn’t a lack of security, it was the absence of safety constraints and alignment controls.

Security didn’t fail. Safety was never built.

This is the reality facing organizations deploying generative AI: even with traditional protections in place, AI systems can behave in unexpected, uncontrolled, and costly ways, if safety is treated as an afterthought.

Build AI Systems Worth Securing

Regulators and stakeholders aren’t asking if your AI system is “secured”: they’re asking if it’s safe, reliable, and verifiable. They want to know that your controls exist, they work, and they can’t be easily bypassed.

Securing unsafe AI is a waste. Build safe systems and products first. Prove it. Then protect them.

We are living through an AI gold rush. Organizations across every sector are racing to adopt powerful generative and predictive AI technologies, eager to unlock new capabilities, automate decision-making, and stay ahead of the competition. Yet, beneath the promise of rapid innovation lies an unsettling reality: many AI systems marketed today lack even basic safety assurances.

Why does this matter to you, as a decision-maker or risk professional?

Because when safety is treated as an afterthought by the companies building AI, it’s your organization (your brand, reputation, and bottom line) that suffers. The vendor’s risk becomes your risk.

A Troubling Reality: AI Safety Grades Exposed

Recently, an independent assessment by the Future of Life Institute graded leading AI companies on their safety practices. The results are alarming.

No vendor scored higher than a “C.” Industry giants (those at the forefront of AI development) earned predominantly “D” grades or worse. Several failed outright.

These aren’t obscure companies; these are household names providing the AI technologies your enterprise is likely evaluating or already deploying. The assessment evaluated key safety dimensions critical for business continuity and compliance:

AI Safety is No Longer Someone Else’s Problem

AI safety is now an organizational imperative, and the responsibility is landing on the shoulders of those closest to real-world impact.

• CISOs are the last line of defense against unsafe or unvetted AI tools reaching production.
• Risk assessors must spot weaknesses in AI governance before they become business liabilities.
• IT and procurement leaders are expected to push beyond vendor claims and demand provable safeguards.
• Compliance officers are being called to ensure AI outputs align with regulatory obligations, not just internal policy.

Why You Shouldn’t Trust Vendor Claims Blindly

Here’s the uncomfortable reality: AI vendors have strong incentives to downplay risks and emphasize benefits. Without independent oversight or third-party validation, assurances of safety, alignment, and responsible AI practices remain mere promises.

You can’t afford to rely exclusively on vendor claims when your organization’s reputation and operational effectiveness are at stake. Always demand evidence to substantiate assertions: trust must be backed by thorough verification.

Start asking your vendors pointed questions:

• Can you show detailed evidence of your AI risk assessments?
• Have independent audits or safety validations been performed?
• What red-teaming or adversarial testing has the AI undergone?
• Do your AI systems meet emerging compliance standards?

If answers are vague, unavailable, or incomplete, proceed cautiously.

Remember: it’s your reputation, compliance posture, and competitive edge at stake, not theirs.

The Path Forward: Independent AI Assurance

Here’s the good news: embracing AI doesn’t have to mean exposing yourself to unmanaged risk. It simply means adopting a disciplined approach to validating AI safety. Leading companies already rely on independent AI assurance as a critical step in their technology adoption processes, much as they do for cybersecurity and financial controls.