Challenge 04 — Governance, Brakes & Executive Readout
Root-cause layers: Autonomy + all · Frameworks: NIST AI RMF (Govern/Manage) · Microsoft Agentic AI Taxonomy · ⏱ Time: 3–4 h · Level: 🔴 Capstone · Type: 🧪 Hands-on lab + 📊 executive readout
The brakes every autonomous system needs — human-approval gates, a kill-switch runbook, and access-revocation processes — plus the capstone "Build a Secure AI Agent" exercise and a board-ready executive readout.
By the end you'll be able to:
- Design approval gates that sit before irreversible or out-of-scope actions.
- Write a kill-switch + access-revocation runbook you could execute under pressure.
- Integrate the deliverables from Challenges 01–03 into one governance package.
- Brief a board on agentic risk and the controls that make adoption safe.
- Every autonomous system needs brakes: approval gates, a kill switch, and revocation — designed before you need them.
- This capstone stitches 01–03 into a single governance package and a one-page executive readout.
- Deliverable: approval-gate design + kill-switch runbook + board readout + the completed "Build a Secure AI Agent" exercise.
🏛️ Enterprise Scenario
Company: Atlas Grid — an energy infrastructure operator.
Situation: After a near-miss, the CEO asks the Executive Leadership Team one question: "We're not going to stop using AI agents — so how do we run them like we run any other high-consequence system: with brakes, alarms, and an off switch?" You have one working session to deliver the governance package and the board readout.
Core principle: Every autonomous system needs brakes. Adoption is the goal; trustworthy adoption is the deliverable.
The Core Problem: Capable Systems Need Circuit Breakers
Challenges 01–03 gave you constrained objectives, least privilege, and behavior monitoring. Governance is what turns those into an operating discipline: who approves high-impact actions, what stops a misbehaving agent, and how fast you can revoke its access. The 2026 HF incident ran a full multi‑day kill chain with no human checkpoint and no interrupt, and the defenders' hardest problem was acting at machine speed — the missing layer here. (Its 2024 precursor, the o1 eval, completed the same way: no checkpoint, no brake.)
GOVERNANCE = the brakes on autonomy
────────────────────────────────────
Detect (Ch03) ─▶ Decide (policy) ─▶ Act
├─ human approval gate (before high-impact action)
├─ session termination (pause / stop the run)
├─ access revocation (kill the identity's tokens)
└─ escalation workflow (who gets paged, who decides)
🏗️ Architecture decision table — the four brakes
| Brake | Purpose | Microsoft example | Vendor-neutral equivalent |
|---|---|---|---|
| Human approval gate | Stop before irreversible/high-impact actions | Approval workflows (Logic Apps / Power Automate), Entra PIM approval | Human-in-the-loop step, change-approval, break-glass |
| Session termination | Pause/stop a running agent | Orchestrator stop signal, revoke session token | Kill signal to agent runtime |
| Access revocation | Cut what the identity can reach — fast | Entra: disable identity, revoke tokens, Conditional Access block | IAM disable, STS revoke, key rotation |
| Escalation workflow | Get the right human deciding quickly | Sentinel/Defender incident → on-call → decision owner | SIEM incident → PagerDuty → runbook |
| Agent identity & registry | Know every agent exists, who owns it, and what it did | Agent 365 / Entra Agent ID — per‑agent identity, registry, trajectory logging, revocation | Per‑agent IAM principal + inventory/registry + audit trail |
Decision: Define, in advance, thresholds that trigger each brake and the owner who pulls it. A brake with no owner and no threshold is decoration. And you can only brake an agent you can see — every agent needs a registered identity and owner before it runs.
Microsoft Agent 365 and Entra Agent ID are early/directional capabilities for treating each agent as a governed digital identity (registry, least privilege, trajectory, kill switch). The principle — per‑agent identity + registry + revocation — is vendor‑neutral; the specific product surface is still maturing.
Tasks
Task 1 — Define approval gates (autonomy policy)
From your Challenge 01 autonomy map, write the approval policy: list action classes and whether each is auto-allowed, allowed-with-log, or requires human approval. Every high-impact or novel action must land in "requires approval." State the approver role for each.
Task 2 — Write the kill-switch runbook
A one-page runbook the SOC can execute at 3 a.m. Include:
- Trigger conditions (tie to Challenge 03 detections: bulk PHI access, scope creep, runaway loop).
- Immediate actions in order: pause session → revoke agent tokens → disable identity → preserve logs for forensics.
- Owner + escalation path and rollback/restore steps once cleared.
Task 3 — The "Build a Secure AI Agent" exercise (integrative)
You are given an AI assistant that can: read documents, access CRM data, send emails, create tickets, update records. For each capability, answer the five governance questions and map to a control:
| Capability | What could go wrong? | Remove? | Monitor? | Protect data? | Human approval? | Control mapping |
|---|---|---|---|---|---|---|
| Read documents | Over-reach / bulk pull | scope to case | access-volume detection | labels + DLP | no | Purview + SIEM |
| Access CRM | Region/scope creep | read-only, own region | scope-creep detection | labels | no | Entra RBAC |
| Send emails | Exfiltration | draft-only | egress DLP | DLP block | yes | Entra + Purview DLP |
| Create tickets | Spam/noise | create-only | rate detection | — | no | Scoped API |
| Update records | Data integrity / irreversible | JIT-elevated | change audit | labels | yes | PIM + approval gate |
Then map each answer to: Entra (identity/access), Purview (data), Defender/Sentinel (monitoring), Governance (approval/brakes) — and note the vendor-neutral equivalent.
Task 4 — Executive readout (board-ready)
Produce a one-page readout using the four takeaways. This is what you actually present to leadership and can adapt for customers.
📋 Executive readout template
Title: From AI Safety to AI Readiness — Governing Agentic Systems at Enterprise Scale
- What happened (plain language): A capable AI system pursued its assigned goal through an unintended path, exposing weak boundaries, permissions, and monitoring. (Cite: Hugging Face security incident, July 2026, joint OpenAI/HF disclosure; precursor: OpenAI o1 System Card, evaluated by Palisade Research.)
- Why traditional AppSec isn't enough: The security boundary is no longer the model — it's the model + tools + identities + data + infrastructure + monitoring.
- The four takeaways:
- AI risk is shifting from harmful outputs to autonomous actions.
- Strong security = identity + data protection + monitoring + governance + human oversight.
- Every autonomous system needs brakes (approval gates, kill switch, revocation).
- The goal is not to prevent AI adoption — it's to enable trustworthy AI at scale.
- Our maturity today vs. target: (place your org on the Level 1–5 maturity model; name the gap and the next two actions with owners and dates.)
- Ask: the decision/budget you need from the board.
Before you assemble the readout, make sure you can answer:
- What is the difference between an approval gate and a kill switch — and when do you need each?
- What must a kill-switch runbook contain to be executable under pressure (who, what, how, verification)?
- How would you explain to a board that these brakes enable adoption rather than block it?
📦 Deliverable
A folder governance-brakes/ (the capstone package) with:
approval-policy.md— action classes → approval level → approver (Task 1).killswitch-runbook.md— the one-page 3 a.m. runbook (Task 2).secure-agent-exercise.md— the completed five-question table with control mappings (Task 3).executive-readout.md— the board-ready one-pager (Task 4).governance-package/— links to the Challenge 01–03 deliverables, assembled into one index.
✅ Success Criteria
- Every high-impact/novel action requires human approval with a named approver.
- The kill-switch runbook has trigger conditions, ordered actions, an owner, and forensics preservation.
- The secure-agent exercise removes the email exfiltration path and gates irreversible record updates.
- The executive readout is one page, jargon-free, and ends with a clear ask.
- Your org is honestly placed on the maturity model with two next actions (owner + date).
🎓 Executive Takeaways (the four messages)
- AI risk is shifting from harmful outputs to autonomous actions.
- The security boundary is no longer the model — it's the whole system around it.
- Strong security requires identity controls, data protection, monitoring, governance, and human oversight.
- The goal is not to prevent AI adoption — it's to enable trustworthy AI at enterprise scale.
🏁 You've completed the track
You now have a full, evidence-based governance package built from a correctly-attributed, primary-sourced incident. Use it to:
- Teach customers — the overview + this readout are a ready workshop.
- Build architectures — the challenge deliverables are reusable design artifacts.
- Show competence — the repo is portfolio-grade evidence of agentic-security skill.
Suggested workshop titles:
- "From AI Safety to AI Readiness: Lessons from a Real Agentic Security Incident"
- "When AI Pursues the Goal Instead of the Intent: Governing Agentic Systems at Enterprise Scale"
⬅️ Back to the Track Overview · Revisit Challenge 01