The five layers

A production AI agent is kept safe by five controls working together: human approval gates on anything irreversible or money-touching, scoped permissions so an agent can only reach the systems its job requires, hard spend caps that stop runaway model usage, an audit trail recording what the agent did and on what basis, and reversibility — no action taken that cannot be undone without a human decision.

This is the model built as TripWire for the Microsoft agentic-security hackathon, and it is what every client build inherits rather than an enterprise upsell.

The failure everyone worries about, and the one that actually happens

The imagined failure is an agent going rogue. The real failure is far more boring: an agent doing exactly what it was told, at volume, when the instruction was slightly wrong. Approval gates and spend caps catch the second case, which is the one that shows up.

A concrete example from my own systems: a chat endpoint without rate limiting is one scripted loop away from a large API bill. That limiter exists because of a real incident, not a hypothetical one.