Building an enterprise AI security program from zero

Enterprise AI needs a control point

Enterprise LLM adoption tends to arrive team by team rather than as a single rollout, and each team reasonably solves its own access problem first. Securing it centrally means giving that traffic somewhere sanctioned to go, and building the inspection, policy, and detection that a control point makes possible.

The instinct in that situation is to block. I think that is usually the wrong call: the business value is real, and a hard block tends to push the same traffic somewhere with less visibility. The program had to make the safe path the easy path.

What the design had to live with

• Regulated environment. The company operates under FedRAMP obligations, so anything introduced here had to hold up against that bar.
• More than one surface. Internal developer tooling and customer-facing product features have different threat models, different data, and different tolerance for latency — and the design has to anticipate the second before it exists.
• Adoption is the actual success metric. A control nobody routes through protects nothing, so friction had to stay low enough that the sanctioned path beat the unsanctioned one.
• Vendor landscape still forming. Few of the categories involved had a settled market, so decisions needed to survive being wrong about a vendor.

One path in, built to be inspected

The architecture is deliberately boring: consolidate LLM calls onto a single gateway, and put inspection in that path. Internal clients talk to a LiteLLM gateway rather than to model providers directly. The gateway is deployed and carrying traffic — it handles authentication, routing, and per-team attribution, which is what makes everything downstream possible.

Guardrails are the next layer in. I ran the evaluation, selected Onyx, and am rolling it out deliberately: inventory and logging first, enforcement second, once there is enough observed traffic to tune against. Request and response inspection are both in the target design — response-side matters more than it first appears, and I did not want it treated as a phase two.

The important property is that the control point is architectural, not procedural — it does not depend on anyone remembering a rule. The closer the gateway gets to being the only route that works, the less coverage depends on goodwill.

Governance closes the rest of the gap. An AI Acceptable Use Policy defines what may be sent where, backed by tooling review and required training, and it is in force today. Netskope enforcement against unsanctioned AI tools is specified, and awaiting the organisational approval that a change of that blast radius should require.

Tradeoffs worth naming

• Gateway over per-application SDKs. A shared gateway adds a hop and a dependency, but per-application integration would have made coverage a permanent negotiation. Centralising it made coverage the default and let controls ship once instead of per team.
• Buy, then extend. Running proofs of concept across AI and MCP gateway and guardrails platforms before committing was slower up front than building in-house, but it kept the program from carrying a bespoke inference proxy forever. The POCs also did double duty: they shaped the governance policy by showing which controls were actually enforceable.
• Flag before block. A new detection starts by flagging, not by blocking, and enforcement waits on a behavioural baseline. A false positive on a blocking control costs far more than a missed detection during tuning — the first bad block is what teaches people to route around the gateway.
• Policy and enforcement written together. An acceptable-use policy that says something different from what the technical controls actually do is worse than either on its own — so the policy refresh ships with the enforcement it describes, not ahead of it.

Where it stands

• A single gateway path for internal LLM traffic, deployed, attributed per team, and positioned so inspection can be enforced in it.
• Onyx evaluated and selected, deployed in a logging-first posture while the enforcement baseline is established.
• Prompt injection, data leakage, and shadow AI are a written threat model with owned coverage plans rather than theoretical risks on a slide.
• Engineering teams have a written AI acceptable-use policy and required training to reason against, in force today.

What I would tell someone starting this

The security engineering was the easier half. The harder half was organisational: getting a company to route through one path voluntarily, before there is any enforcement to point at. That is won by making the sanctioned path genuinely better — faster access, more models, no key management — not by citing policy.

The second thing: build the control point first and the detections second. A gateway with weak rules can be improved. Excellent detection with no chokepoint to run it in is a research project.

Worth saying plainly, since it shaped the rest: this started while my title still said cloud security. The AI security role came after the work did, not before it.

Details here are limited to what is approved for public disclosure. Happy to go deeper in conversation.