AI Gateway
New Features
A summary of what shipped in the latest AI Gateway release.
Extensibility via Policy Templates
- The AI Gateway now offers the same extensible policy framework as the AAM API Gateway
- Teams/apps can inherit policy templates and add custom code to the request chain
- Unlocks a wide range of custom use cases (custom routing, DLP, guardrails, etc.)
JWT Authentication
- The AI Gateway now supports JWT auth, along with the default API key auth
- The gateway can take different actions based on JWT claim values (budget, routing, authz)
Per-User Budgets
- Set spend/usage budgets scoped to an individual user, not just an app or team
- Typically a user sub, user ID, or session ID would be forwarded via header
- Ability to natively define unique budgets for specific users in the portal UI Coming Soon
Multi-Model Bundling
- A single gateway endpoint can front multiple models/providers
- Support for fallback/backup models for resilience and cost-based routing
- Smart routing based on semantic prompt match (see Smart Match below)
Smart Match (Smart Router)
- Classifies each prompt by complexity (low, medium, high) and intent, then routes it to the model you choose for that level
- Simple prompts go to cheaper, faster models; complex prompts go to the most capable ones
- Set a confidence threshold for routing, or run classification only while you evaluate the results
- Custom policies can read the classification to drive their own routing, caching, or guardrail logic
- Smart Router policy documentation
User Pools
- Pools (formerly Teams) group apps with shared policy templates and cascading budgets, and can be nested as sub-pools
- User apps give each person their own personal API key, so usage is tracked per individual instead of per shared app key
- A personal budget rule applies to every user in the app (e.g. $50/day, $100/month per person), with per-user overrides where someone needs more
- Budget alerts notify users and pool admins when a budget is approaching or exceeded Coming Soon
- SCIM provisioning to sync users and groups from your identity provider (e.g. Okta, Microsoft Entra ID, Google Workspace) Coming Soon
Support for New Models
- Native support for additional models including Bedrock Mantle (Vertex and Azure are coming soon)
- Custom providers are supported as well, as long as they are OpenAI-compatible
- See the full list of supported providers
Expanded Analytics & Observability
- Team/app cost, token, and request rollups: hierarchical visualization of spend and usage for the entire organization, individual teams, and apps/agents
- Model usage dashboards with drilldowns: breakdown by cost, tokens, and requests by model
- Cache hit reporting and other usage-pattern reporting
- Detailed request-level logging and audit trail: customizable per-request logs showing status code, route, model, latency, client geo/network info, tools called, the user prompt, etc.