Mosaic AI Gateway - Secure and Govern your AI

Introduction

Databricks is thrilled to announce the release of the Mosaic AI Gateway—a cutting-edge solution designed for unified access, centralized governance, and comprehensive observability into your generative applications. The AI Gateway is a highly scalable Enterprise API Gateway that allows organizations to efficiently manage their Large Language Models (LLMs) while accelerating outcomes through secure and governed AI, catering to both experimentation and production use cases.

Getting Started with Mosaic AI Gateway

To leverage the AI Gateway, organizations first need to enable it on their model serving endpoints, which can include existing endpoints.

Enabling Usage Tracking

The first step is enabling usage tracking, which creates a system table that provides insights into who is calling each endpoint and the token usage. This feature acts like admin privileges, allowing administrators to reference it when setting rate limits.

Enabling Inference Tables

Next, the feature for inference tables can be activated to manage incoming requests and outgoing responses at the account level. Organizations can specify their table location in Unity Catalog, ensuring that all payloads are automatically logged in a table that is governed alongside your model and other data.

Implementing AI Guardrails

The AI guardrails feature safeguards both inputs and outputs. Users can enable safety checks along with valid topics that can be processed by models. For instance, in a banking chatbot, opening a checking account might be a valid topic, while check fraud could be classified as invalid. This helps to fine-tune the inputs and outputs while ensuring PII (Personally Identifiable Information) protection.

Rate Limiting

Initially, there may be no rate limit set, but administrators can adjust this once they analyze usage tracking data. This contributes to effective resource management.

Dashboard Capabilities

The AI Gateway serves as a unified catalog for serverless endpoints, facilitating payload logging via inference tables. All requests and responses are stored in a Delta table within the user's account, presenting a comprehensive overview including latency, request details, response specifics, and associated metadata.

A/B Testing

With this data, organizations can conduct A/B testing on latency across different models to decide which model should proceed to production. The analysis can extend beyond latency to different performance metrics, bolstering decision-making based on data-driven insights.

Usage Tracking Insights

The usage tracking capability is valuable for cost attribution and abuse detection. Administrators can view all endpoints and token usage, allowing for identification of potential overuse or high-cost model interactions.

PII Audits

To ensure compliance, PII audits can classify inputs and outputs based on logged requests. Guardrails can prevent real-time PII leakage, creating an environment for secure data interactions.

Data Quality Monitoring

Lakehouse monitoring can be set up on the payload logging table for tracking data quality over time, which aids in creating evaluation and fine-tuning datasets. This dashboard harnesses usage stored in easy-to-access Delta tables within Unity Catalog, ensuring full access, unified governance, and observability of AI systems.

Keywords

Mosaic AI Gateway, secure AI, governed AI, usage tracking, inference tables, AI guardrails, rate limiting, A/B testing, data quality, PII audits

FAQ

What is the Mosaic AI Gateway?
The Mosaic AI Gateway is an Enterprise API Gateway that allows organizations to manage their Large Language Models (LLMs) with secure governance and observability.

How does usage tracking work in the AI Gateway?
Usage tracking provides administrators with insights on who is accessing each endpoint and their token usage, allowing for effective resource management and rate limit adjustments.

What are inference tables?
Inference tables are used to log incoming requests and outgoing responses, capturing essential data like latency and response details at the account level.

How can I implement AI guardrails?
AI guardrails can be enabled to filter valid and invalid topics for processing by AI models, thus helping to prevent PII leakage.

Can the AI Gateway be used for A/B testing scenarios?
Yes, the data logged can be analyzed to perform A/B testing on latency or other performance metrics to determine the optimal model for production.