Add MLflow AI Gateway LLM integration example#21239
Add MLflow AI Gateway LLM integration example#21239PattaraS wants to merge 4 commits intorun-llama:mainfrom
Conversation
|
Check out this pull request on See visual diffs & provide feedback on Jupyter Notebooks. Powered by ReviewNB |
dosubot
Bot
added
the
size:L
| "- **Traffic splitting** — route a percentage of requests to different models (A/B testing)\n", | ||
| "- **OpenAI-compatible API** — works with any OpenAI SDK client\n", | ||
| "\n", | ||
| "The gateway is **database-backed** and configured through the MLflow UI — no YAML files required. It runs as part of `mlflow server` (MLflow ≥ 3.0)." |
There was a problem hiding this comment.
I'd remove this line. Users don't need to know "— no YAML files required"
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "%pip install llama-index-llms-openai-like" |
There was a problem hiding this comment.
Does this work on jupyter notebooks?
| "\n", | ||
| "Before making LLM requests, you need to create a **gateway endpoint** — a named route that maps to a specific LLM provider and model. You can do this via the UI, the Python client, or the REST API.\n", | ||
| "\n", | ||
| "### Option 1: MLflow UI\n", |
There was a problem hiding this comment.
Can we add documentation link and add a screenshot?
| "6. Enter your provider API key — it is stored encrypted on the server\n", | ||
| "7. Click **Save**\n", | ||
| "\n", | ||
| "### Option 2: Python Client\n", |
There was a problem hiding this comment.
Do we think users will use option 2 and 3?
| "\n", | ||
| "Route a percentage of requests to different models for A/B testing. For example, 90% to `gpt-4o-mini` and 10% to `gpt-4o`. Configure via **AI Gateway → Edit Endpoint → Routing Strategy**.\n", | ||
| "\n", | ||
| "### Budget Tracking\n", |
| "\n", | ||
| "Set token or cost budgets per endpoint or per user. When the budget is exhausted, the gateway returns an error. Configure via **AI Gateway → Budgets**.\n", | ||
| "\n", | ||
| "### Usage Tracing\n", |
There was a problem hiding this comment.
Let's also add screenshots, I think it's interesting to see what information is logged in the tracing and how cost is tracked when users use Llama index agents.
Adds a Jupyter notebook demonstrating how to use MLflow AI Gateway as an LLM backend in LlamaIndex via the OpenAI-compatible endpoint. Includes setup instructions (UI, Python client, curl), chat/streaming/complete examples using OpenAILike, and an mlflow.deployments client alternative. Also adds MLflow Gateway to the available LLM integrations list.
- Remove "no YAML files required" from intro - Add documentation link to MLflow AI Gateway docs - Simplify endpoint creation section to UI-only (remove Python/curl options) - Add TODO screenshot placeholders for Create Endpoint UI, Budgets UI, and Traces UI with LlamaIndex agent traces
The mlflow.deployments.get_deploy_client API is deprecated. Replace the "Alternative" section with the current recommended approaches: OpenAI SDK with base_url pointing to the gateway, and plain HTTP requests using the MLflow Invocations API.
Adds four screenshots to the MLflow Gateway integration notebook: - Create Endpoint UI form - Budget Policy creation dialog - Usage dashboard (requests, latency, errors) - Usage trace list and trace detail view
PattaraS
force-pushed
the
add-mlflow-gateway-integration
branch
from
28ae010 to
8a4ba3f
Compare
dosubot
Bot
added
size:M
|
Hi @logan-markewich, friendly ping on this PR. We're from the MLflow team and working on adding MLflow AI Gateway integration guides across major agent frameworks. MLflow AI Gateway (MLflow ≥ 3.0) is an open-source LLM proxy with built-in secrets management, fallback/retry, traffic splitting, and usage tracing — all configured |
2 participants
Description
Adds MLflow AI Gateway integration documentation — a Jupyter notebook showing how to use MLflow AI Gateway as an LLM backend in LlamaIndex via
OpenAILike, plus screenshots of the gateway UI and an entry in the integrations list.New Package?
Version Bump?
Type of Change
How Has This Been Tested?
Notebook code cells were executed against a running MLflow server with a gateway endpoint configured.
Suggested Checklist:
AI Disclosure
This pull request was AI-assisted by Claude. All content was reviewed and validated by a human contributor. Screenshots were captured manually from a running MLflow server.