> For the complete documentation index, see [llms.txt](https://docs.ox.security/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.ox.security/ox-integrations/3rd-party-integrations/ai-security-litellm.md).

# AI Security: LiteLLM

## About LiteLLM

LiteLLM is a self-hosted LLM gateway that provides access to multiple model providers through a single OpenAI-compatible API.

Connect LiteLLM to OX to discover and monitor virtual keys (services) configured in your LiteLLM gateway. Using the OX Guardrail, OX inspects prompts before they reach the model provider to detect sensitive data exposure and, when configured, block requests that violate security policies.

## Prerequisites

Before configuring the connector, make sure you have the following:

| System  | Requirement                                                                                             |
| ------- | ------------------------------------------------------------------------------------------------------- |
| OX      | Permission to configure connectors and create API keys                                                  |
| LiteLLM | A deployed LiteLLM Proxy accessible from OX                                                             |
| LiteLLM | Permission to create users and Management virtual keys                                                  |
| LiteLLM | Access to the proxy's `config.yaml` file and environment variables, and permission to restart the proxy |

> **Note:** If your LiteLLM Proxy is behind a firewall or other network access controls, allow inbound HTTPS connections from the OX IP addresses listed in [Prerequisites and access](https://docs.ox.security/get-started/onboarding-to-ox/prerequisites-and-access). The LiteLLM Proxy must also be able to send outbound HTTPS requests to the OX API URL.

## Step 1: Create a Management virtual key in LiteLLM

OX uses a Management virtual key to connect to your LiteLLM instance and discover available services. For read-only access, create the key under a user assigned the Admin (View Only) role.

1. Open your self-hosted LiteLLM Proxy UI, for example, `https://litellm.your-company.com/ui`.
2. Go to **Virtual Keys** and select **+ Create New Key**.
3. Under **Owned By**, select **Another User**.
4. Select **Create User** and assign the **Admin (View Only)** role.
5. Configure the virtual key using the following parameters:

   | Parameter | Value                                          |
   | --------- | ---------------------------------------------- |
   | User ID   | The user you created                           |
   | Key Type  | Management                                     |
   | Key Name  | A descriptive name, for example, `ox-security` |
6. Select **Create Key**.
7. Copy the generated key and store it securely. LiteLLM displays the key only once.

## Step 2: Connect LiteLLM to OX

1. In OX, go to **Connectors > AI Security > LiteLLM**.

<figure><img src="https://884876233-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FdK3XMLdV8zRg847RmGmZ%2Fuploads%2Fgit-blob-7012c8f29a931f9a7441664bdd30fe423e266581%2Fllm_1.png?alt=media" alt="" width="563"><figcaption></figcaption></figure>

2. In the **Configure your LiteLLM credentials** dialog, enter the following parameters:

| Parameter        | Description                                                                                   |
| ---------------- | --------------------------------------------------------------------------------------------- |
| LiteLLM Host URL | The base URL of your deployed LiteLLM Proxy, for example, `https://litellm.your-company.com`. |
| Token            | The Management virtual key created in Step 1.                                                 |

3. Select **Verify Connectivity** to validate the credentials and connection.
4. Select **Connect**.
5. In the service selection dialog, select the virtual keys (services) you want OX to monitor. Use the search field to locate specific services, or select **Choose all** to include all discovered services.

<figure><img src="https://884876233-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FdK3XMLdV8zRg847RmGmZ%2Fuploads%2Fgit-blob-c94a356a47f0a60e7040935bc3c95460b1f885d0%2Fllm_2.png?alt=media" alt="" width="375"><figcaption></figcaption></figure>

6. To automatically monitor services created in the future, select **Monitor all newly created services**.
7. Select **Save**.

> **Note:** The Management virtual key is used for service discovery. It is not the API key used by the OX Guardrail for prompt inspection.

## [Step 3: Create an OX API key](https://docs.ox.security/admin-settings/creating-api-keys.md)

## Step 4: Configure the OX Guardrail in LiteLLM

Configure the OX Guardrail to inspect prompts before LiteLLM forwards requests to the model provider.

1. Open the `config.yaml` file of your LiteLLM Proxy.
2. Add the following configuration under `guardrails`:

```yaml
guardrails:
  - guardrail_name: ox-prompt-inspection
    litellm_params:
      guardrail: generic_guardrail_api
      mode: pre_call
      api_base: os.environ/OX_API_URL
      api_key: os.environ/OX_API_KEY
      unreachable_fallback: fail_open
      default_on: true
      fail_on_error: false
```

3. Configure the following environment variables in your LiteLLM Proxy environment, such as an `.env` file, Docker Compose service, or Kubernetes secret:

```bash
OX_API_URL=<your OX API URL>
OX_API_KEY=<the API Key Secret from Step 3>
```

4. Restart the **LiteLLM Proxy** to apply the configuration.

The following table describes the guardrail parameters.

| Parameter              | Description                                                                   |
| ---------------------- | ----------------------------------------------------------------------------- |
| `guardrail_name`       | Identifies the guardrail in LiteLLM.                                          |
| `guardrail`            | `generic_guardrail_api` configures LiteLLM to use an external guardrail API.  |
| `mode`                 | `pre_call` inspects prompts before forwarding requests to the model provider. |
| `api_base`             | References the OX API endpoint through `OX_API_URL`.                          |
| `api_key`              | References the OX API key through `OX_API_KEY`.                               |
| `unreachable_fallback` | `fail_open` allows requests to continue when OX is unreachable.               |
| `default_on`           | `true` enables the guardrail for all requests by default.                     |
| `fail_on_error`        | `false` prevents guardrail errors from automatically failing requests.        |

> **Important:** Do not store the OX API key directly in `config.yaml` or commit it to source control. Use environment variables backed by a secure secret store.

> **Note:** The provided configuration allows requests to continue if OX is unreachable or the guardrail encounters an error. To reject requests when the guardrail fails, set `fail_on_error: true`. If `unreachable_fallback` is omitted, LiteLLM defaults to `fail_on_error` behavior.

## Manage monitored services

After connecting LiteLLM, you can update the virtual keys that OX monitors.

1. In OX, open the **LiteLLM** connector.
2. Select the **Settings** (gear) icon.
3. In the service selection dialog, select or clear the services you want to monitor.
4. To automatically include future services, select **Monitor all newly created services**.
5. Select **Save**.

## Troubleshooting

| Issue                           | Possible cause and resolution                                                                                                           |
| ------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------- |
| Invalid token                   | The Management virtual key is incorrect, expired, or revoked. Create a new key in LiteLLM.                                              |
| Missing permissions             | Verify that the Management virtual key belongs to a user assigned the Admin (View Only) role.                                           |
| Invalid host URL                | Verify that the URL is the base URL of your LiteLLM Proxy and is accessible from OX.                                                    |
| Connectivity verification fails | Check the firewall settings and allow the required OX IP addresses to access the LiteLLM Proxy.                                         |
| Prompts are not inspected       | Verify the OX API key type, environment variables, outbound connectivity, and that the LiteLLM Proxy was restarted after configuration. |


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.ox.security/ox-integrations/3rd-party-integrations/ai-security-litellm.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
