Skip to main content

LiteLLM

LiteLLM Proxy ships an Alice guardrail. It forwards each prompt before the model call and each completion after it; Alice returns ALLOW, BLOCK, MASK or DETECT and the proxy enforces it. Which policies apply is decided per virtual key, so one proxy serves many applications.

Status: Generally available · Evaluates: Prompts, Responses, Tool calls · Vendor: BerriAI

Prompts (pre_call) and responses (post_call).

Setup​

pip install "litellm[proxy]" # v1.101.0 or later

Configuration: ALICE_API_KEY, ALICE_API_BASE (optional), alice_app_id in key metadata

  1. Add the guardrail: alice block to config.yaml (LiteLLM v1.101.0 or later) and start the proxy.
  2. Generate one virtual key per application with alice_app_id in its metadata (the key alias is the fallback).
  3. Call the proxy with that key. Blocked requests return HTTP 400.

Example​

model_list:
- model_name: gpt-4o
litellm_params:
model: openai/gpt-4o
api_key: os.environ/OPENAI_API_KEY

guardrails:
- guardrail_name: alice
litellm_params:
guardrail: alice
mode: [pre_call, post_call]
default_on: true
api_key: os.environ/ALICE_API_KEY
# litellm --config config.yaml
#
# curl -sSLX POST 'http://0.0.0.0:4000/key/generate' \
# --header "Authorization: Bearer $LITELLM_API_KEY" \
# --header 'Content-Type: application/json' \
# --data '{"key_alias": "payments-bot", "metadata": {"alice_app_id": "payments-bot"}}'

Good to know​

unreachable_fallback defaults to fail_closed. If some of an application's policies time out or hit a server error on a message, the verdict comes from the policies that did evaluate it; any other evaluation failure, including part of a message no policy could evaluate, is treated as unreachable. Masking does not apply to streamed responses; blocking does. Sensitive request fields are stripped before anything reaches Alice. Claude Code's <system-reminder> blocks are not evaluated: a block holding only reminders is skipped, and reminders at the start or end of a block are left out of its evaluation and kept in place when it is masked. Tool calls and tool results are evaluated against the policies that apply to prompts, unless your policies target tool calls and tool results separately; then each is evaluated only against the policies that list it. If your policies target system prompts, the system prompt, an OpenAI developer message included, is evaluated whole against the policies that list it, and again on later requests until it is allowed; a block on it blocks the request it arrived on. If your policies target tool definitions, each tool's definition (name, description and parameter schema) is evaluated against the policies that list it, and again on later requests until the tool set is allowed; a block on any definition blocks the request. Only the first 32 tools are evaluated, and a definition over 8,000 characters is evaluated without its parameter schema.