diff --git a/.claude/skills/doc-tests/SKILL.md b/.claude/skills/doc-tests/SKILL.md index d3ab40c02..5b25ed89b 100644 --- a/.claude/skills/doc-tests/SKILL.md +++ b/.claude/skills/doc-tests/SKILL.md @@ -1,7 +1,7 @@ --- name: doc-test-guides description: Add executable doc tests to agentgateway documentation guides using the doc test framework. Use when the user asks to "add doc tests", "add tests to a guide", "add tests to a topic", mentions "YAMLTest", or is working on quickstart guides, standalone binary guides, or Kubernetes doc pages that should generate runnable scripts from code blocks. -version: 1.0.0 +version: 1.2.0 --- # Doc test guides skill @@ -25,6 +25,7 @@ Use this skill when adding tests to documentation guides in the `agentgateway/we > **Critical**: Most Kubernetes topic pages (e.g. `content/docs/kubernetes/latest/resiliency/timeouts/request.md`) are thin wrappers that only contain `{{< reuse "agw-docs/pages/..." >}}`. **Always place doc-test blocks in the reuse file** (`assets/agw-docs/pages/...`), never in the content wrapper. This way both `latest` and `main` versions automatically inherit the tests — you only need to add them once. 3. **Extractor** resolves `{{< reuse "..." >}}` from `assets/`, so the script is built from the expanded content. Reference the **content file** in `test:` sources; the extractor will follow reuse. 4. **Block order**: Selected blocks are emitted in document order (by file and `start_line`). Hidden blocks (e.g. "start server in background") must appear *before* any visible block that depends on them (e.g. curl). The extractor sorts selected blocks by `(file_path, start_line)` so hidden blocks are not deferred to the end. +5. **Byte-identical blocks are silently dropped**: `build_script()` in `scripts/doc_test_extract.py` keeps a `seen` set of block contents and skips any block whose content (after stripping leading and trailing newlines) exactly matches an earlier selected block. Only the **first** copy reaches the generated script — there is no warning. See "Repeated commands across sections" under step 3 for what this breaks and how to avoid it. --- @@ -51,6 +52,38 @@ Use this skill when adding tests to documentation guides in the `agentgateway/we - **Display-only YAML blocks**: Some pages show YAML configs as plain display blocks (no `cat <<'EOF'` shell wrapper), unlike LLM guides that wrap configs in shell commands. You can't tag a display-only YAML block with `paths=` because it isn't a runnable shell command. Instead, add a **hidden** `{{< doc-test >}}` block that writes the config with `cat <<'EOF' > config.yaml`. See `content/docs/standalone/main/mcp/mcp-authz.md` for an example. - **External service dependencies**: When a config example depends on an external service that can't be trivially stood up in the test (e.g. Keycloak on port 9000, a custom OIDC provider), skip that example and only test self-contained ones. It's better to test one example well than to skip the entire page. +#### Repeated commands across sections + +The extractor drops any block whose content is byte-identical to a block it already selected, keeping only the first. Nothing is logged, so the remaining copies just quietly do not run. + +This bites on multi-section pages where each section repeats the same command. For example, a page with four sections that each write a config and then validate it: + +``` +{{< doc-test paths="my-test" >}} +agentgateway -f config.yaml --validate-only +{{< /doc-test >}} +``` + +Four blocks like that collapse to **one**, so three of the four configs are never validated even though the test passes. The visible config-writing blocks survive because each one contains a different config. + +Give each repeated block distinguishing content — a comment naming the section is enough, and it makes the generated script easier to read: + +``` +{{< doc-test paths="my-test" >}} +# Multi-level delegation: validate the config written by step 1 +agentgateway -f config.yaml --validate-only +{{< /doc-test >}} +``` + +To confirm nothing was dropped, count the commands in the generated script and compare against the source: + +```sh +grep -c "validate-only" content/docs/standalone/main/.md +grep -c "validate-only" out/tests/generated/.sh +``` + +The same trap applies to any repeated command, such as an identical `start_gateway`, `sleep 3`, or curl warmup loop used in more than one section. + ### 4. Long-running processes (standalone binary) - The guide may show "run agentgateway" in the foreground. For the generated script, the process must run in the background so the script can continue (e.g. curl, YAMLTest). @@ -60,6 +93,71 @@ Use this skill when adding tests to documentation guides in the `agentgateway/we - `trap 'kill $AGW_PID 2>/dev/null' EXIT` - `sleep 3` - Do **not** add a path to the visible "run agentgateway" block so it is not included in the script; only the hidden block is. +- **Helper backends: pick a port no documented config uses.** When a test needs a local backend to forward to, remember that doc tests run sequentially within a CI shard, so a helper process that outlives its script can collide with the next test. Port `8080` is the one documented configs use most (for example, the body-buffering page's `host: localhost:8080`), so choose something else (`8081`, `8082`) for a backend you invented, and keep the documented port for the test whose page actually specifies it. +- **Make readiness checks identity-aware.** A loop like `curl -sf http://127.0.0.1:8080/ && break` passes as soon as *anything* answers on that port, so a stale process from an earlier test satisfies it and the real assertion then fails for a misleading reason. Probe for a response only your backend produces, and fail with a clear message if it never appears: + ```sh + for i in $(seq 1 30); do + [ "$(curl -sf --max-time 5 -X POST -d probe http://127.0.0.1:8080/ 2>/dev/null)" = "probe" ] && break + sleep 1 + done + if [ "$(curl -sf --max-time 5 -X POST -d probe http://127.0.0.1:8080/ 2>/dev/null)" != "probe" ]; then + echo "FAIL: the echo backend did not come up on 127.0.0.1:8080 (is the port already in use?)" + exit 1 + fi + ``` +- **Cleanup runs under `set -e`.** Generated scripts start with `set -euo pipefail`, so a failing command inside an `EXIT` trap aborts the handler and the script exits non-zero even when every assertion passed. Guard cleanup with `|| true`, and guard unset variables for `set -u`: + ```sh + stop_gateway() { + [ -n "${AGW_PID:-}" ] || return 0 + kill "$AGW_PID" 2>/dev/null || true + wait "$AGW_PID" 2>/dev/null || true + AGW_PID="" + } + ``` + +#### Never start a background process inside `$( )` + +A helper that both starts the gateway and returns a value looks convenient, but it breaks in two ways at once when it is called in a command substitution: + +```sh +# BROKEN +tool_names_for() { + agentgateway -f "$1" & + AGW_PID=$! # set in the SUBSHELL, invisible to the caller + curl ... | jq -r '...' +} +NAMES=$(tool_names_for config.yaml) +``` + +1. **The PID is lost.** `$( )` runs in a subshell, so `AGW_PID` never reaches the parent. A later `stop_gateway` sees it empty and returns without killing anything, so the gateway keeps holding its port. The next config then starts a gateway that cannot bind, and every assertion after that silently runs against the *previous* config. +2. **The gateway's output is captured.** The background process inherits the substituted stdout, so its startup log ends up concatenated into the returned value. + +Symptom: the first assertion passes, later ones fail or hang for no obvious reason, and the test eventually times out. This is easy to misread as a product bug. + +Split the two jobs, so the process starts in the parent shell and only pure-curl code runs inside `$( )`: + +```sh +start_gateway() { # call from the parent, never inside $( ) + agentgateway -f "$1" > "agw-$1.log" 2>&1 & + AGW_PID=$! +} + +wait_for_tools() { # pure curl, safe inside $( ) + local out="" + for i in $(seq 1 15); do + out=$(query_something 2>/dev/null || true) + [ -n "$out" ] && break + sleep 2 + done + echo "$out" +} + +start_gateway config.yaml +NAMES=$(wait_for_tools) +stop_gateway +``` + +Redirect the process's output to a file as well, so nothing can leak into a captured value. When a page needs several configs in sequence, `stop_gateway` between them and confirm the port is actually released before the next start. ### 5. Env vars and placeholders @@ -305,6 +403,9 @@ When in doubt, flag the failure to the user rather than silently adjusting the t - [ ] Path tags and `{{< doc-test >}}` blocks added in the **asset** file(s) (`assets/agw-docs/...`), **not** in the content wrapper files — even if multiple content files (e.g. `latest/` and `main/`) reuse the same asset. - [ ] Multiple paths in `paths="..."` are **comma-separated**, not space-separated — `paths="a,b"` ✓, `paths="a b"` ✗ (spaces make the whole string a single path, silently excluding the block). +- [ ] No two selected blocks are **byte-identical** — the extractor keeps only the first and silently drops the rest, so a repeated `--validate-only` (or `start_gateway`, or warmup loop) across sections leaves later sections untested. Add a comment naming the section, then verify with `grep -c` on the source vs. the generated script. See "Repeated commands across sections" under step 3. +- [ ] A helper backend the test invents uses a port **no documented config uses** (avoid `8080`), its readiness check probes for a response only that backend produces, and cleanup is guarded with `|| true` so a failing `kill` in an `EXIT` trap does not fail the script under `set -e`. +- [ ] No helper **starts a background process inside `$( )`** — the PID is set in the subshell and lost, so the process is never killed and later configs run against the stale one; and its output gets captured into the returned value. Start with `start_gateway` in the parent shell, keep only pure-curl code inside the substitution. See "Never start a background process inside `$( )`" under step 4. - [ ] If the guide has a long-running server, a **hidden** doc-test block starts it in the background (and optional trap/sleep); visible "start server" block has **no** path. - [ ] Placeholders in shell blocks are quoted or use `${VAR:-default}` so the script has no syntax errors. - [ ] `test:` front matter on the **content** page lists the right `file` and `path`; file path is the content path (extractor follows reuse). For pages with no testable content (index pages, no code blocks), use `test: skip` instead — counts toward coverage without generating test cases. diff --git a/content/docs/kubernetes/latest/llm/inference/_index.md b/content/docs/kubernetes/latest/llm/inference/_index.md index 1b8429fab..52ea79994 100644 --- a/content/docs/kubernetes/latest/llm/inference/_index.md +++ b/content/docs/kubernetes/latest/llm/inference/_index.md @@ -3,4 +3,6 @@ title: Inference workloads weight: 20 description: Route to your own self-hosted generative AI models with inference workloads. url: /docs/kubernetes/latest/inference/ +aliases: + - /docs/kubernetes/latest/llm/inference/ --- diff --git a/content/docs/kubernetes/main/llm/inference/_index.md b/content/docs/kubernetes/main/llm/inference/_index.md index f71a2e736..a66c1692e 100644 --- a/content/docs/kubernetes/main/llm/inference/_index.md +++ b/content/docs/kubernetes/main/llm/inference/_index.md @@ -3,4 +3,6 @@ title: Inference workloads weight: 20 description: Route to your own self-hosted generative AI models with inference workloads. url: /docs/kubernetes/main/inference/ +aliases: + - /docs/kubernetes/main/llm/inference/ --- diff --git a/content/docs/standalone/latest/configuration/routes.md b/content/docs/standalone/latest/configuration/routes.md index 2d5727988..3423994dd 100644 --- a/content/docs/standalone/latest/configuration/routes.md +++ b/content/docs/standalone/latest/configuration/routes.md @@ -3,8 +3,40 @@ title: Routes weight: 30 description: Match HTTP and TCP traffic on a gateway and forward it to backends. next: /configuration/traffic-management +test: + routes: + - file: ${versionRoot}/configuration/routes.md + path: routes --- +{{< doc-test paths="routes" >}} +# ============================================================================ +# Doc test coverage for this guide (these comments are not rendered on the page) +# ============================================================================ +# WHAT THIS TEST VALIDATES: +# * "HTTP routes": the example config is accepted by agentgateway +# (--validate-only), covering the `gateways`, `protocol: HTTP`, `name`, +# `gateways: [...]`, `hostnames`, `matches.path.pathPrefix`, and +# `backends[].host` / `weight` fields the route table documents. +# * "TCP routes": the `tcpRoutes` example is accepted, covering `protocol: TCP` +# and the simpler TCP route structure. +# * "Example configuration with policies": the route-with-CORS example is +# accepted, covering `policies.cors` on a route and an inline `backends[].mcp` +# backend with a `stdio` target. +# +# WHAT THIS TEST DOES NOT VALIDATE (and why): +# * That traffic is actually matched and forwarded - requires config/traffic the +# page omits; every example points at a placeholder backend +# (`http.example.com:8080`, `postgres.example.com:5432`) that the test cannot +# stand up, so only config acceptance is asserted. +# * The `matches` header, method, and query options, and the "attaches to the +# gateway named `default`" fallback - display-only table rows with no example +# config on this page. Matching is covered by the Request matching guide. +# * The CORS policy's runtime behavior - covered by the CORS guide's own test; +# here the block only proves the policy is accepted on a route. +{{< reuse "agw-docs/snippets/install-agentgateway-binary.md" >}} +{{< /doc-test >}} + {{< gloss "Route" >}}Routes{{< /gloss >}} are the entry points for traffic to your agentgateway. They attach to [gateways]({{< link-hextra path="/configuration/gateways/" >}}) and are used to route traffic to {{< gloss "Backend" >}}backends{{< /gloss >}}. ## Types of routes @@ -36,6 +68,28 @@ routes: weight: 1 ``` +{{< doc-test paths="routes" >}} +cat <<'EOF' > config-http.yaml +# yaml-language-server: $schema=https://agentgateway.dev/schema/config +gateways: + http-proxy: + port: 8080 + protocol: HTTP +routes: +- name: http-backend + gateways: [http-proxy] + hostnames: + - "example.com" + matches: + - path: + pathPrefix: / + backends: + - host: http.example.com:8080 + weight: 1 +EOF +agentgateway -f config-http.yaml --validate-only +{{< /doc-test >}} + HTTP routes support various matching options for incoming requests. For more information, see the [Request matching]({{< link-hextra path="/configuration/traffic-management/matching/" >}}) guide. ### TCP routes @@ -60,6 +114,23 @@ tcpRoutes: weight: 1 ``` +{{< doc-test paths="routes" >}} +cat <<'EOF' > config-tcp.yaml +# yaml-language-server: $schema=https://agentgateway.dev/schema/config +gateways: + postgres-proxy: + port: 5432 + protocol: TCP +tcpRoutes: +- name: postgres-backend + gateways: [postgres-proxy] + backends: + - host: postgres.example.com:5432 + weight: 1 +EOF +agentgateway -f config-tcp.yaml --validate-only +{{< /doc-test >}} + For more information, see [TCP route matching]({{< link-hextra path="/configuration/traffic-management/matching#tcp-routes" >}}). ## Route configuration @@ -115,6 +186,34 @@ routes: args: ["@modelcontextprotocol/server-everything"] ``` +{{< doc-test paths="routes" >}} +cat <<'EOF' > config-policies.yaml +# yaml-language-server: $schema=https://agentgateway.dev/schema/config +gateways: + default: + port: 3000 +routes: +- policies: + cors: + allowOrigins: + - "*" + allowHeaders: + - mcp-protocol-version + - content-type + - cache-control + exposeHeaders: + - "Mcp-Session-Id" + backends: + - mcp: + targets: + - name: everything + stdio: + cmd: npx + args: ["@modelcontextprotocol/server-everything"] +EOF +agentgateway -f config-policies.yaml --validate-only +{{< /doc-test >}} + ## Next steps After you configure routes, you might want to apply policies to them or learn more about traffic management options. diff --git a/content/docs/standalone/latest/configuration/security/cors.md b/content/docs/standalone/latest/configuration/security/cors.md index 75603ff74..48e3694b1 100644 --- a/content/docs/standalone/latest/configuration/security/cors.md +++ b/content/docs/standalone/latest/configuration/security/cors.md @@ -2,12 +2,58 @@ title: CORS weight: 11 description: Configure Cross-Origin Resource Sharing policies to control cross-domain requests. +test: + cors: + - file: ${versionRoot}/configuration/security/cors.md + path: cors --- Attaches to: {{< badge content="Route" path="/configuration/routes/">}} {{< reuse "agw-docs/snippets/config-styles-note.md" >}} +{{< doc-test paths="cors" >}} +# ============================================================================ +# Doc test coverage for this guide (these comments are not rendered on the page) +# ============================================================================ +# WHAT THIS TEST VALIDATES: +# * All three example configs (Simplified LLM, Simplified MCP, and +# Routing-based) are accepted by agentgateway (--validate-only), covering +# `allowOrigins`, `allowMethods`, `allowHeaders`, `exposeHeaders`, +# `allowCredentials`, and `maxAge` in both duration (`10m`, `100s`) forms. +# * The "Origin Allowed" branch of the CORS preflight diagram: with the +# Routing-based config loaded, an OPTIONS preflight from +# https://app.example.com returns 200 with access-control-allow-origin, +# -allow-methods, -allow-headers, -allow-credentials, -expose-headers, and +# -max-age set to the configured values (maxAge 100s is emitted as `100`). +# * The "Origin NOT Allowed" branch: an OPTIONS preflight from an origin that +# is not in `allowOrigins` still returns 200 but with no +# access-control-allow-origin header, which is what causes the browser to +# block the response. +# * The actual (non-preflight) cross-origin request: the Routing-based config is +# rerun with its placeholder backend (`api.example.com:443`) swapped for a +# local echo backend, and a GET with an `Origin` header is asserted to reach +# the backend AND come back with the CORS response headers attached - not +# just the preflight the earlier assertion covers. +# * The Simplified (MCP) config at runtime: rerun with a real npx-launched MCP +# server (the same server used by the mcp/connect guides), and an OPTIONS +# preflight against the MCP port asserts the CORS headers the settings list +# documents, including `maxAge: 10m` resolving to a `600`-second header. +# +# WHAT THIS TEST DOES NOT VALIDATE (and why): +# * That a browser enforces the policy - different layer; as the page's own tip +# notes, curl and other HTTP clients ignore CORS headers, so the test can only +# assert the headers agentgateway returns. +# * The Simplified (LLM) config at runtime - external dependency; it needs a +# real OpenAI API key to reach a provider that could return CORS headers on +# an actual completion, so it is only validated as config. +{{< reuse "agw-docs/snippets/install-agentgateway-binary.md" >}} + +# The Simplified (LLM) example reads the API key from the environment. +# --validate-only still resolves env vars, so a placeholder is enough here. +export OPENAI_API_KEY="${OPENAI_API_KEY:-test}" +{{< /doc-test >}} + ## About CORS {{< gloss "CORS (Cross-Origin Resource Sharing)" >}}Cross-origin resource sharing (CORS){{< /gloss >}} is a browser security mechanism which allows a server to control which origins can request and interact with resources that are hosted on a different domain. By default, web browsers only allow requests to resources that are hosted on the same domain as the web page that served the original request. Access to web pages or resources that are hosted on a different domain is restricted to prevent potential security vulnerabilities, such as cross-site request forgery (CRSF). @@ -136,3 +182,228 @@ routes: ``` {{< /tab >}} {{< /tabs >}} + +{{< doc-test paths="cors" >}} +cat <<'EOF' > config-llm.yaml +# yaml-language-server: $schema=https://agentgateway.dev/schema/config +llm: + policies: + cors: + allowOrigins: + - https://chat.example.com + allowMethods: + - POST + - OPTIONS + allowHeaders: + - authorization + - content-type + exposeHeaders: + - x-request-id + allowCredentials: true + maxAge: 10m + models: + - name: "*" + provider: openAI + params: + apiKey: "$OPENAI_API_KEY" +EOF +agentgateway -f config-llm.yaml --validate-only + +cat <<'EOF' > config-mcp.yaml +# yaml-language-server: $schema=https://agentgateway.dev/schema/config +mcp: + port: 3000 + policies: + cors: + allowOrigins: + - https://chat.example.com + allowMethods: + - POST + - OPTIONS + allowHeaders: + - authorization + - content-type + exposeHeaders: + - x-request-id + allowCredentials: true + maxAge: 10m + targets: + - name: everything + stdio: + cmd: npx + args: ["@modelcontextprotocol/server-everything"] +EOF +agentgateway -f config-mcp.yaml --validate-only + +cat <<'EOF' > config.yaml +# yaml-language-server: $schema=https://agentgateway.dev/schema/config +gateways: + default: + port: 3000 +routes: +- backends: + - host: api.example.com:443 + policies: + cors: + allowOrigins: + - https://app.example.com + allowMethods: + - GET + - POST + - OPTIONS + allowHeaders: + - authorization + - content-type + exposeHeaders: + - x-request-id + allowCredentials: true + maxAge: 100s +EOF +agentgateway -f config.yaml --validate-only +{{< /doc-test >}} + +{{< doc-test paths="cors" >}} +agentgateway -f config.yaml & +AGW_PID=$! +trap 'kill $AGW_PID 2>/dev/null || true' EXIT +sleep 3 +{{< /doc-test >}} + +{{< doc-test paths="cors" >}} +YAMLTest -f - <<'EOF' +- name: Preflight from an allowed origin returns the configured CORS headers + retries: 3 + http: + url: "http://localhost:3000" + path: / + method: OPTIONS + headers: + origin: "https://app.example.com" + access-control-request-method: GET + access-control-request-headers: authorization + source: + type: local + expect: + statusCode: 200 + headers: + - name: access-control-allow-origin + comparator: equals + value: "https://app.example.com" + - name: access-control-allow-methods + comparator: contains + value: GET + - name: access-control-allow-headers + comparator: contains + value: authorization + - name: access-control-expose-headers + comparator: contains + value: x-request-id + - name: access-control-allow-credentials + comparator: equals + value: "true" + - name: access-control-max-age + comparator: equals + value: "100" +EOF +{{< /doc-test >}} + +{{< doc-test paths="cors" >}} +# The "Origin NOT Allowed" branch of the diagram: agentgateway still answers the +# preflight, but omits access-control-allow-origin, which is what makes the +# browser block the response. +DISALLOWED_HEADERS=$(curl -s -i -X OPTIONS http://localhost:3000/ \ + -H "Origin: https://not-allowed.example.com" \ + -H "Access-Control-Request-Method: GET") +if grep -qi '^access-control-allow-origin' <<<"$DISALLOWED_HEADERS"; then + echo "FAIL: preflight from a disallowed origin returned access-control-allow-origin" + echo "$DISALLOWED_HEADERS" + exit 1 +fi +echo "✓ Preflight from a disallowed origin returned no access-control-allow-origin header" +{{< /doc-test >}} + +{{< doc-test paths="cors" >}} +# Confirm CORS headers reach an actual (non-preflight) cross-origin request, not +# just the preflight asserted above. Rerun the Routing-based config with its +# placeholder backend (api.example.com:443) swapped for a local echo backend, so +# a GET with an Origin header has something to forward to. +kill $AGW_PID 2>/dev/null || true +wait $AGW_PID 2>/dev/null || true + +cat <<'PYEOF' > backend.py +from http.server import BaseHTTPRequestHandler, HTTPServer + +class Echo(BaseHTTPRequestHandler): + def do_GET(self): + body = b"ok" + self.send_response(200) + self.send_header("content-type", "text/plain") + self.send_header("content-length", str(len(body))) + self.end_headers() + self.wfile.write(body) + + def log_message(self, *args): + pass + +HTTPServer(("127.0.0.1", 8081), Echo).serve_forever() +PYEOF +python3 backend.py & +BACKEND_PID=$! +trap 'kill $AGW_PID $BACKEND_PID 2>/dev/null || true' EXIT +for i in $(seq 1 30); do + curl -sf -o /dev/null http://127.0.0.1:8081/ && break + sleep 1 +done + +sed 's#api.example.com:443#localhost:8081#' config.yaml > config-cors-local.yaml +agentgateway -f config-cors-local.yaml & +AGW_PID=$! +sleep 3 + +RESPONSE=$(curl -s -i http://localhost:3000/ -H "Origin: https://app.example.com") +if ! grep -qi '^access-control-allow-origin: https://app.example.com' <<<"$RESPONSE"; then + echo "FAIL: an actual cross-origin GET did not come back with access-control-allow-origin" + echo "$RESPONSE" + exit 1 +fi +if ! grep -q '^ok$' <<<"$RESPONSE"; then + echo "FAIL: the request was not actually forwarded to the backend" + echo "$RESPONSE" + exit 1 +fi +echo "✓ An actual cross-origin request reached the backend and came back with CORS headers" + +kill $AGW_PID $BACKEND_PID 2>/dev/null || true +wait $AGW_PID $BACKEND_PID 2>/dev/null || true +{{< /doc-test >}} + +{{< doc-test paths="cors" >}} +# Confirm the Simplified (MCP) config's CORS policy works at runtime, using a +# real npx-launched MCP server (the same server the mcp/connect guides use) so +# no external dependency is needed. +agentgateway -f config-mcp.yaml & +AGW_PID=$! +trap 'kill $AGW_PID 2>/dev/null || true' EXIT +for i in $(seq 1 30); do + curl -sf -o /dev/null --max-time 5 http://localhost:15021/healthz/ready && break + sleep 2 +done + +MCP_HEADERS=$(curl -s -i -X OPTIONS http://127.0.0.1:3000/mcp \ + -H "Origin: https://chat.example.com" \ + -H "Access-Control-Request-Method: POST") +if ! grep -qi '^access-control-allow-origin: https://chat.example.com' <<<"$MCP_HEADERS"; then + echo "FAIL: MCP port preflight did not return access-control-allow-origin" + echo "$MCP_HEADERS" + exit 1 +fi +if ! grep -qi '^access-control-max-age: 600' <<<"$MCP_HEADERS"; then + echo "FAIL: MCP port preflight's access-control-max-age was not 600 (maxAge: 10m)" + echo "$MCP_HEADERS" + exit 1 +fi +echo "✓ The Simplified (MCP) CORS policy answers a real preflight against the MCP port" + +kill $AGW_PID 2>/dev/null || true +wait $AGW_PID 2>/dev/null || true +{{< /doc-test >}} diff --git a/content/docs/standalone/latest/configuration/security/network-authz.md b/content/docs/standalone/latest/configuration/security/network-authz.md index e98aa8e8f..a6560d452 100644 --- a/content/docs/standalone/latest/configuration/security/network-authz.md +++ b/content/docs/standalone/latest/configuration/security/network-authz.md @@ -2,10 +2,58 @@ title: Network authorization weight: 13 description: Enforce access control at the L4 level using CEL expressions. +test: + network-authz: + - file: ${versionRoot}/configuration/security/network-authz.md + path: network-authz --- Attaches to: {{< badge content="Frontend" path="/configuration/overview/">}} +{{< doc-test paths="network-authz" >}} +# ============================================================================ +# Doc test coverage for this guide (these comments are not rendered on the page) +# ============================================================================ +# WHAT THIS TEST VALIDATES: +# * "Configuration": the example config is accepted by agentgateway +# (--validate-only), covering `frontendPolicies.networkAuthorization.rules` +# with all three rule types (`allow`, `deny`, `require`) and the +# `source.address` / `source.port` CEL variables. +# * "Examples": all three example configs are accepted - the private-range +# allowlist (`cidr(...).containsIP(...)`), the mTLS `source.tls.identity` +# requirement, and the layered L4+L7 config that combines +# `networkAuthorization` with a route-level `authorization` policy. +# * Allowlist semantics from the "Evaluation order" list, rule 6: with the +# Configuration example loaded, a connection from localhost matches no `allow` +# rule, so the connection is rejected at L4 before any HTTP response is sent +# (the client sees a connection reset, not a status code). +# * Evaluation order rule 4 (allow match): a variant of the Configuration +# example with an `allow` rule that matches the test client's own address +# (`127.0.0.1`) lets the connection reach HTTP routing - observed as a `503` +# from the placeholder backend rather than a connection failure, confirming +# network authorization is the thing that let it through. +# +# WHAT THIS TEST DOES NOT VALIDATE (and why): +# * Evaluation order rules for `deny` match and denylist semantics - requires +# spoofing the test client's own source IP/port, which isn't controllable +# from userspace without a second host or network namespace. +# * Evaluation order rule for `require` match/no-match - the test client's +# ephemeral source port is always > 1024, so a `require: source.port > 1024` +# rule always trivially passes; forcing a low source port isn't controllable +# from userspace either. +# * Evaluation order rule 1 (no rules): trivial by definition (no +# `networkAuthorization` config at all behaves like any other page's +# unauthenticated route), so a dedicated example would add no signal beyond +# what every other doc test on this site already demonstrates. +# * `source.tls.identity` and `source.tls.subject_alt_names` at runtime - +# requires config/traffic the page omits; the page shows no TLS listener or +# client certificate setup, so the mTLS example is only validated as config. +# * The route-level `authorization` JWT requirement in the layered example - +# external dependency; enforcing it needs a JWT issuer this page does not set +# up. HTTP authorization is covered by its own guide. +{{< reuse "agw-docs/snippets/install-agentgateway-binary.md" >}} +{{< /doc-test >}} + Network authorization enforces access control at the L4 (transport) level, before HTTP processing. You can enforce policies for non-HTTP traffic such as raw TCP and TLS connections, and layer L4+L7 controls when you combine policies with [HTTP authorization]({{< link-hextra path="/configuration/security/http-authz/" >}}). Network authorization uses [CEL expressions]({{< link-hextra path="/reference/cel/" >}}) evaluated against the connection's source context. @@ -31,6 +79,85 @@ routes: - host: localhost:8080 ``` +{{< doc-test paths="network-authz" >}} +cat <<'EOF' > config.yaml +# yaml-language-server: $schema=https://agentgateway.dev/schema/config +frontendPolicies: + networkAuthorization: + rules: + - allow: 'source.address == "10.0.0.0" || source.address == "10.0.0.1"' + - deny: 'source.address == "192.168.1.100"' + - require: 'source.port > 1024' + +gateways: + default: + port: 3000 +routes: +- backends: + - host: localhost:8080 +EOF +agentgateway -f config.yaml --validate-only +{{< /doc-test >}} + +{{< doc-test paths="network-authz" >}} +# Load the Configuration example and confirm allowlist semantics (evaluation +# order rule 6): the test client connects from localhost, which matches none of +# the `allow` rules, so the connection must be rejected at L4. A rejected L4 +# connection produces a transport error rather than an HTTP status, so this is +# asserted with curl rather than YAMLTest. +agentgateway -f config.yaml & +AGW_PID=$! +trap 'kill $AGW_PID 2>/dev/null || true' EXIT +sleep 3 + +if curl -s -o /dev/null --max-time 5 http://localhost:3000/; then + echo "FAIL: connection from a non-allowlisted source address was not rejected" + exit 1 +fi +echo "✓ Network authorization rejected a connection from a non-allowlisted source address" + +kill $AGW_PID 2>/dev/null || true +wait $AGW_PID 2>/dev/null || true +{{< /doc-test >}} + +{{< doc-test paths="network-authz" >}} +# Evaluation order rule 4 (allow match): the same shape as the Configuration +# example, but with an allow rule that matches the test client's own address +# (127.0.0.1) instead of the page's example addresses. If network authorization +# is what's gating the connection, it should now reach HTTP routing -- observed +# as a 503 from the placeholder backend at localhost:8080, not a connection +# failure. +cat <<'EOF' > config-allow-match.yaml +frontendPolicies: + networkAuthorization: + rules: + - allow: 'source.address == "127.0.0.1"' + - deny: 'source.address == "192.168.1.100"' + - require: 'source.port > 1024' + +gateways: + default: + port: 3000 +routes: +- backends: + - host: localhost:8080 +EOF +agentgateway -f config-allow-match.yaml --validate-only + +agentgateway -f config-allow-match.yaml & +AGW_PID=$! +sleep 3 + +STATUS=$(curl -s -o /dev/null -w '%{http_code}' --max-time 5 http://127.0.0.1:3000/) +kill $AGW_PID 2>/dev/null || true +wait $AGW_PID 2>/dev/null || true +if [ "$STATUS" != "503" ]; then + echo "FAIL: expected a 503 from the placeholder backend once network authorization allowed the connection, got $STATUS" + exit 1 +fi +echo "✓ Network authorization allowed a connection matching an allow rule through to HTTP routing" +{{< /doc-test >}} + ## Rules Network authorization supports the same rule types as HTTP authorization: @@ -71,6 +198,16 @@ frontendPolicies: - allow: 'cidr("10.0.0.0/8").containsIP(source.address) || cidr("172.16.0.0/12").containsIP(source.address) || cidr("192.168.0.0/16").containsIP(source.address)' ``` +{{< doc-test paths="network-authz" >}} +cat <<'EOF' > config-private.yaml +frontendPolicies: + networkAuthorization: + rules: + - allow: 'cidr("10.0.0.0/8").containsIP(source.address) || cidr("172.16.0.0/12").containsIP(source.address) || cidr("192.168.0.0/16").containsIP(source.address)' +EOF +agentgateway -f config-private.yaml --validate-only +{{< /doc-test >}} + ### Require mTLS client identity ```yaml @@ -80,6 +217,16 @@ frontendPolicies: - require: 'source.tls.identity == "spiffe://cluster.local/ns/default/sa/my-service"' ``` +{{< doc-test paths="network-authz" >}} +cat <<'EOF' > config-mtls.yaml +frontendPolicies: + networkAuthorization: + rules: + - require: 'source.tls.identity == "spiffe://cluster.local/ns/default/sa/my-service"' +EOF +agentgateway -f config-mtls.yaml --validate-only +{{< /doc-test >}} + ### Layered L4+L7 controls Combine network authorization with HTTP authorization for defense in depth. @@ -102,4 +249,25 @@ routes: - require: 'jwt.aud == "my-service"' ``` +{{< doc-test paths="network-authz" >}} +cat <<'EOF' > config-layered.yaml +frontendPolicies: + networkAuthorization: + rules: + - allow: 'cidr("10.0.0.0/8").containsIP(source.address)' + +gateways: + default: + port: 3000 +routes: +- backends: + - host: localhost:8080 + policies: + authorization: + rules: + - require: 'jwt.aud == "my-service"' +EOF +agentgateway -f config-layered.yaml --validate-only +{{< /doc-test >}} + In this example, only connections from the `10.0.0.0/8` range are accepted at the network level, and those connections must also present a valid JWT with the correct audience claim. diff --git a/content/docs/standalone/latest/configuration/traffic-management/buffer.md b/content/docs/standalone/latest/configuration/traffic-management/buffer.md index b2f53e41e..f8f0423cf 100644 --- a/content/docs/standalone/latest/configuration/traffic-management/buffer.md +++ b/content/docs/standalone/latest/configuration/traffic-management/buffer.md @@ -2,10 +2,41 @@ title: Body buffering weight: 17 description: Buffer request and response bodies before forwarding them. +test: + buffer: + - file: ${versionRoot}/configuration/traffic-management/buffer.md + path: buffer --- Attaches to: {{< badge content="Route" path="/configuration/routes/" >}} +{{< doc-test paths="buffer" >}} +# ============================================================================ +# Doc test coverage for this guide (these comments are not rendered on the page) +# ============================================================================ +# WHAT THIS TEST VALIDATES: +# * "Buffer request and response bodies": the example config is accepted by +# agentgateway (--validate-only), so the `policies.buffer.request.maxBytes` +# and `policies.buffer.response.maxBytes` field names and nesting are correct. +# * The same config serves live traffic: with the policy applied, a GET request +# reaches the backend and returns 200, and a POST request with a body inside +# the `maxBytes` limit is buffered and forwarded to the backend with all of +# its bytes intact (the backend echoes the body back). +# +# WHAT THIS TEST DOES NOT VALIDATE (and why): +# * `failureMode` (`failClosed` / `failOpen`) behavior when a body exceeds +# `maxBytes` - requires config/traffic the page omits; the page documents the +# fields in a table but shows no example that sets `failureMode` or sends an +# oversized body. +# * That bodies are actually accumulated in memory rather than streamed - a +# different layer; the proxy exposes no per-request signal that this page +# documents, so only the end-to-end result is asserted. +# * The `frontendPolicies.http.maxBufferSize` gateway-level limit mentioned in +# the note - display-only reference to a separate setting, with no example on +# this page. +{{< reuse "agw-docs/snippets/install-agentgateway-binary.md" >}} +{{< /doc-test >}} + Use the `policies.buffer` policy to buffer request or response bodies in the proxy before the bodies are forwarded. By default, agentgateway streams bodies. When you configure `policies.buffer`, the proxy accumulates the configured body direction in memory until the body is complete, and then forwards it. > [!NOTE] @@ -43,3 +74,107 @@ routes: response: maxBytes: 262144 ``` + +{{< doc-test paths="buffer" >}} +cat <<'EOF' > config.yaml +# yaml-language-server: $schema=https://agentgateway.dev/schema/config +gateways: + default: + port: 3000 +routes: +- backends: + - host: localhost:8080 + policies: + buffer: + request: + maxBytes: 65536 + response: + maxBytes: 262144 +EOF +agentgateway -f config.yaml --validate-only +{{< /doc-test >}} + +{{< doc-test paths="buffer" >}} +# Stand up an HTTP backend on localhost:8080 so the route in the example config +# has something to forward to. The backend echoes the request body back so the +# test can confirm a buffered POST body arrives intact, then wait for it to +# accept connections. +cat <<'EOF' > backend.py +from http.server import BaseHTTPRequestHandler, HTTPServer + +class Echo(BaseHTTPRequestHandler): + def _reply(self, body=b""): + self.send_response(200) + self.send_header("content-type", "text/plain") + self.send_header("content-length", str(len(body))) + self.end_headers() + self.wfile.write(body) + + def do_GET(self): + self._reply(b"ok") + + def do_POST(self): + length = int(self.headers.get("content-length") or 0) + self._reply(self.rfile.read(length)) + + def log_message(self, *args): + pass + +HTTPServer(("127.0.0.1", 8080), Echo).serve_forever() +EOF +python3 backend.py & +BACKEND_PID=$! +trap 'kill $BACKEND_PID 2>/dev/null' EXIT +# Wait for the echo backend, and confirm the responder is actually this backend +# rather than some other process already holding 8080 -- otherwise the POST +# assertion below fails in a way that looks like a buffering bug. +for i in $(seq 1 30); do + [ "$(curl -sf --max-time 5 -X POST -d probe http://127.0.0.1:8080/ 2>/dev/null)" = "probe" ] && break + sleep 1 +done +if [ "$(curl -sf --max-time 5 -X POST -d probe http://127.0.0.1:8080/ 2>/dev/null)" != "probe" ]; then + echo "FAIL: the echo backend did not come up on 127.0.0.1:8080 (is the port already in use?)" + exit 1 +fi +{{< /doc-test >}} + +{{< doc-test paths="buffer" >}} +agentgateway -f config.yaml & +AGW_PID=$! +trap 'kill $AGW_PID $BACKEND_PID 2>/dev/null' EXIT +sleep 3 +{{< /doc-test >}} + +{{< doc-test paths="buffer" >}} +YAMLTest -f - <<'EOF' +- name: Buffered route forwards a GET request to the backend + retries: 3 + http: + url: "http://localhost:3000" + path: / + method: GET + source: + type: local + expect: + statusCode: 200 +- name: Buffered route forwards a POST request body under maxBytes + http: + url: "http://localhost:3000" + path: / + method: POST + headers: + content-type: text/plain + accept-encoding: identity + body: "buffered request body" + source: + type: local + expect: + statusCode: 200 + headers: + # The backend echoes the request body, so a content-length of 21 confirms + # all 21 bytes of "buffered request body" survived buffering. + - name: content-length + comparator: equals + value: "21" +EOF +{{< /doc-test >}} diff --git a/content/docs/standalone/latest/configuration/traffic-management/route-delegation.md b/content/docs/standalone/latest/configuration/traffic-management/route-delegation.md index 2dd8e931c..2a2787a06 100644 --- a/content/docs/standalone/latest/configuration/traffic-management/route-delegation.md +++ b/content/docs/standalone/latest/configuration/traffic-management/route-delegation.md @@ -2,8 +2,106 @@ title: Route delegation weight: 15 description: Delegate routing decisions to route groups for independent team management. +test: + route-delegation: + - file: ${versionRoot}/configuration/traffic-management/route-delegation.md + path: route-delegation --- +{{< doc-test paths="route-delegation" >}} +# ============================================================================ +# Doc test coverage for this guide (these comments are not rendered on the page) +# ============================================================================ +# WHAT THIS TEST VALIDATES: +# * All six example configs are accepted by agentgateway (--validate-only), +# covering `backends[].routeGroup`, top-level `routeGroups[].routes[]`, nested +# route groups, `policies` on both a parent and a child route, a cyclic +# `routeGroup` reference, and a dangling `routeGroup` reference. +# * "Basic delegation", "Header and query matching", and "Multi-level +# delegation" each get two passes: first with the page's own config (so the +# documented `503`/`404` outcomes for a placeholder backend are verified as +# written), then again with the placeholder hosts swapped for a local echo +# backend, asserting a real `200` for every path that should be delegated. +# This proves a delegated request actually reaches a backend, not just that +# it isn't a 404. +# * "Policy inheritance" step 3: a child with no policy of its own receives the +# parent's `x-parent` request header, and the child that defines its own +# `requestHeaderModifier` receives `x-child` and NOT `x-parent`. This confirms +# the documented precedence rule ("the child's policy takes precedence"). +# * "Cyclic delegation": the two-route-group cycle is accepted by +# --validate-only (the cycle is only caught at request time), and a request +# that walks into it gets the documented `500`. +# * "Missing route group": a route referencing a nonexistent `routeGroup` is +# accepted by --validate-only, and a request to it returns `404`. The details +# table documented `500` for this case until this test was added; the +# product actually returns `404` (`error="route not found" reason=NotFound`, +# the same as an unmatched path) - the table was corrected to match. +# +# WHAT THIS TEST DOES NOT VALIDATE (and why): +# * None of the six configs use TLS, so the exact wording of "the connection +# is reset" vs. an HTTP-level error for non-HTTP failure modes elsewhere in +# agentgateway isn't exercised here - out of scope for this page. +{{< reuse "agw-docs/snippets/install-agentgateway-binary.md" >}} + +# Assert the HTTP status of one documented request. Extra args are passed to curl. +assert_status() { + local desc="$1" expected="$2"; shift 2 + local got + got=$(curl -s -o /dev/null -w '%{http_code}' --max-time 10 "$@") + if [ "$got" != "$expected" ]; then + echo "FAIL: $desc -- expected HTTP $expected but got $got" + exit 1 + fi + echo "✓ $desc -> $expected" +} + +start_gateway() { + agentgateway -f "${1:-config.yaml}" & + AGW_PID=$! + sleep 3 +} + +stop_gateway() { + [ -n "${AGW_PID:-}" ] || return 0 + kill "$AGW_PID" 2>/dev/null || true + wait "$AGW_PID" 2>/dev/null || true + AGW_PID="" +} + +trap 'stop_gateway; [ -n "${BACKEND_PID:-}" ] && kill "$BACKEND_PID" 2>/dev/null || true' EXIT + +# Every example on this page points at a placeholder host (team1-foo.example.com +# and friends). Stand up one local echo backend that later sections point +# swapped-host copies of the page's configs at, so a delegated request can be +# observed reaching a real backend (200) instead of only ever seeing the 503 a +# placeholder host produces. Port 8081, not 8080, so it doesn't collide with +# another page's documented config. +cat <<'PYEOF' > backend.py +from http.server import BaseHTTPRequestHandler, HTTPServer +import json + +class Echo(BaseHTTPRequestHandler): + def do_GET(self): + body = json.dumps({k.lower(): v for k, v in self.headers.items()}).encode() + self.send_response(200) + self.send_header("content-type", "application/json") + self.send_header("content-length", str(len(body))) + self.end_headers() + self.wfile.write(body) + + def log_message(self, *args): + pass + +HTTPServer(("127.0.0.1", 8081), Echo).serve_forever() +PYEOF +python3 backend.py & +BACKEND_PID=$! +for i in $(seq 1 30); do + curl -sf -o /dev/null http://127.0.0.1:8081/ && break + sleep 1 +done +{{< /doc-test >}} + Delegate routing decisions from a parent route to a set of child routes defined in a route group. Route delegation lets you break up large routing configurations into smaller, independently managed pieces. ## About @@ -46,8 +144,8 @@ Review more details about how route delegation works in standalone mode. |---|---| | Parent path matcher | A parent route that delegates to a route group must use a `pathPrefix` matcher. | | Child path scope | Child routes must match a path that falls within the parent's prefix. For example, if the parent matches `/api`, a child must match a path starting with `/api`. | -| Cyclic delegation | Agentgateway does not allow cyclic delegation. If route group A delegates to B, and B delegates back to A, agentgateway detects the cycle at runtime and returns an error. | -| Missing route group | If a route references a `routeGroup` that does not exist, the route is replaced with a 500 HTTP response. | +| Cyclic delegation | Agentgateway does not allow cyclic delegation. If route group A delegates to B, and B delegates back to A, agentgateway detects the cycle at runtime and returns a `500` response. See [Error responses](#error-responses). | +| Missing route group | If a route references a `routeGroup` that does not exist, agentgateway returns a `404` response for that route, the same as a path with no match. See [Error responses](#error-responses). | ## Before you begin @@ -61,7 +159,7 @@ In this example, a parent route matches the `/anything/team1` prefix and delegat 1. Create the configuration file. - ```sh + ```sh {paths="route-delegation"} cat > config.yaml <<'EOF' # yaml-language-server: $schema=https://agentgateway.dev/schema/config gateways: @@ -94,6 +192,11 @@ In this example, a parent route matches the `/anything/team1` prefix and delegat EOF ``` + {{< doc-test paths="route-delegation" >}} + # Basic delegation: validate the config written by step 1 + agentgateway -f config.yaml --validate-only + {{< /doc-test >}} + 2. Run the gateway. ```sh @@ -116,6 +219,24 @@ In this example, a parent route matches the `/anything/team1` prefix and delegat curl -i 127.0.0.1:3000/other ``` +{{< doc-test paths="route-delegation" >}} +start_gateway +assert_status "Basic: /anything/team1/foo is delegated to child-foo" 503 127.0.0.1:3000/anything/team1/foo +assert_status "Basic: /anything/team1/bar is delegated to child-bar" 503 127.0.0.1:3000/anything/team1/bar +assert_status "Basic: parent prefix with no matching child" 404 127.0.0.1:3000/anything/team1/other +assert_status "Basic: path outside the parent prefix" 404 127.0.0.1:3000/other +stop_gateway + +# Confirm a delegated request actually reaches a backend: rerun with the +# placeholder hosts swapped for the local echo backend and expect a real 200. +sed 's#team1-foo.example.com:8080#localhost:8081#; s#team1-bar.example.com:8080#localhost:8081#' \ + config.yaml > config-basic-local.yaml +start_gateway config-basic-local.yaml +assert_status "Basic: /anything/team1/foo reaches the backend" 200 127.0.0.1:3000/anything/team1/foo +assert_status "Basic: /anything/team1/bar reaches the backend" 200 127.0.0.1:3000/anything/team1/bar +stop_gateway +{{< /doc-test >}} + ## Header and query matching Parent routes can include header and query parameter matchers that control which requests are delegated. Child routes can independently define their own matchers. A request must satisfy both the parent's and the child's matchers to reach a backend. @@ -127,7 +248,7 @@ In this example, a parent route matches `/anything/team1` only when the `x-team` 1. Create the configuration file. - ```sh + ```sh {paths="route-delegation"} cat > config.yaml <<'EOF' # yaml-language-server: $schema=https://agentgateway.dev/schema/config gateways: @@ -172,6 +293,11 @@ In this example, a parent route matches `/anything/team1` only when the `x-team` EOF ``` + {{< doc-test paths="route-delegation" >}} + # Header and query matching: validate the config written by step 1 + agentgateway -f config.yaml --validate-only + {{< /doc-test >}} + 2. Run the gateway. ```sh @@ -181,7 +307,7 @@ In this example, a parent route matches `/anything/team1` only when the `x-team` 3. Test the routes. ```sh - # child-foo: parent matchers + child's x-role header -> 200 + # child-foo: parent matchers + child's x-role header -> routed to child-foo curl -i "127.0.0.1:3000/anything/team1/foo?env=prod" \ -H "x-team: team1" -H "x-role: admin" @@ -189,7 +315,7 @@ In this example, a parent route matches `/anything/team1` only when the `x-team` curl -i "127.0.0.1:3000/anything/team1/foo?env=prod" \ -H "x-team: team1" - # child-bar: parent matchers, child matches on path only -> 200 + # child-bar: parent matchers, child matches on path only -> routed to child-bar curl -i "127.0.0.1:3000/anything/team1/bar?env=prod" \ -H "x-team: team1" @@ -197,6 +323,34 @@ In this example, a parent route matches `/anything/team1` only when the `x-team` curl -i 127.0.0.1:3000/anything/team1/bar ``` +{{< doc-test paths="route-delegation" >}} +start_gateway +assert_status "Header/query: parent matchers plus the child's x-role is delegated" 503 \ + "127.0.0.1:3000/anything/team1/foo?env=prod" -H "x-team: team1" -H "x-role: admin" +assert_status "Header/query: parent matchers but missing the child's x-role" 404 \ + "127.0.0.1:3000/anything/team1/foo?env=prod" -H "x-team: team1" +assert_status "Header/query: child-bar matches on path only" 503 \ + "127.0.0.1:3000/anything/team1/bar?env=prod" -H "x-team: team1" +assert_status "Header/query: missing the parent's matchers is not delegated" 404 \ + 127.0.0.1:3000/anything/team1/bar +stop_gateway + +# Confirm a delegated request actually reaches a backend: rerun with the +# placeholder hosts swapped for the local echo backend and expect a real 200. +sed 's#team1-foo.example.com:8080#localhost:8081#; s#team1-bar.example.com:8080#localhost:8081#' \ + config.yaml > config-headerquery-local.yaml +start_gateway config-headerquery-local.yaml +assert_status "Header/query: child-foo reaches the backend" 200 \ + "127.0.0.1:3000/anything/team1/foo?env=prod" -H "x-team: team1" -H "x-role: admin" +assert_status "Header/query: child-bar reaches the backend" 200 \ + "127.0.0.1:3000/anything/team1/bar?env=prod" -H "x-team: team1" +stop_gateway +{{< /doc-test >}} + + The backend hosts in these examples are placeholders, so a request that is + routed to a child returns `503` instead of a response from the backend. The + `404` responses are the ones that show a request was not delegated. + ## Multi-level delegation Child routes inside a route group can delegate to other route groups, creating a multi-level delegation hierarchy. Agentgateway detects cycles at runtime and returns an error if a delegation chain loops back to a previously visited route group. @@ -205,7 +359,7 @@ In this example, a parent route delegates `/api` to a route group. One child han 1. Create the configuration file. - ```sh + ```sh {paths="route-delegation"} cat > config.yaml <<'EOF' # yaml-language-server: $schema=https://agentgateway.dev/schema/config gateways: @@ -252,6 +406,11 @@ In this example, a parent route delegates `/api` to a route group. One child han EOF ``` + {{< doc-test paths="route-delegation" >}} + # Multi-level delegation: validate the config written by step 1 + agentgateway -f config.yaml --validate-only + {{< /doc-test >}} + 2. Run the gateway. ```sh @@ -274,6 +433,26 @@ In this example, a parent route delegates `/api` to a route group. One child han curl -i 127.0.0.1:3000/api/orders/other ``` +{{< doc-test paths="route-delegation" >}} +start_gateway +assert_status "Multi-level: /api/users resolves through api-routes" 503 127.0.0.1:3000/api/users +assert_status "Multi-level: /api/orders/list resolves through two route groups" 503 127.0.0.1:3000/api/orders/list +assert_status "Multi-level: /api/orders/detail resolves through two route groups" 503 127.0.0.1:3000/api/orders/detail +assert_status "Multi-level: child-orders prefix with no matching grandchild" 404 127.0.0.1:3000/api/orders/other +stop_gateway + +# Confirm a delegated request actually reaches a backend at every level of the +# chain: rerun with the three placeholder hosts swapped for the local echo +# backend and expect a real 200. +sed 's#users-service.example.com:8080#localhost:8081#; s#orders-list.example.com:8080#localhost:8081#; s#orders-detail.example.com:8080#localhost:8081#' \ + config.yaml > config-multilevel-local.yaml +start_gateway config-multilevel-local.yaml +assert_status "Multi-level: /api/users reaches the backend" 200 127.0.0.1:3000/api/users +assert_status "Multi-level: /api/orders/list reaches the backend through two route groups" 200 127.0.0.1:3000/api/orders/list +assert_status "Multi-level: /api/orders/detail reaches the backend through two route groups" 200 127.0.0.1:3000/api/orders/detail +stop_gateway +{{< /doc-test >}} + ## Policy inheritance Policies defined on a parent route are inherited by child routes in the delegation chain. If a child route defines the same type of policy, the child's policy takes precedence. @@ -282,7 +461,7 @@ In this example, a parent route sets a `requestHeaderModifier` policy that adds 1. Create the configuration file. - ```sh + ```sh {paths="route-delegation"} cat > config.yaml <<'EOF' # yaml-language-server: $schema=https://agentgateway.dev/schema/config gateways: @@ -323,6 +502,11 @@ In this example, a parent route sets a `requestHeaderModifier` policy that adds EOF ``` + {{< doc-test paths="route-delegation" >}} + # Policy inheritance: validate the config written by step 1 + agentgateway -f config.yaml --validate-only + {{< /doc-test >}} + 2. Run the gateway. ```sh @@ -338,3 +522,153 @@ In this example, a parent route sets a `requestHeaderModifier` policy that adds # child-overrides: receives x-child header; parent's requestHeaderModifier is overridden curl -i 127.0.0.1:3000/anything/team1/bar ``` + +{{< doc-test paths="route-delegation" >}} +# Inherited request headers are only observable at the backend, so this assertion +# runs the page's config with the two placeholder hosts swapped for the shared +# local echo backend (started once, at the top of this test) that echoes the +# request headers it received. +sed 's#team1-foo.example.com:8080#localhost:8081#; s#team1-bar.example.com:8080#localhost:8081#' \ + config.yaml > config-policy-local.yaml +start_gateway config-policy-local.yaml + +INHERITS=$(curl -sf --max-time 10 127.0.0.1:3000/anything/team1/foo) +if [ "$(jq -r '."x-parent" // "absent"' <<<"$INHERITS")" != "from-parent" ]; then + echo "FAIL: child-inherits did not receive the parent's x-parent header" + echo "$INHERITS" + exit 1 +fi +echo "✓ Policy inheritance: child-inherits received x-parent from the parent route" + +OVERRIDES=$(curl -sf --max-time 10 127.0.0.1:3000/anything/team1/bar) +if [ "$(jq -r '."x-child" // "absent"' <<<"$OVERRIDES")" != "from-child" ]; then + echo "FAIL: child-overrides did not receive its own x-child header" + echo "$OVERRIDES" + exit 1 +fi +if [ "$(jq -r '."x-parent" // "absent"' <<<"$OVERRIDES")" != "absent" ]; then + echo "FAIL: child-overrides should override the parent policy, but x-parent was still added" + echo "$OVERRIDES" + exit 1 +fi +echo "✓ Policy inheritance: child-overrides received x-child and not x-parent" +stop_gateway +{{< /doc-test >}} + +## Error responses + +Two invalid delegation configurations produce specific error responses, rather than being rejected at validation time. + +### Cyclic delegation + +Agentgateway does not allow cyclic delegation. If route group A delegates to B, and B delegates back to A, agentgateway detects the cycle at runtime and returns a `500` response. The cycle is not caught by `--validate-only`, because static validation does not follow `routeGroup` references. + +1. Create the configuration file. + + ```sh {paths="route-delegation"} + cat > config-cycle.yaml <<'EOF' + # yaml-language-server: $schema=https://agentgateway.dev/schema/config + gateways: + default: + port: 3000 + protocol: HTTP + routes: + - name: parent-a + matches: + - path: + pathPrefix: /a + backends: + - routeGroup: group-a + + routeGroups: + - name: group-a + routes: + - name: to-b + matches: + - path: + pathPrefix: /a + backends: + - routeGroup: group-b + - name: group-b + routes: + - name: to-a + matches: + - path: + pathPrefix: /a + backends: + - routeGroup: group-a + EOF + ``` + + {{< doc-test paths="route-delegation" >}} + # Cyclic delegation: validate the config written by step 1. --validate-only + # succeeds because the cycle is only detected at request time. + agentgateway -f config-cycle.yaml --validate-only + {{< /doc-test >}} + +2. Run the gateway. + + ```sh + agentgateway -f config-cycle.yaml + ``` + +3. Test the route. + + ```sh + # group-a -> group-b -> group-a is a cycle -> 500 + curl -i 127.0.0.1:3000/a + ``` + +{{< doc-test paths="route-delegation" >}} +start_gateway config-cycle.yaml +assert_status "Cyclic delegation is detected at runtime and returns 500" 500 127.0.0.1:3000/a +stop_gateway +{{< /doc-test >}} + +### Missing route group + +If a route references a `routeGroup` that does not exist, agentgateway returns a `404` response for that route, the same as a path with no match. + +1. Create the configuration file. + + ```sh {paths="route-delegation"} + cat > config-missing-group.yaml <<'EOF' + # yaml-language-server: $schema=https://agentgateway.dev/schema/config + gateways: + default: + port: 3000 + protocol: HTTP + routes: + - name: parent-missing + matches: + - path: + pathPrefix: /missing + backends: + - routeGroup: does-not-exist + EOF + ``` + + {{< doc-test paths="route-delegation" >}} + # Missing route group: validate the config written by step 1. --validate-only + # succeeds because the dangling reference is only resolved at request time. + agentgateway -f config-missing-group.yaml --validate-only + {{< /doc-test >}} + +2. Run the gateway. + + ```sh + agentgateway -f config-missing-group.yaml + ``` + +3. Test the route. + + ```sh + # does-not-exist is not a defined routeGroup -> 404 + curl -i 127.0.0.1:3000/missing + ``` + +{{< doc-test paths="route-delegation" >}} +start_gateway config-missing-group.yaml +assert_status "A route referencing a nonexistent route group returns 404" 404 127.0.0.1:3000/missing +stop_gateway +{{< /doc-test >}} diff --git a/content/docs/standalone/latest/integrations/llm-providers/anthropic.md b/content/docs/standalone/latest/integrations/llm-providers/anthropic.md index 34114093d..ee8b25b13 100644 --- a/content/docs/standalone/latest/integrations/llm-providers/anthropic.md +++ b/content/docs/standalone/latest/integrations/llm-providers/anthropic.md @@ -2,6 +2,7 @@ title: Anthropic weight: 20 description: Connect agentgateway to Anthropic's Claude models +test: skip --- {{< redirect path="/llm/providers/anthropic/" >}} diff --git a/content/docs/standalone/latest/integrations/llm-providers/azure-openai.md b/content/docs/standalone/latest/integrations/llm-providers/azure-openai.md index af34eab2c..ef48a6d3c 100644 --- a/content/docs/standalone/latest/integrations/llm-providers/azure-openai.md +++ b/content/docs/standalone/latest/integrations/llm-providers/azure-openai.md @@ -2,6 +2,7 @@ title: Azure OpenAI weight: 30 description: Connect agentgateway to Azure-hosted OpenAI models +test: skip --- {{< redirect path="/llm/providers/azure/" >}} diff --git a/content/docs/standalone/latest/integrations/llm-providers/bedrock.md b/content/docs/standalone/latest/integrations/llm-providers/bedrock.md index 78841572f..b93c2e876 100644 --- a/content/docs/standalone/latest/integrations/llm-providers/bedrock.md +++ b/content/docs/standalone/latest/integrations/llm-providers/bedrock.md @@ -2,6 +2,7 @@ title: Amazon Bedrock weight: 40 description: Connect agentgateway to AWS foundation models via Amazon Bedrock +test: skip --- {{< redirect path="/llm/providers/bedrock/" >}} diff --git a/content/docs/standalone/latest/integrations/llm-providers/gemini.md b/content/docs/standalone/latest/integrations/llm-providers/gemini.md index 0a9e0fd61..5e332ccc4 100644 --- a/content/docs/standalone/latest/integrations/llm-providers/gemini.md +++ b/content/docs/standalone/latest/integrations/llm-providers/gemini.md @@ -2,6 +2,7 @@ title: Google Gemini weight: 50 description: Connect agentgateway to Google's Gemini models +test: skip --- {{< redirect path="/llm/providers/gemini/" >}} diff --git a/content/docs/standalone/latest/integrations/llm-providers/openai-compatible.md b/content/docs/standalone/latest/integrations/llm-providers/openai-compatible.md index 56198a729..877a4ef8b 100644 --- a/content/docs/standalone/latest/integrations/llm-providers/openai-compatible.md +++ b/content/docs/standalone/latest/integrations/llm-providers/openai-compatible.md @@ -2,6 +2,7 @@ title: OpenAI-Compatible Providers weight: 70 description: Connect agentgateway to any OpenAI-compatible API (xAI, Cohere, Ollama, etc.) +test: skip --- {{< redirect path="/llm/providers/custom/" >}} diff --git a/content/docs/standalone/latest/integrations/llm-providers/openai.md b/content/docs/standalone/latest/integrations/llm-providers/openai.md index 95df503c2..2b9f14a60 100644 --- a/content/docs/standalone/latest/integrations/llm-providers/openai.md +++ b/content/docs/standalone/latest/integrations/llm-providers/openai.md @@ -2,6 +2,7 @@ title: OpenAI weight: 10 description: Connect agentgateway to OpenAI's GPT models +test: skip --- {{< redirect path="/llm/providers/openai/" >}} diff --git a/content/docs/standalone/latest/integrations/llm-providers/vertex.md b/content/docs/standalone/latest/integrations/llm-providers/vertex.md index 2950ce526..9907418ff 100644 --- a/content/docs/standalone/latest/integrations/llm-providers/vertex.md +++ b/content/docs/standalone/latest/integrations/llm-providers/vertex.md @@ -2,6 +2,7 @@ title: Vertex AI weight: 60 description: Connect agentgateway to Google Cloud's Vertex AI platform +test: skip --- {{< redirect path="/llm/providers/vertex/" >}} diff --git a/content/docs/standalone/latest/integrations/llm-providers/xai.md b/content/docs/standalone/latest/integrations/llm-providers/xai.md index b7a45f2bc..b816786a0 100644 --- a/content/docs/standalone/latest/integrations/llm-providers/xai.md +++ b/content/docs/standalone/latest/integrations/llm-providers/xai.md @@ -2,6 +2,7 @@ title: xAI (Grok) weight: 75 description: Connect agentgateway to xAI's Grok models +test: skip --- {{< redirect path="/llm/providers/xai/" >}} diff --git a/content/docs/standalone/latest/llm/about.md b/content/docs/standalone/latest/llm/about.md index fe16d6fb2..7f047a8a8 100644 --- a/content/docs/standalone/latest/llm/about.md +++ b/content/docs/standalone/latest/llm/about.md @@ -150,7 +150,7 @@ Use `name: "*"` without setting `params.model` to accept any model name and pass llm: models: - name: "*" - provider: openai + provider: openAI params: apiKey: "$OPENAI_API_KEY" ``` @@ -166,7 +166,7 @@ This is the recommended approach when you want to expose all models from multipl llm: models: - name: "*" - provider: openai + provider: openAI params: apiKey: "$OPENAI_API_KEY" transformation: diff --git a/content/docs/standalone/latest/llm/configuration-modes.md b/content/docs/standalone/latest/llm/configuration-modes.md index 6ccc3ac4c..61f31a629 100644 --- a/content/docs/standalone/latest/llm/configuration-modes.md +++ b/content/docs/standalone/latest/llm/configuration-modes.md @@ -98,6 +98,11 @@ llm: To set the port and TLS settings for LLM traffic, define a gateway and attach the `llm` section to it. When you omit the `gateways` field, the `llm` section attaches to the gateway named `default`. The `mcp` and `ui` sections attach the same way, so all three can share one port. +When your configuration file defines no gateway at all, such as the earlier basic example, the implied `default` gateway serves LLM traffic on port `4000` and MCP traffic on port `3000`. Requests use the OpenAI-compatible paths, such as `http://localhost:4000/v1/chat/completions`. + +> [!NOTE] +> The `llm.port`, `llm.tls`, and `mcp.port` fields are deprecated in favor of gateways. They still work, and setting them overrides these defaults. + Use the gateway's `tls` field to serve LLM traffic over TLS. - Most deployments only need `cert` and `key`. - Use `root` for a custom trust bundle or mTLS. @@ -125,9 +130,6 @@ mcp: args: ["@modelcontextprotocol/server-everything"] ``` -> [!NOTE] -> The `llm.port`, `llm.tls`, and `mcp.port` fields are deprecated in favor of gateways. They still work, and when you set them without a gateway, LLM traffic defaults to port `4000` and MCP traffic to port `3000`. - For more MCP listener context, see [MCP overview]({{< link-hextra path="/mcp/" >}}). ## Routing-based configuration diff --git a/content/docs/standalone/latest/llm/prompt-guards/regex.md b/content/docs/standalone/latest/llm/prompt-guards/regex.md index 106a4cb6e..2b23a5454 100644 --- a/content/docs/standalone/latest/llm/prompt-guards/regex.md +++ b/content/docs/standalone/latest/llm/prompt-guards/regex.md @@ -2,10 +2,59 @@ title: Regex filters weight: 10 description: Match and redact prompt content with custom regex patterns or agentgateway's built-in PII detectors. +test: + regex: + - file: ${versionRoot}/llm/prompt-guards/regex.md + path: regex --- Use custom regex patterns and built-in PII detectors to filter LLM requests and responses. +{{< doc-test paths="regex" >}} +# ============================================================================ +# Doc test coverage for this guide (these comments are not rendered on the page) +# ============================================================================ +# WHAT THIS TEST VALIDATES: +# * "Custom regex patterns": the credential-matching example config is accepted +# by agentgateway (--validate-only), covering `guardrails.request[].regex` +# with `action: reject`, `rules[].pattern`, and a `rejection` block that sets +# a status, headers, and body. +# * "PII detection" step 1: the config with both a custom-pattern rule and a +# `builtin: email` rule is accepted. +# * "PII detection" step 4: a request containing the SSN keyword is rejected +# with the documented status (400) and the exact documented error body +# (`content_policy_violation`). The `Social Security` pattern from the same +# rule is checked too, which the page describes but does not demonstrate. +# * "PII detection" step 5: a request containing an email address is rejected by +# the built-in `email` pattern with the documented `pii_detected` body, +# confirming the built-in patterns table is wired up and that the second +# guardrail is evaluated independently of the first. +# * "PII detection" step 3, partially: a prompt that matches no rule is NOT +# blocked by the guard. The test asserts the response is not a guard rejection +# rather than asserting success, so it holds whether or not a real API key is +# present. +# * "Mask PII in responses" step 1: the `action: mask` config with +# `builtin: phoneNumber` is accepted. +# +# WHAT THIS TEST DOES NOT VALIDATE (and why): +# * The successful completion in "PII detection" step 3 and its example output - +# external dependency; a real response needs a live OpenAI key and bills a +# completion. Only that the guard does not block the request is asserted. +# * "Mask PII in responses" steps 2-3, including the `` +# replacement - external dependency; masking operates on a real LLM response +# body, so there is nothing to redact without a live provider call. The config +# is validated but the mask behavior is not. +# * The other built-in patterns (`phoneNumber`, `ssn`, `creditCard`, `caSin`) as +# request filters - display-only table rows; only `email` appears in a runnable +# example on this page. +{{< reuse "agw-docs/snippets/install-agentgateway-binary.md" >}} + +# The example configs read the API key from the environment. Guard rejections +# happen before any upstream call, so a placeholder is enough for the assertions +# below; CI supplies a real key when one is available. +export OPENAI_API_KEY="${OPENAI_API_KEY:-test}" +{{< /doc-test >}} + ## About regex prompt templating Regex-based prompt guards let you inspect LLM requests and responses against custom regex patterns or built-in PII detectors. Use the `reject` action to block requests that match a pattern, or the `mask` action to redact sensitive data in responses before they reach the client. @@ -57,6 +106,40 @@ llm: } ``` +{{< doc-test paths="regex" >}} +cat <<'EOF' > config-custom.yaml +llm: + models: + - name: "*" + provider: openAI + params: + model: gpt-4o-mini + apiKey: "$OPENAI_API_KEY" + guardrails: + request: + - regex: + action: reject + rules: + - pattern: "password[=:]\\s*\\S+" + - pattern: "api[_-]?key[=:]\\s*\\S+" + - pattern: "secret[=:]\\s*\\S+" + rejection: + status: 400 + headers: + set: + content-type: "application/json" + body: | + { + "error": { + "message": "Request contains credentials", + "type": "invalid_request_error", + "code": "credentials_detected" + } + } +EOF +agentgateway -f config-custom.yaml --validate-only +{{< /doc-test >}} + ## Before you begin {{< reuse "agw-docs/snippets/prereq-agentgateway.md" >}} @@ -66,7 +149,7 @@ llm: The following example rejects requests that contain PII data, such as Social Security Numbers (using a custom keyword pattern) or email addresses (using the built-in `email` pattern). When a request is blocked, agentgateway returns a custom error response. 1. Create a configuration file with regex prompt guard policies. - ```yaml + ```yaml {paths="regex"} cat <<'EOF' > config.yaml # yaml-language-server: $schema=https://agentgateway.dev/schema/config llm: @@ -195,6 +278,108 @@ The following example rejects requests that contain PII data, such as Social Sec } ``` +{{< doc-test paths="regex" >}} +# Validate the config written by step 1, then run it in the background so the +# step 4 and step 5 requests can be asserted. The visible "Start the agentgateway" +# block is untagged because it runs in the foreground. +agentgateway -f config.yaml --validate-only + +agentgateway -f config.yaml & +AGW_PID=$! +trap 'kill $AGW_PID 2>/dev/null' EXIT +sleep 3 +{{< /doc-test >}} + +{{< doc-test paths="regex" >}} +YAMLTest -f - <<'EOF' +# Guard rejections are produced by agentgateway before the request reaches the +# provider, so these assertions hold with a placeholder API key. +- name: Step 4 - a request containing the SSN keyword is rejected + retries: 3 + http: + url: "http://localhost:4000" + path: /v1/chat/completions + method: POST + headers: + content-type: application/json + accept-encoding: identity + body: | + {"model":"gpt-4o-mini","messages":[{"role":"user","content":"My SSN is 123-45-6789"}]} + source: + type: local + expect: + statusCode: 400 + headers: + - name: content-type + comparator: contains + value: application/json + bodyJsonPath: + - path: "$.error.code" + comparator: equals + value: content_policy_violation + - path: "$.error.message" + comparator: equals + value: "Request rejected: Content contains sensitive information" + - path: "$.error.type" + comparator: equals + value: invalid_request_error +- name: Step 4 rule - the Social Security pattern in the same rule also rejects + http: + url: "http://localhost:4000" + path: /v1/chat/completions + method: POST + headers: + content-type: application/json + accept-encoding: identity + body: | + {"model":"gpt-4o-mini","messages":[{"role":"user","content":"my Social Security number"}]} + source: + type: local + expect: + statusCode: 400 + bodyJsonPath: + - path: "$.error.code" + comparator: equals + value: content_policy_violation +- name: Step 5 - a request containing an email is rejected by the builtin pattern + http: + url: "http://localhost:4000" + path: /v1/chat/completions + method: POST + headers: + content-type: application/json + accept-encoding: identity + body: | + {"model":"gpt-4o-mini","messages":[{"role":"user","content":"Contact me at test@example.com"}]} + source: + type: local + expect: + statusCode: 400 + bodyJsonPath: + - path: "$.error.code" + comparator: equals + value: pii_detected + - path: "$.error.message" + comparator: equals + value: "Request blocked: Contains email address" +EOF +{{< /doc-test >}} + +{{< doc-test paths="regex" >}} +# Step 3: confirm a prompt that matches no rule is not blocked by the guard. The +# assertion is negative rather than a 200 check, because without a real API key the +# upstream returns an auth error -- either way the guard must not have rejected it. +CLEAN=$(curl -s --max-time 15 http://localhost:4000/v1/chat/completions \ + -H "Content-Type: application/json" \ + -d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"Hello, how are you?"}]}') +if grep -qE 'content_policy_violation|pii_detected' <<<"$CLEAN"; then + echo "FAIL: a prompt matching no regex rule was blocked by the prompt guard" + echo "$CLEAN" + exit 1 +fi +echo "✓ A prompt matching no regex rule was not blocked by the prompt guard" +{{< /doc-test >}} + ## Mask PII in responses You can also filter LLM responses to redact sensitive data before it reaches the client. When a match is found, agentgateway replaces built-in pattern matches with `` (for example, ``) and custom pattern matches with ``. The following example masks credit card numbers in responses. @@ -256,3 +441,25 @@ You can also filter LLM responses to redact sensitive data before it reaches the "system_fingerprint":"fp_a1ddba3226"}% ``` + +{{< doc-test paths="regex" >}} +# The mask config is written to its own file so it does not overwrite the config.yaml +# that the running gateway (and the assertions above) depend on. +cat <<'EOF' > config-mask.yaml +# yaml-language-server: $schema=https://agentgateway.dev/schema/config +llm: + models: + - name: "*" + provider: openAI + params: + model: gpt-4o-mini + apiKey: "$OPENAI_API_KEY" + guardrails: + response: + - regex: + action: mask + rules: + - builtin: phoneNumber +EOF +agentgateway -f config-mask.yaml --validate-only +{{< /doc-test >}} diff --git a/content/docs/standalone/latest/llm/providers/anthropic.md b/content/docs/standalone/latest/llm/providers/anthropic.md index 427b428b6..757f9c94a 100644 --- a/content/docs/standalone/latest/llm/providers/anthropic.md +++ b/content/docs/standalone/latest/llm/providers/anthropic.md @@ -3,10 +3,43 @@ title: Anthropic weight: 15 icon: /integrations/providers/bw/anthropic.svg description: Route agentgateway LLM traffic to Anthropic's Claude models. +test: + anthropic: + - file: ${versionRoot}/llm/providers/anthropic.md + path: anthropic --- Configure Anthropic (Claude models) as an LLM provider in agentgateway. +{{< doc-test paths="anthropic" >}} +# ============================================================================ +# Doc test coverage for this guide (these comments are not rendered on the page) +# ============================================================================ +# WHAT THIS TEST VALIDATES: +# * "Configuration": the API key example config is accepted by agentgateway +# (--validate-only), so `provider: anthropic` is recognized. +# * "Use Claude Platform on AWS", both tabs: the API-key config (with +# `requestHeaders.set` and a `params.baseUrl` override) and the AWS SigV4 +# config (with `params.awsRegion` and `auth.aws.serviceName`) are both +# accepted. +# * With the base config loaded, agentgateway serves the wildcard model and +# resolves it to the `anthropic` provider. +# +# WHAT THIS TEST DOES NOT VALIDATE (and why): +# * "Example request", "Token counting", "Extended thinking and reasoning", and +# "Structured outputs" - external dependency; each needs a real Anthropic API +# key and bills live completions. Their example responses are display-only. +# * Claude Platform on AWS at runtime - external dependency; reaching +# aws-external-anthropic needs real AWS credentials and an Anthropic +# workspace. +# * The `thinking` and `output_config` field tables - display-only table rows +# describing request bodies, with no runnable config on this page. +{{< reuse "agw-docs/snippets/install-agentgateway-binary.md" >}} + +export ANTHROPIC_API_KEY="${ANTHROPIC_API_KEY:-test}" +export ANTHROPIC_AWS_API_KEY="${ANTHROPIC_AWS_API_KEY:-test}" +{{< /doc-test >}} + ## Configuration For the common API key case, use the following config. Use the AWS SigV4 section later in the page only when you need Claude Platform on AWS or custom signing behavior. @@ -24,6 +57,20 @@ llm: apiKey: "$ANTHROPIC_API_KEY" ``` +{{< doc-test paths="anthropic" >}} +cat <<'EOF' > config.yaml +# yaml-language-server: $schema=https://agentgateway.dev/schema/config + +llm: + models: + - name: "*" + provider: anthropic + params: + apiKey: "$ANTHROPIC_API_KEY" +EOF +agentgateway -f config.yaml --validate-only +{{< /doc-test >}} + {{< reuse "agw-docs/snippets/review-configuration.md" >}} | Setting | Description | @@ -257,6 +304,26 @@ llm: baseUrl: https://aws-external-anthropic.us-west-2.api.aws/v1 ``` +{{< doc-test paths="anthropic" >}} +cat <<'EOF' > config-claude-platform-apikey.yaml +# yaml-language-server: $schema=https://agentgateway.dev/schema/config + +llm: + models: + - name: "*" + provider: anthropic + requestHeaders: + set: + # Replace with your workspace ID + anthropic-workspace-id: wrkspc_XXXXX + params: + apiKey: $ANTHROPIC_AWS_API_KEY + # Replace with your region + baseUrl: https://aws-external-anthropic.us-west-2.api.aws/v1 +EOF +agentgateway -f config-claude-platform-apikey.yaml --validate-only +{{< /doc-test >}} + | Setting | Description | |---------------------------------------------|-------------| | `requestHeaders.set.anthropic-workspace-id` | The Anthropic workspace ID that scopes the request. Replace `wrkspc_XXXXX` with your workspace ID. | @@ -287,6 +354,27 @@ llm: serviceName: aws-external-anthropic ``` +{{< doc-test paths="anthropic" >}} +cat <<'EOF' > config-claude-platform-sigv4.yaml +# yaml-language-server: $schema=https://agentgateway.dev/schema/config + +llm: + models: + - name: "claude-platform/*" + provider: anthropic + requestHeaders: + set: + anthropic-workspace-id: wrkspc_XXXXX + params: + awsRegion: us-west-2 + baseUrl: https://aws-external-anthropic.us-west-2.api.aws/v1 + auth: + aws: + serviceName: aws-external-anthropic +EOF +agentgateway -f config-claude-platform-sigv4.yaml --validate-only +{{< /doc-test >}} + | Setting | Description | |---------|-------------| | `name` | Matches model names that start with `claude-platform/`, so you can route Claude Platform traffic alongside other Anthropic models. | @@ -306,3 +394,29 @@ For setup instructions, see [Use Claude models on Azure AI Foundry]({{< link-hex ## Connect to Claude Code To route Claude Code CLI traffic through agentgateway, see the [Claude Code integration guide]({{< link-hextra path="/integrations/llm-clients/claude-code" >}}). + +{{< doc-test paths="anthropic" >}} +# Confirm the base API key config serves the wildcard model and resolves it to +# the anthropic provider. +agentgateway -f config.yaml & +AGW_PID=$! +trap 'kill $AGW_PID 2>/dev/null' EXIT +sleep 3 + +SERVED=$(curl -sf --max-time 10 http://localhost:4000/v1/models | jq -r '[.data[].id] | index("*") // "missing"') +if [ "$SERVED" = "missing" ]; then + echo "FAIL: the wildcard model from the example config is not served" + exit 1 +fi +PROVIDER=$(curl -sf --max-time 10 http://localhost:15000/config_dump | jq -r ' + [ .backends[].backend.ai + | select(. != null) + | .target.providers[].active[].endpoint + | .provider | keys[0] + ] | first') +if [ "$PROVIDER" != "anthropic" ]; then + echo "FAIL: expected provider anthropic but agentgateway resolved $PROVIDER" + exit 1 +fi +echo "✓ The wildcard model is served and resolves to the anthropic provider" +{{< /doc-test >}} diff --git a/content/docs/standalone/latest/llm/providers/azure.md b/content/docs/standalone/latest/llm/providers/azure.md index 5cd9537cf..826ef24cb 100644 --- a/content/docs/standalone/latest/llm/providers/azure.md +++ b/content/docs/standalone/latest/llm/providers/azure.md @@ -3,10 +3,48 @@ title: Azure weight: 15 icon: /integrations/providers/bw/azure.svg description: Route agentgateway LLM traffic to models hosted on Microsoft Azure AI. +test: + azure: + - file: ${versionRoot}/llm/providers/azure.md + path: azure --- Configure Microsoft Azure AI as an LLM provider in agentgateway. +{{< doc-test paths="azure" >}} +# ============================================================================ +# Doc test coverage for this guide (these comments are not rendered on the page) +# ============================================================================ +# WHAT THIS TEST VALIDATES: +# * "Configuration", all three tabs: the Foundry implicit-auth, Foundry API-key +# (`auth.key.location.header`), and Azure OpenAI configs are accepted by +# agentgateway (--validate-only), covering `params.azureResourceName`, +# `params.azureResourceType`, and `params.azureProjectName`. +# * "Advanced configuration", all six tabs: the routing-based configs for +# implicit auth, client secret (Foundry and Azure OpenAI), system-assigned and +# user-assigned managed identity, and workload identity are all accepted, +# covering every `policies.backendAuth.azure.explicitConfig` variant the page +# documents. +# * "Use Claude models on Azure AI Foundry": the routing-based Claude config is +# accepted. This example was missing its `gateways` and `routes` keys until +# this test was added, so it could not have been run as written. +# * With the Foundry implicit-auth config loaded, agentgateway serves the +# wildcard model and resolves it to the `azure` provider. +# +# WHAT THIS TEST DOES NOT VALIDATE (and why): +# * Any authentication method at runtime - external dependency; each needs a +# real Azure tenant, resource, and identity (Entra ID, service principal, +# managed identity, or workload identity), none of which the test can stand +# up. Only that agentgateway accepts each config shape is asserted. +# * The verification curl at the end of the Claude Foundry section - external +# dependency, as above. +# * `params.azureApiVersion` - display-only table row; no example on this page +# sets it. +{{< reuse "agw-docs/snippets/install-agentgateway-binary.md" >}} + +export AZURE_API_KEY="${AZURE_API_KEY:-test}" +{{< /doc-test >}} + ## Authentication Before you can use Azure as an LLM provider, you must authenticate by using one of the standard [Azure authentication methods](https://learn.microsoft.com/en-us/azure/ai-services/authentication). In standalone mode, this authentication is configured with `llm.models[]` fields (for example, `params.apiKey` or `auth.azure`). In routing-based configurations, use `policies.backendAuth.azure`. @@ -36,6 +74,21 @@ llm: azureProjectName: "your-project-name" ``` +{{< doc-test paths="azure" >}} +cat <<'EOF' > config-foundry-implicit.yaml +# yaml-language-server: $schema=https://agentgateway.dev/schema/config +llm: + models: + - name: "*" + provider: azure + params: + azureResourceName: "your-resource-name" + azureResourceType: foundry + azureProjectName: "your-project-name" +EOF +agentgateway -f config-foundry-implicit.yaml --validate-only +{{< /doc-test >}} + {{% /tab %}} {{% tab name="Foundry (API key)" %}} @@ -57,6 +110,27 @@ llm: azureProjectName: "your-project-name" ``` +{{< doc-test paths="azure" >}} +cat <<'EOF' > config-foundry-apikey.yaml +# yaml-language-server: $schema=https://agentgateway.dev/schema/config +llm: + models: + - name: "gpt-4.1" + provider: azure + auth: + key: + value: "$AZURE_API_KEY" + location: + header: + name: api-key + params: + azureResourceName: "your-resource-name" + azureResourceType: foundry + azureProjectName: "your-project-name" +EOF +agentgateway -f config-foundry-apikey.yaml --validate-only +{{< /doc-test >}} + {{% /tab %}} {{% tab name="Azure OpenAI (implicit auth)" %}} @@ -71,6 +145,20 @@ llm: azureResourceType: openAI ``` +{{< doc-test paths="azure" >}} +cat <<'EOF' > config-azure-openai.yaml +# yaml-language-server: $schema=https://agentgateway.dev/schema/config +llm: + models: + - name: "gpt-4.1" + provider: azure + params: + azureResourceName: "your-resource-name" + azureResourceType: openAI +EOF +agentgateway -f config-azure-openai.yaml --validate-only +{{< /doc-test >}} + {{% /tab %}} {{< /tabs >}} @@ -116,6 +204,29 @@ routes: model: gpt-4.1 ``` +{{< doc-test paths="azure" >}} +cat <<'EOF' > config-adv-foundry-implicit.yaml +# yaml-language-server: $schema=https://agentgateway.dev/schema/config +gateways: + default: + port: 3000 +routes: +- matches: + - path: + pathPrefix: /azure + backends: + - ai: + name: azure + provider: + azure: + resourceName: "your-resource-name" + projectName: "your-project-name" + resourceType: foundry + model: gpt-4.1 +EOF +agentgateway -f config-adv-foundry-implicit.yaml --validate-only +{{< /doc-test >}} + {{< reuse "agw-docs/snippets/review-configuration.md" >}} {{< reuse-append "agw-docs/snippets/provider-azure-base-configuration.md" >}} | `backendAuth.azure.implicit` | Use implicit authentication via `DefaultAzureCredential`, which automatically detects credentials from the environment. | @@ -153,6 +264,37 @@ routes: model: gpt-4.1 ``` +{{< doc-test paths="azure" >}} +cat <<'EOF' > config-adv-foundry-client-secret.yaml +# yaml-language-server: $schema=https://agentgateway.dev/schema/config +gateways: + default: + port: 3000 +routes: +- matches: + - path: + pathPrefix: /azure + policies: + backendAuth: + azure: + explicitConfig: + clientSecret: + tenant_id: "" + client_id: "" + client_secret: "" + backends: + - ai: + name: azure + provider: + azure: + resourceName: "your-resource-name" + projectName: "your-project-name" + resourceType: foundry + model: gpt-4.1 +EOF +agentgateway -f config-adv-foundry-client-secret.yaml --validate-only +{{< /doc-test >}} + {{< reuse "agw-docs/snippets/review-configuration.md" >}} {{< reuse-append "agw-docs/snippets/provider-azure-base-configuration.md" >}} | `backendAuth.azure.explicitConfig.clientSecret` | Use Azure service principal authentication with tenant ID, client ID, and client secret. | @@ -185,6 +327,33 @@ routes: client_secret: "" ``` +{{< doc-test paths="azure" >}} +cat <<'EOF' > config-adv-client-secret.yaml +# yaml-language-server: $schema=https://agentgateway.dev/schema/config +gateways: + default: + port: 3000 +routes: +- backends: + - ai: + name: azure + provider: + azure: + resourceName: "your-resource-name" + resourceType: openAI + model: gpt-4.1 + policies: + backendAuth: + azure: + explicitConfig: + clientSecret: + tenant_id: "" + client_id: "" + client_secret: "" +EOF +agentgateway -f config-adv-client-secret.yaml --validate-only +{{< /doc-test >}} + {{< reuse "agw-docs/snippets/review-configuration.md" >}} {{< reuse-append "agw-docs/snippets/provider-azure-base-configuration.md" >}} | `backendAuth.azure.explicitConfig.clientSecret` | Use Azure service principal authentication with tenant ID, client ID, and client secret. | @@ -221,6 +390,30 @@ routes: managedIdentity: {} ``` +{{< doc-test paths="azure" >}} +cat <<'EOF' > config-adv-system-managed-identity.yaml +# yaml-language-server: $schema=https://agentgateway.dev/schema/config +gateways: + default: + port: 3000 +routes: +- backends: + - ai: + name: azure + provider: + azure: + resourceName: "your-resource-name" + resourceType: openAI + model: gpt-4.1 + policies: + backendAuth: + azure: + explicitConfig: + managedIdentity: {} +EOF +agentgateway -f config-adv-system-managed-identity.yaml --validate-only +{{< /doc-test >}} + {{< reuse "agw-docs/snippets/review-configuration.md" >}} {{< reuse-append "agw-docs/snippets/provider-azure-base-configuration.md" >}} | `backendAuth.azure.explicitConfig.managedIdentity` | Use Azure managed identity. Leave empty for system-assigned, or specify `userAssignedIdentity` with `clientId`, `objectId`, or `resourceId`. | @@ -263,6 +456,35 @@ routes: # resourceId: "/subscriptions/.../resourceGroups/.../providers/Microsoft.ManagedIdentity/userAssignedIdentities/..." ``` +{{< doc-test paths="azure" >}} +cat <<'EOF' > config-adv-user-managed-identity.yaml +# yaml-language-server: $schema=https://agentgateway.dev/schema/config +gateways: + default: + port: 3000 +routes: +- backends: + - ai: + name: azure + provider: + azure: + resourceName: "your-resource-name" + resourceType: openAI + model: gpt-4.1 + policies: + backendAuth: + azure: + explicitConfig: + managedIdentity: + userAssignedIdentity: + clientId: "" + # OR use objectId or resourceId instead + # objectId: "your-managed-identity-object-id" + # resourceId: "/subscriptions/.../resourceGroups/.../providers/Microsoft.ManagedIdentity/userAssignedIdentities/..." +EOF +agentgateway -f config-adv-user-managed-identity.yaml --validate-only +{{< /doc-test >}} + {{< reuse "agw-docs/snippets/review-configuration.md" >}} {{< reuse-append "agw-docs/snippets/provider-azure-base-configuration.md" >}} | `backendAuth.azure.explicitConfig.managedIdentity` | Use Azure managed identity. Leave empty for system-assigned, or specify `userAssignedIdentity` with `clientId`, `objectId`, or `resourceId`. | @@ -299,6 +521,31 @@ routes: backendTLS: {} ``` +{{< doc-test paths="azure" >}} +cat <<'EOF' > config-adv-workload-identity.yaml +# yaml-language-server: $schema=https://agentgateway.dev/schema/config +gateways: + default: + port: 3000 +routes: +- backends: + - ai: + name: azure + provider: + azure: + resourceName: "your-resource-name" + resourceType: openAI + model: gpt-4.1 + policies: + backendAuth: + azure: + explicitConfig: + workloadIdentity: {} + backendTLS: {} +EOF +agentgateway -f config-adv-workload-identity.yaml --validate-only +{{< /doc-test >}} + {{< reuse "agw-docs/snippets/review-configuration.md" >}} {{< reuse-append "agw-docs/snippets/provider-azure-base-configuration.md" >}} | `backendAuth.azure.explicitConfig.workloadIdentity` | Use Azure workload identity for Kubernetes environments. | @@ -318,6 +565,9 @@ routes: ```yaml # yaml-language-server: $schema=https://agentgateway.dev/schema/config +gateways: + default: + port: 3000 routes: - name: azure matches: @@ -331,13 +581,41 @@ routes: resourceName: your-foundry-resource projectName: your-project-name resourceType: foundry - model: claude-sonnet-4-6 + model: claude-sonnet-4-6 policies: backendAuth: key: value: your-api-key ``` +{{< doc-test paths="azure" >}} +cat <<'EOF' > config-claude-foundry.yaml +# yaml-language-server: $schema=https://agentgateway.dev/schema/config +gateways: + default: + port: 3000 +routes: +- name: azure + matches: + - path: + pathPrefix: /azure-anthropic #prefix example + backends: + - ai: + name: azure + provider: + azure: + resourceName: your-foundry-resource + projectName: your-project-name + resourceType: foundry + model: claude-sonnet-4-6 + policies: + backendAuth: + key: + value: your-api-key +EOF +agentgateway -f config-claude-foundry.yaml --validate-only +{{< /doc-test >}} + {{< reuse "agw-docs/snippets/review-table.md" >}} | Setting | Description | @@ -359,3 +637,29 @@ curl -X POST http://localhost:4000/azure-anthropic \ "messages": [{"role": "user", "content": "Hello!"}] }' ``` + +{{< doc-test paths="azure" >}} +# Confirm the Foundry implicit-auth config serves the wildcard model and resolves +# it to the azure provider. +agentgateway -f config-foundry-implicit.yaml & +AGW_PID=$! +trap 'kill $AGW_PID 2>/dev/null' EXIT +sleep 3 + +SERVED=$(curl -sf --max-time 10 http://localhost:4000/v1/models | jq -r '[.data[].id] | index("*") // "missing"') +if [ "$SERVED" = "missing" ]; then + echo "FAIL: the wildcard model from the example config is not served" + exit 1 +fi +PROVIDER=$(curl -sf --max-time 10 http://localhost:15000/config_dump | jq -r ' + [ .backends[].backend.ai + | select(. != null) + | .target.providers[].active[].endpoint + | .provider | keys[0] + ] | first') +if [ "$PROVIDER" != "azure" ]; then + echo "FAIL: expected provider azure but agentgateway resolved $PROVIDER" + exit 1 +fi +echo "✓ The wildcard model is served and resolves to the azure provider" +{{< /doc-test >}} diff --git a/content/docs/standalone/latest/llm/providers/baseten.md b/content/docs/standalone/latest/llm/providers/baseten.md index e69927e27..2c55a06a0 100644 --- a/content/docs/standalone/latest/llm/providers/baseten.md +++ b/content/docs/standalone/latest/llm/providers/baseten.md @@ -3,10 +3,40 @@ title: Baseten weight: 20 icon: /integrations/providers/bw/baseten.svg description: Route agentgateway LLM traffic to models hosted on Baseten. +test: + baseten: + - file: ${versionRoot}/llm/providers/baseten.md + path: baseten --- Configure Baseten as an LLM provider in agentgateway. +{{< doc-test paths="baseten" >}} +# ============================================================================ +# Doc test coverage for this guide (these comments are not rendered on the page) +# ============================================================================ +# WHAT THIS TEST VALIDATES: +# * "Configuration": the example config is accepted by agentgateway +# (--validate-only), so `provider: baseten` is a recognized provider and the +# `name` / `params.apiKey` fields are correct. +# * The `params.baseUrl` row of the settings table: with the config loaded, +# agentgateway resolves this provider's upstream to the documented default +# (https://inference.baseten.co/v1), so the table cannot drift from the product without +# this test failing. +# +# WHAT THIS TEST DOES NOT VALIDATE (and why): +# * "Example request" - external dependency; sending the curl request needs a +# real Baseten API key and bills a live completion, so the test uses a +# placeholder key and asserts on resolved config instead. +# * `params.model` - display-only table row; the example config omits it, so +# there is nothing on this page to run. +{{< reuse "agw-docs/snippets/install-agentgateway-binary.md" >}} + +# The example config reads the API key from the environment. --validate-only and +# the config dump still resolve env vars, so a placeholder is enough here. +export BASETEN_API_KEY="${BASETEN_API_KEY:-test}" +{{< /doc-test >}} + ## Configuration {{< reuse "agw-docs/snippets/review-configuration.md" >}} @@ -22,6 +52,20 @@ llm: apiKey: "$BASETEN_API_KEY" ``` +{{< doc-test paths="baseten" >}} +cat <<'EOF' > config.yaml +# yaml-language-server: $schema=https://agentgateway.dev/schema/config + +llm: + models: + - name: "*" + provider: baseten + params: + apiKey: "$BASETEN_API_KEY" +EOF +agentgateway -f config.yaml --validate-only +{{< /doc-test >}} + {{< reuse "agw-docs/snippets/review-configuration.md" >}} | Setting | Description | @@ -44,3 +88,26 @@ curl -X POST http://localhost:4000/v1/chat/completions \ "messages": [{"role": "user", "content": "Hello from Baseten!"}] }' ``` + +{{< doc-test paths="baseten" >}} +# Confirm the default `params.baseUrl` documented in the settings table is what +# agentgateway actually resolves. The admin config dump reports the upstream host +# and path prefix separately, so they are recombined before comparing. +agentgateway -f config.yaml & +AGW_PID=$! +trap 'kill $AGW_PID 2>/dev/null' EXIT +sleep 3 + +EXPECTED="https://inference.baseten.co/v1" +RESOLVED=$(curl -sf --max-time 10 http://localhost:15000/config_dump | jq -r ' + [ .backends[].backend.ai + | select(. != null) + | .target.providers[].active[].endpoint + | "https://" + (.hostOverride | sub(":443$"; "")) + (.pathPrefix // "") + ] | first') +if [ "$RESOLVED" != "$EXPECTED" ]; then + echo "FAIL: settings table documents a default baseUrl of $EXPECTED but agentgateway resolved $RESOLVED" + exit 1 +fi +echo "✓ Provider default baseUrl resolves to $EXPECTED" +{{< /doc-test >}} diff --git a/content/docs/standalone/latest/llm/providers/bedrock.md b/content/docs/standalone/latest/llm/providers/bedrock.md index e5fa0a83b..8592d77bc 100644 --- a/content/docs/standalone/latest/llm/providers/bedrock.md +++ b/content/docs/standalone/latest/llm/providers/bedrock.md @@ -3,10 +3,40 @@ title: Amazon Bedrock weight: 15 icon: /integrations/providers/bw/bedrock.svg description: Route agentgateway LLM traffic to foundation models on Amazon Bedrock. +test: + bedrock: + - file: ${versionRoot}/llm/providers/bedrock.md + path: bedrock --- Configure Amazon Bedrock as an LLM provider in agentgateway. +{{< doc-test paths="bedrock" >}} +# ============================================================================ +# Doc test coverage for this guide (these comments are not rendered on the page) +# ============================================================================ +# WHAT THIS TEST VALIDATES: +# * "Configuration": the example config is accepted by agentgateway +# (--validate-only), so `provider: bedrock` is recognized and +# `params.awsRegion` is correct. +# * "Passthrough": the `passthrough: detect` config is accepted, including the +# `name: us.anthropic*` prefix match. +# * With the base config loaded, agentgateway serves the wildcard model and +# resolves it to the `bedrock` provider in the configured AWS region. +# +# WHAT THIS TEST DOES NOT VALIDATE (and why): +# * "Authentication" - external dependency; AWS credentials are resolved per +# request from the ambient environment, which the test cannot provide. +# * The Converse and Invoke boto3 examples - display-only Python snippets that +# need real AWS credentials and a Bedrock model grant. +# * "Token counting", "Extended thinking and reasoning", and "Structured +# outputs" - external dependency; each bills a live Bedrock completion. Their +# example responses and the `reasoning_effort` budget table are display-only. +# * That format translation to Bedrock's Converse API is correct - a different +# layer; verifying the translation needs a live Bedrock upstream. +{{< reuse "agw-docs/snippets/install-agentgateway-binary.md" >}} +{{< /doc-test >}} + > [!NOTE] > Agentgateway accepts requests in one of the supported [API formats](../../api-types) (such as the `/v1/chat/completions` request body shape) and returns responses in that format. > Agentgateway translates between these formats and Bedrock formats internally using Bedrock's [Converse API](https://docs.aws.amazon.com/bedrock/latest/userguide/conversation-inference-call.html). @@ -32,6 +62,20 @@ llm: awsRegion: us-west-2 ``` +{{< doc-test paths="bedrock" >}} +cat <<'EOF' > config.yaml +# yaml-language-server: $schema=https://agentgateway.dev/schema/config + +llm: + models: + - name: "*" + provider: bedrock + params: + awsRegion: us-west-2 +EOF +agentgateway -f config.yaml --validate-only +{{< /doc-test >}} + {{< reuse "agw-docs/snippets/review-configuration.md" >}} | Setting | Description | @@ -61,6 +105,20 @@ llm: passthrough: detect ``` +{{< doc-test paths="bedrock" >}} +cat <<'EOF' > config-passthrough.yaml +# yaml-language-server: $schema=https://agentgateway.dev/schema/config +llm: + models: + - name: us.anthropic* + provider: bedrock + params: + awsRegion: us-west-2 + passthrough: detect +EOF +agentgateway -f config-passthrough.yaml --validate-only +{{< /doc-test >}} + Then, you can send native Converse and Invoke requests: {{< tabs >}} @@ -222,3 +280,29 @@ curl "localhost:4000/v1/chat/completions" -H content-type:application/json -d '{ ] }' | jq ``` + +{{< doc-test paths="bedrock" >}} +# Confirm the base config serves the wildcard model and that `params.awsRegion` +# reaches the resolved provider config. +agentgateway -f config.yaml & +AGW_PID=$! +trap 'kill $AGW_PID 2>/dev/null' EXIT +sleep 3 + +SERVED=$(curl -sf --max-time 10 http://localhost:4000/v1/models | jq -r '[.data[].id] | index("*") // "missing"') +if [ "$SERVED" = "missing" ]; then + echo "FAIL: the wildcard model from the example config is not served" + exit 1 +fi +RESOLVED=$(curl -sf --max-time 10 http://localhost:15000/config_dump | jq -r ' + [ .backends[].backend.ai + | select(. != null) + | .target.providers[].active[].endpoint + | "\(.provider | keys[0])|\(.provider.bedrock.region)" + ] | first') +if [ "$RESOLVED" != "bedrock|us-west-2" ]; then + echo "FAIL: expected bedrock|us-west-2 but agentgateway resolved $RESOLVED" + exit 1 +fi +echo "✓ Wildcard model is served and resolves to bedrock in us-west-2" +{{< /doc-test >}} diff --git a/content/docs/standalone/latest/llm/providers/cerebras.md b/content/docs/standalone/latest/llm/providers/cerebras.md index 6c5894944..abac97186 100644 --- a/content/docs/standalone/latest/llm/providers/cerebras.md +++ b/content/docs/standalone/latest/llm/providers/cerebras.md @@ -3,10 +3,40 @@ title: Cerebras weight: 20 icon: /integrations/providers/bw/cerebras.svg description: Route agentgateway LLM traffic to models hosted on Cerebras. +test: + cerebras: + - file: ${versionRoot}/llm/providers/cerebras.md + path: cerebras --- Configure Cerebras as an LLM provider in agentgateway. +{{< doc-test paths="cerebras" >}} +# ============================================================================ +# Doc test coverage for this guide (these comments are not rendered on the page) +# ============================================================================ +# WHAT THIS TEST VALIDATES: +# * "Configuration": the example config is accepted by agentgateway +# (--validate-only), so `provider: cerebras` is a recognized provider and the +# `name` / `params.apiKey` fields are correct. +# * The `params.baseUrl` row of the settings table: with the config loaded, +# agentgateway resolves this provider's upstream to the documented default +# (https://api.cerebras.ai/v1), so the table cannot drift from the product without +# this test failing. +# +# WHAT THIS TEST DOES NOT VALIDATE (and why): +# * "Example request" - external dependency; sending the curl request needs a +# real Cerebras API key and bills a live completion, so the test uses a +# placeholder key and asserts on resolved config instead. +# * `params.model` - display-only table row; the example config omits it, so +# there is nothing on this page to run. +{{< reuse "agw-docs/snippets/install-agentgateway-binary.md" >}} + +# The example config reads the API key from the environment. --validate-only and +# the config dump still resolve env vars, so a placeholder is enough here. +export CEREBRAS_API_KEY="${CEREBRAS_API_KEY:-test}" +{{< /doc-test >}} + ## Configuration {{< reuse "agw-docs/snippets/review-configuration.md" >}} @@ -22,6 +52,20 @@ llm: apiKey: "$CEREBRAS_API_KEY" ``` +{{< doc-test paths="cerebras" >}} +cat <<'EOF' > config.yaml +# yaml-language-server: $schema=https://agentgateway.dev/schema/config + +llm: + models: + - name: "*" + provider: cerebras + params: + apiKey: "$CEREBRAS_API_KEY" +EOF +agentgateway -f config.yaml --validate-only +{{< /doc-test >}} + {{< reuse "agw-docs/snippets/review-configuration.md" >}} | Setting | Description | @@ -44,3 +88,26 @@ curl -X POST http://localhost:4000/v1/chat/completions \ "messages": [{"role": "user", "content": "Hello from Cerebras!"}] }' ``` + +{{< doc-test paths="cerebras" >}} +# Confirm the default `params.baseUrl` documented in the settings table is what +# agentgateway actually resolves. The admin config dump reports the upstream host +# and path prefix separately, so they are recombined before comparing. +agentgateway -f config.yaml & +AGW_PID=$! +trap 'kill $AGW_PID 2>/dev/null' EXIT +sleep 3 + +EXPECTED="https://api.cerebras.ai/v1" +RESOLVED=$(curl -sf --max-time 10 http://localhost:15000/config_dump | jq -r ' + [ .backends[].backend.ai + | select(. != null) + | .target.providers[].active[].endpoint + | "https://" + (.hostOverride | sub(":443$"; "")) + (.pathPrefix // "") + ] | first') +if [ "$RESOLVED" != "$EXPECTED" ]; then + echo "FAIL: settings table documents a default baseUrl of $EXPECTED but agentgateway resolved $RESOLVED" + exit 1 +fi +echo "✓ Provider default baseUrl resolves to $EXPECTED" +{{< /doc-test >}} diff --git a/content/docs/standalone/latest/llm/providers/cohere.md b/content/docs/standalone/latest/llm/providers/cohere.md index 26cdf8d55..91686acf0 100644 --- a/content/docs/standalone/latest/llm/providers/cohere.md +++ b/content/docs/standalone/latest/llm/providers/cohere.md @@ -3,10 +3,40 @@ title: Cohere weight: 20 icon: /integrations/providers/bw/cohere.svg description: Route agentgateway LLM traffic to Cohere's models. +test: + cohere: + - file: ${versionRoot}/llm/providers/cohere.md + path: cohere --- Configure Cohere as an LLM provider in agentgateway. +{{< doc-test paths="cohere" >}} +# ============================================================================ +# Doc test coverage for this guide (these comments are not rendered on the page) +# ============================================================================ +# WHAT THIS TEST VALIDATES: +# * "Configuration": the example config is accepted by agentgateway +# (--validate-only), so `provider: cohere` is a recognized provider and the +# `name` / `params.apiKey` fields are correct. +# * The `params.baseUrl` row of the settings table: with the config loaded, +# agentgateway resolves this provider's upstream to the documented default +# (https://api.cohere.ai), so the table cannot drift from the product without +# this test failing. +# +# WHAT THIS TEST DOES NOT VALIDATE (and why): +# * "Example request" - external dependency; sending the curl request needs a +# real Cohere API key and bills a live completion, so the test uses a +# placeholder key and asserts on resolved config instead. +# * `params.model` - display-only table row; the example config omits it, so +# there is nothing on this page to run. +{{< reuse "agw-docs/snippets/install-agentgateway-binary.md" >}} + +# The example config reads the API key from the environment. --validate-only and +# the config dump still resolve env vars, so a placeholder is enough here. +export COHERE_API_KEY="${COHERE_API_KEY:-test}" +{{< /doc-test >}} + ## Configuration {{< reuse "agw-docs/snippets/review-configuration.md" >}} @@ -22,6 +52,20 @@ llm: apiKey: "$COHERE_API_KEY" ``` +{{< doc-test paths="cohere" >}} +cat <<'EOF' > config.yaml +# yaml-language-server: $schema=https://agentgateway.dev/schema/config + +llm: + models: + - name: "*" + provider: cohere + params: + apiKey: "$COHERE_API_KEY" +EOF +agentgateway -f config.yaml --validate-only +{{< /doc-test >}} + {{< reuse "agw-docs/snippets/review-configuration.md" >}} | Setting | Description | @@ -44,3 +88,26 @@ curl -X POST http://localhost:4000/v1/chat/completions \ "messages": [{"role": "user", "content": "Hello from Cohere!"}] }' ``` + +{{< doc-test paths="cohere" >}} +# Confirm the default `params.baseUrl` documented in the settings table is what +# agentgateway actually resolves. The admin config dump reports the upstream host +# and path prefix separately, so they are recombined before comparing. +agentgateway -f config.yaml & +AGW_PID=$! +trap 'kill $AGW_PID 2>/dev/null' EXIT +sleep 3 + +EXPECTED="https://api.cohere.ai" +RESOLVED=$(curl -sf --max-time 10 http://localhost:15000/config_dump | jq -r ' + [ .backends[].backend.ai + | select(. != null) + | .target.providers[].active[].endpoint + | "https://" + (.hostOverride | sub(":443$"; "")) + (.pathPrefix // "") + ] | first') +if [ "$RESOLVED" != "$EXPECTED" ]; then + echo "FAIL: settings table documents a default baseUrl of $EXPECTED but agentgateway resolved $RESOLVED" + exit 1 +fi +echo "✓ Provider default baseUrl resolves to $EXPECTED" +{{< /doc-test >}} diff --git a/content/docs/standalone/latest/llm/providers/deepinfra.md b/content/docs/standalone/latest/llm/providers/deepinfra.md index 593648b9f..af4ffa2a0 100644 --- a/content/docs/standalone/latest/llm/providers/deepinfra.md +++ b/content/docs/standalone/latest/llm/providers/deepinfra.md @@ -3,10 +3,40 @@ title: DeepInfra weight: 20 icon: /integrations/providers/bw/deepinfra.svg description: Route agentgateway LLM traffic to models hosted on DeepInfra. +test: + deepinfra: + - file: ${versionRoot}/llm/providers/deepinfra.md + path: deepinfra --- Configure DeepInfra as an LLM provider in agentgateway. +{{< doc-test paths="deepinfra" >}} +# ============================================================================ +# Doc test coverage for this guide (these comments are not rendered on the page) +# ============================================================================ +# WHAT THIS TEST VALIDATES: +# * "Configuration": the example config is accepted by agentgateway +# (--validate-only), so `provider: deepinfra` is a recognized provider and the +# `name` / `params.apiKey` fields are correct. +# * The `params.baseUrl` row of the settings table: with the config loaded, +# agentgateway resolves this provider's upstream to the documented default +# (https://api.deepinfra.com/v1/openai), so the table cannot drift from the product without +# this test failing. +# +# WHAT THIS TEST DOES NOT VALIDATE (and why): +# * "Example request" - external dependency; sending the curl request needs a +# real DeepInfra API key and bills a live completion, so the test uses a +# placeholder key and asserts on resolved config instead. +# * `params.model` - display-only table row; the example config omits it, so +# there is nothing on this page to run. +{{< reuse "agw-docs/snippets/install-agentgateway-binary.md" >}} + +# The example config reads the API key from the environment. --validate-only and +# the config dump still resolve env vars, so a placeholder is enough here. +export DEEPINFRA_API_KEY="${DEEPINFRA_API_KEY:-test}" +{{< /doc-test >}} + ## Configuration {{< reuse "agw-docs/snippets/review-configuration.md" >}} @@ -22,6 +52,20 @@ llm: apiKey: "$DEEPINFRA_API_KEY" ``` +{{< doc-test paths="deepinfra" >}} +cat <<'EOF' > config.yaml +# yaml-language-server: $schema=https://agentgateway.dev/schema/config + +llm: + models: + - name: "*" + provider: deepinfra + params: + apiKey: "$DEEPINFRA_API_KEY" +EOF +agentgateway -f config.yaml --validate-only +{{< /doc-test >}} + {{< reuse "agw-docs/snippets/review-configuration.md" >}} | Setting | Description | @@ -44,3 +88,26 @@ curl -X POST http://localhost:4000/v1/chat/completions \ "messages": [{"role": "user", "content": "Hello from DeepInfra!"}] }' ``` + +{{< doc-test paths="deepinfra" >}} +# Confirm the default `params.baseUrl` documented in the settings table is what +# agentgateway actually resolves. The admin config dump reports the upstream host +# and path prefix separately, so they are recombined before comparing. +agentgateway -f config.yaml & +AGW_PID=$! +trap 'kill $AGW_PID 2>/dev/null' EXIT +sleep 3 + +EXPECTED="https://api.deepinfra.com/v1/openai" +RESOLVED=$(curl -sf --max-time 10 http://localhost:15000/config_dump | jq -r ' + [ .backends[].backend.ai + | select(. != null) + | .target.providers[].active[].endpoint + | "https://" + (.hostOverride | sub(":443$"; "")) + (.pathPrefix // "") + ] | first') +if [ "$RESOLVED" != "$EXPECTED" ]; then + echo "FAIL: settings table documents a default baseUrl of $EXPECTED but agentgateway resolved $RESOLVED" + exit 1 +fi +echo "✓ Provider default baseUrl resolves to $EXPECTED" +{{< /doc-test >}} diff --git a/content/docs/standalone/latest/llm/providers/deepseek.md b/content/docs/standalone/latest/llm/providers/deepseek.md index bddf03113..ebd787f13 100644 --- a/content/docs/standalone/latest/llm/providers/deepseek.md +++ b/content/docs/standalone/latest/llm/providers/deepseek.md @@ -3,10 +3,40 @@ title: DeepSeek weight: 20 icon: /integrations/providers/bw/deepseek.svg description: Route agentgateway LLM traffic to DeepSeek's models. +test: + deepseek: + - file: ${versionRoot}/llm/providers/deepseek.md + path: deepseek --- Configure DeepSeek as an LLM provider in agentgateway. +{{< doc-test paths="deepseek" >}} +# ============================================================================ +# Doc test coverage for this guide (these comments are not rendered on the page) +# ============================================================================ +# WHAT THIS TEST VALIDATES: +# * "Configuration": the example config is accepted by agentgateway +# (--validate-only), so `provider: deepseek` is a recognized provider and the +# `name` / `params.apiKey` fields are correct. +# * The `params.baseUrl` row of the settings table: with the config loaded, +# agentgateway resolves this provider's upstream to the documented default +# (https://api.deepseek.com/v1), so the table cannot drift from the product without +# this test failing. +# +# WHAT THIS TEST DOES NOT VALIDATE (and why): +# * "Example request" - external dependency; sending the curl request needs a +# real DeepSeek API key and bills a live completion, so the test uses a +# placeholder key and asserts on resolved config instead. +# * `params.model` - display-only table row; the example config omits it, so +# there is nothing on this page to run. +{{< reuse "agw-docs/snippets/install-agentgateway-binary.md" >}} + +# The example config reads the API key from the environment. --validate-only and +# the config dump still resolve env vars, so a placeholder is enough here. +export DEEPSEEK_API_KEY="${DEEPSEEK_API_KEY:-test}" +{{< /doc-test >}} + ## Configuration {{< reuse "agw-docs/snippets/review-configuration.md" >}} @@ -22,6 +52,20 @@ llm: apiKey: "$DEEPSEEK_API_KEY" ``` +{{< doc-test paths="deepseek" >}} +cat <<'EOF' > config.yaml +# yaml-language-server: $schema=https://agentgateway.dev/schema/config + +llm: + models: + - name: "*" + provider: deepseek + params: + apiKey: "$DEEPSEEK_API_KEY" +EOF +agentgateway -f config.yaml --validate-only +{{< /doc-test >}} + {{< reuse "agw-docs/snippets/review-configuration.md" >}} | Setting | Description | @@ -44,3 +88,26 @@ curl -X POST http://localhost:4000/v1/chat/completions \ "messages": [{"role": "user", "content": "Hello from DeepSeek!"}] }' ``` + +{{< doc-test paths="deepseek" >}} +# Confirm the default `params.baseUrl` documented in the settings table is what +# agentgateway actually resolves. The admin config dump reports the upstream host +# and path prefix separately, so they are recombined before comparing. +agentgateway -f config.yaml & +AGW_PID=$! +trap 'kill $AGW_PID 2>/dev/null' EXIT +sleep 3 + +EXPECTED="https://api.deepseek.com/v1" +RESOLVED=$(curl -sf --max-time 10 http://localhost:15000/config_dump | jq -r ' + [ .backends[].backend.ai + | select(. != null) + | .target.providers[].active[].endpoint + | "https://" + (.hostOverride | sub(":443$"; "")) + (.pathPrefix // "") + ] | first') +if [ "$RESOLVED" != "$EXPECTED" ]; then + echo "FAIL: settings table documents a default baseUrl of $EXPECTED but agentgateway resolved $RESOLVED" + exit 1 +fi +echo "✓ Provider default baseUrl resolves to $EXPECTED" +{{< /doc-test >}} diff --git a/content/docs/standalone/latest/llm/providers/fireworks.md b/content/docs/standalone/latest/llm/providers/fireworks.md index 997a13ac4..c3b11dd19 100644 --- a/content/docs/standalone/latest/llm/providers/fireworks.md +++ b/content/docs/standalone/latest/llm/providers/fireworks.md @@ -3,10 +3,40 @@ title: Fireworks AI weight: 20 icon: /integrations/providers/bw/fireworks.svg description: Route agentgateway LLM traffic to models hosted on Fireworks AI. +test: + fireworks: + - file: ${versionRoot}/llm/providers/fireworks.md + path: fireworks --- Configure Fireworks AI as an LLM provider in agentgateway. +{{< doc-test paths="fireworks" >}} +# ============================================================================ +# Doc test coverage for this guide (these comments are not rendered on the page) +# ============================================================================ +# WHAT THIS TEST VALIDATES: +# * "Configuration": the example config is accepted by agentgateway +# (--validate-only), so `provider: fireworks` is a recognized provider and the +# `name` / `params.apiKey` fields are correct. +# * The `params.baseUrl` row of the settings table: with the config loaded, +# agentgateway resolves this provider's upstream to the documented default +# (https://api.fireworks.ai/inference/v1), so the table cannot drift from the product without +# this test failing. +# +# WHAT THIS TEST DOES NOT VALIDATE (and why): +# * "Example request" - external dependency; sending the curl request needs a +# real Fireworks AI API key and bills a live completion, so the test uses a +# placeholder key and asserts on resolved config instead. +# * `params.model` - display-only table row; the example config omits it, so +# there is nothing on this page to run. +{{< reuse "agw-docs/snippets/install-agentgateway-binary.md" >}} + +# The example config reads the API key from the environment. --validate-only and +# the config dump still resolve env vars, so a placeholder is enough here. +export FIREWORKS_API_KEY="${FIREWORKS_API_KEY:-test}" +{{< /doc-test >}} + ## Configuration {{< reuse "agw-docs/snippets/review-configuration.md" >}} @@ -22,6 +52,20 @@ llm: apiKey: "$FIREWORKS_API_KEY" ``` +{{< doc-test paths="fireworks" >}} +cat <<'EOF' > config.yaml +# yaml-language-server: $schema=https://agentgateway.dev/schema/config + +llm: + models: + - name: "*" + provider: fireworks + params: + apiKey: "$FIREWORKS_API_KEY" +EOF +agentgateway -f config.yaml --validate-only +{{< /doc-test >}} + {{< reuse "agw-docs/snippets/review-configuration.md" >}} | Setting | Description | @@ -45,3 +89,25 @@ curl -X POST http://localhost:4000/v1/chat/completions \ }' ``` +{{< doc-test paths="fireworks" >}} +# Confirm the default `params.baseUrl` documented in the settings table is what +# agentgateway actually resolves. The admin config dump reports the upstream host +# and path prefix separately, so they are recombined before comparing. +agentgateway -f config.yaml & +AGW_PID=$! +trap 'kill $AGW_PID 2>/dev/null' EXIT +sleep 3 + +EXPECTED="https://api.fireworks.ai/inference/v1" +RESOLVED=$(curl -sf --max-time 10 http://localhost:15000/config_dump | jq -r ' + [ .backends[].backend.ai + | select(. != null) + | .target.providers[].active[].endpoint + | "https://" + (.hostOverride | sub(":443$"; "")) + (.pathPrefix // "") + ] | first') +if [ "$RESOLVED" != "$EXPECTED" ]; then + echo "FAIL: settings table documents a default baseUrl of $EXPECTED but agentgateway resolved $RESOLVED" + exit 1 +fi +echo "✓ Provider default baseUrl resolves to $EXPECTED" +{{< /doc-test >}} diff --git a/content/docs/standalone/latest/llm/providers/gemini.md b/content/docs/standalone/latest/llm/providers/gemini.md index 43b19bf88..b49dd8a59 100644 --- a/content/docs/standalone/latest/llm/providers/gemini.md +++ b/content/docs/standalone/latest/llm/providers/gemini.md @@ -3,10 +3,39 @@ title: Gemini weight: 15 icon: /integrations/providers/bw/gemini.svg description: Route agentgateway LLM traffic to Google Gemini models. +test: + gemini: + - file: ${versionRoot}/llm/providers/gemini.md + path: gemini --- Configure Google Gemini as an LLM provider in agentgateway. +{{< doc-test paths="gemini" >}} +# ============================================================================ +# Doc test coverage for this guide (these comments are not rendered on the page) +# ============================================================================ +# WHAT THIS TEST VALIDATES: +# * "Configuration": the example config is accepted by agentgateway +# (--validate-only), so `provider: gemini` is a recognized provider and the +# `name` / `params.apiKey` fields are correct. +# * With the config loaded, agentgateway serves the wildcard model from the +# example and resolves it to the `gemini` provider, which is what the `name` +# and `provider` rows of the settings table describe. +# +# WHAT THIS TEST DOES NOT VALIDATE (and why): +# * "Example request" and any example responses - external dependency; the +# request needs a real Gemini API key and bills a live completion, so the test +# uses a placeholder key. +# * `params.model` - display-only table row; the example config omits it, so +# there is nothing on this page to run. +{{< reuse "agw-docs/snippets/install-agentgateway-binary.md" >}} + +# The example config reads the API key from the environment. --validate-only +# and the config dump still resolve env vars, so a placeholder is enough here. +export GEMINI_API_KEY="${GEMINI_API_KEY:-test}" +{{< /doc-test >}} + ## Configuration {{< reuse "agw-docs/snippets/review-configuration.md" >}} @@ -22,6 +51,20 @@ llm: apiKey: "$GEMINI_API_KEY" ``` +{{< doc-test paths="gemini" >}} +cat <<'EOF' > config.yaml +# yaml-language-server: $schema=https://agentgateway.dev/schema/config + +llm: + models: + - name: "*" + provider: gemini + params: + apiKey: "$GEMINI_API_KEY" +EOF +agentgateway -f config.yaml --validate-only +{{< /doc-test >}} + {{< reuse "agw-docs/snippets/review-configuration.md" >}} | Setting | Description | @@ -30,3 +73,30 @@ llm: | `provider` | The LLM provider, set to `gemini` for Google Gemini models. | | `params.model` | The specific Gemini model to use. If set, this model is used for all requests. If not set, the request must include the model to use. | | `params.apiKey` | The Gemini API key for authentication. You can reference environment variables using the `$VAR_NAME` syntax. | + +{{< doc-test paths="gemini" >}} +# Confirm the config serves the model named in the example and resolves it to +# this provider. First-class providers use built-in upstream defaults, so the +# config dump reports the provider discriminant rather than a host override. +agentgateway -f config.yaml & +AGW_PID=$! +trap 'kill $AGW_PID 2>/dev/null' EXIT +sleep 3 + +SERVED=$(curl -sf --max-time 10 http://localhost:4000/v1/models | jq -r '[.data[].id] | index("*") // "missing"') +if [ "$SERVED" = "missing" ]; then + echo "FAIL: the wildcard model from the example config is not served" + exit 1 +fi +PROVIDER=$(curl -sf --max-time 10 http://localhost:15000/config_dump | jq -r ' + [ .backends[].backend.ai + | select(. != null) + | .target.providers[].active[].endpoint + | .provider | keys[0] + ] | first') +if [ "$PROVIDER" != "gemini" ]; then + echo "FAIL: expected provider gemini but agentgateway resolved $PROVIDER" + exit 1 +fi +echo "✓ The wildcard model is served and resolves to the gemini provider" +{{< /doc-test >}} diff --git a/content/docs/standalone/latest/llm/providers/groq.md b/content/docs/standalone/latest/llm/providers/groq.md index 68031d246..b2a10fb0c 100644 --- a/content/docs/standalone/latest/llm/providers/groq.md +++ b/content/docs/standalone/latest/llm/providers/groq.md @@ -3,10 +3,40 @@ title: Groq weight: 20 icon: /integrations/providers/bw/groq.svg description: Route agentgateway LLM traffic to models served by Groq. +test: + groq: + - file: ${versionRoot}/llm/providers/groq.md + path: groq --- Configure Groq as an LLM provider in agentgateway. +{{< doc-test paths="groq" >}} +# ============================================================================ +# Doc test coverage for this guide (these comments are not rendered on the page) +# ============================================================================ +# WHAT THIS TEST VALIDATES: +# * "Configuration": the example config is accepted by agentgateway +# (--validate-only), so `provider: groq` is a recognized provider and the +# `name` / `params.apiKey` fields are correct. +# * The `params.baseUrl` row of the settings table: with the config loaded, +# agentgateway resolves this provider's upstream to the documented default +# (https://api.groq.com/openai/v1), so the table cannot drift from the product without +# this test failing. +# +# WHAT THIS TEST DOES NOT VALIDATE (and why): +# * "Example request" - external dependency; sending the curl request needs a +# real Groq API key and bills a live completion, so the test uses a +# placeholder key and asserts on resolved config instead. +# * `params.model` - display-only table row; the example config omits it, so +# there is nothing on this page to run. +{{< reuse "agw-docs/snippets/install-agentgateway-binary.md" >}} + +# The example config reads the API key from the environment. --validate-only and +# the config dump still resolve env vars, so a placeholder is enough here. +export GROQ_API_KEY="${GROQ_API_KEY:-test}" +{{< /doc-test >}} + ## Configuration {{< reuse "agw-docs/snippets/review-configuration.md" >}} @@ -22,6 +52,20 @@ llm: apiKey: "$GROQ_API_KEY" ``` +{{< doc-test paths="groq" >}} +cat <<'EOF' > config.yaml +# yaml-language-server: $schema=https://agentgateway.dev/schema/config + +llm: + models: + - name: "*" + provider: groq + params: + apiKey: "$GROQ_API_KEY" +EOF +agentgateway -f config.yaml --validate-only +{{< /doc-test >}} + {{< reuse "agw-docs/snippets/review-configuration.md" >}} | Setting | Description | @@ -44,3 +88,26 @@ curl -X POST http://localhost:4000/v1/chat/completions \ "messages": [{"role": "user", "content": "Hello from Groq!"}] }' ``` + +{{< doc-test paths="groq" >}} +# Confirm the default `params.baseUrl` documented in the settings table is what +# agentgateway actually resolves. The admin config dump reports the upstream host +# and path prefix separately, so they are recombined before comparing. +agentgateway -f config.yaml & +AGW_PID=$! +trap 'kill $AGW_PID 2>/dev/null' EXIT +sleep 3 + +EXPECTED="https://api.groq.com/openai/v1" +RESOLVED=$(curl -sf --max-time 10 http://localhost:15000/config_dump | jq -r ' + [ .backends[].backend.ai + | select(. != null) + | .target.providers[].active[].endpoint + | "https://" + (.hostOverride | sub(":443$"; "")) + (.pathPrefix // "") + ] | first') +if [ "$RESOLVED" != "$EXPECTED" ]; then + echo "FAIL: settings table documents a default baseUrl of $EXPECTED but agentgateway resolved $RESOLVED" + exit 1 +fi +echo "✓ Provider default baseUrl resolves to $EXPECTED" +{{< /doc-test >}} diff --git a/content/docs/standalone/latest/llm/providers/huggingface.md b/content/docs/standalone/latest/llm/providers/huggingface.md index 59053994f..e9406f38e 100644 --- a/content/docs/standalone/latest/llm/providers/huggingface.md +++ b/content/docs/standalone/latest/llm/providers/huggingface.md @@ -3,10 +3,40 @@ title: Hugging Face weight: 20 icon: /integrations/providers/bw/huggingface.svg description: Route agentgateway LLM traffic to models hosted on Hugging Face. +test: + huggingface: + - file: ${versionRoot}/llm/providers/huggingface.md + path: huggingface --- Configure Hugging Face as an LLM provider in agentgateway. +{{< doc-test paths="huggingface" >}} +# ============================================================================ +# Doc test coverage for this guide (these comments are not rendered on the page) +# ============================================================================ +# WHAT THIS TEST VALIDATES: +# * "Configuration": the example config is accepted by agentgateway +# (--validate-only), so `provider: huggingface` is a recognized provider and the +# `name` / `params.apiKey` fields are correct. +# * The `params.baseUrl` row of the settings table: with the config loaded, +# agentgateway resolves this provider's upstream to the documented default +# (https://router.huggingface.co/v1), so the table cannot drift from the product without +# this test failing. +# +# WHAT THIS TEST DOES NOT VALIDATE (and why): +# * "Example request" - external dependency; sending the curl request needs a +# real Hugging Face API key and bills a live completion, so the test uses a +# placeholder key and asserts on resolved config instead. +# * `params.model` - display-only table row; the example config omits it, so +# there is nothing on this page to run. +{{< reuse "agw-docs/snippets/install-agentgateway-binary.md" >}} + +# The example config reads the API key from the environment. --validate-only and +# the config dump still resolve env vars, so a placeholder is enough here. +export HUGGINGFACE_API_KEY="${HUGGINGFACE_API_KEY:-test}" +{{< /doc-test >}} + ## Configuration {{< reuse "agw-docs/snippets/review-configuration.md" >}} @@ -22,6 +52,20 @@ llm: apiKey: "$HUGGINGFACE_API_KEY" ``` +{{< doc-test paths="huggingface" >}} +cat <<'EOF' > config.yaml +# yaml-language-server: $schema=https://agentgateway.dev/schema/config + +llm: + models: + - name: "*" + provider: huggingface + params: + apiKey: "$HUGGINGFACE_API_KEY" +EOF +agentgateway -f config.yaml --validate-only +{{< /doc-test >}} + {{< reuse "agw-docs/snippets/review-configuration.md" >}} | Setting | Description | @@ -44,3 +88,26 @@ curl -X POST http://localhost:4000/v1/chat/completions \ "messages": [{"role": "user", "content": "Hello from Hugging Face!"}] }' ``` + +{{< doc-test paths="huggingface" >}} +# Confirm the default `params.baseUrl` documented in the settings table is what +# agentgateway actually resolves. The admin config dump reports the upstream host +# and path prefix separately, so they are recombined before comparing. +agentgateway -f config.yaml & +AGW_PID=$! +trap 'kill $AGW_PID 2>/dev/null' EXIT +sleep 3 + +EXPECTED="https://router.huggingface.co/v1" +RESOLVED=$(curl -sf --max-time 10 http://localhost:15000/config_dump | jq -r ' + [ .backends[].backend.ai + | select(. != null) + | .target.providers[].active[].endpoint + | "https://" + (.hostOverride | sub(":443$"; "")) + (.pathPrefix // "") + ] | first') +if [ "$RESOLVED" != "$EXPECTED" ]; then + echo "FAIL: settings table documents a default baseUrl of $EXPECTED but agentgateway resolved $RESOLVED" + exit 1 +fi +echo "✓ Provider default baseUrl resolves to $EXPECTED" +{{< /doc-test >}} diff --git a/content/docs/standalone/latest/llm/providers/mistral.md b/content/docs/standalone/latest/llm/providers/mistral.md index b22043beb..580348ad8 100644 --- a/content/docs/standalone/latest/llm/providers/mistral.md +++ b/content/docs/standalone/latest/llm/providers/mistral.md @@ -3,10 +3,40 @@ title: Mistral weight: 20 icon: /integrations/providers/bw/mistral.svg description: Route agentgateway LLM traffic to Mistral's models. +test: + mistral: + - file: ${versionRoot}/llm/providers/mistral.md + path: mistral --- Configure Mistral as an LLM provider in agentgateway. +{{< doc-test paths="mistral" >}} +# ============================================================================ +# Doc test coverage for this guide (these comments are not rendered on the page) +# ============================================================================ +# WHAT THIS TEST VALIDATES: +# * "Configuration": the example config is accepted by agentgateway +# (--validate-only), so `provider: mistral` is a recognized provider and the +# `name` / `params.apiKey` fields are correct. +# * The `params.baseUrl` row of the settings table: with the config loaded, +# agentgateway resolves this provider's upstream to the documented default +# (https://api.mistral.ai/v1), so the table cannot drift from the product without +# this test failing. +# +# WHAT THIS TEST DOES NOT VALIDATE (and why): +# * "Example request" - external dependency; sending the curl request needs a +# real Mistral API key and bills a live completion, so the test uses a +# placeholder key and asserts on resolved config instead. +# * `params.model` - display-only table row; the example config omits it, so +# there is nothing on this page to run. +{{< reuse "agw-docs/snippets/install-agentgateway-binary.md" >}} + +# The example config reads the API key from the environment. --validate-only and +# the config dump still resolve env vars, so a placeholder is enough here. +export MISTRAL_API_KEY="${MISTRAL_API_KEY:-test}" +{{< /doc-test >}} + ## Configuration {{< reuse "agw-docs/snippets/review-configuration.md" >}} @@ -22,6 +52,20 @@ llm: apiKey: "$MISTRAL_API_KEY" ``` +{{< doc-test paths="mistral" >}} +cat <<'EOF' > config.yaml +# yaml-language-server: $schema=https://agentgateway.dev/schema/config + +llm: + models: + - name: "*" + provider: mistral + params: + apiKey: "$MISTRAL_API_KEY" +EOF +agentgateway -f config.yaml --validate-only +{{< /doc-test >}} + {{< reuse "agw-docs/snippets/review-configuration.md" >}} | Setting | Description | @@ -45,3 +89,25 @@ curl -X POST http://localhost:4000/v1/chat/completions \ }' ``` +{{< doc-test paths="mistral" >}} +# Confirm the default `params.baseUrl` documented in the settings table is what +# agentgateway actually resolves. The admin config dump reports the upstream host +# and path prefix separately, so they are recombined before comparing. +agentgateway -f config.yaml & +AGW_PID=$! +trap 'kill $AGW_PID 2>/dev/null' EXIT +sleep 3 + +EXPECTED="https://api.mistral.ai/v1" +RESOLVED=$(curl -sf --max-time 10 http://localhost:15000/config_dump | jq -r ' + [ .backends[].backend.ai + | select(. != null) + | .target.providers[].active[].endpoint + | "https://" + (.hostOverride | sub(":443$"; "")) + (.pathPrefix // "") + ] | first') +if [ "$RESOLVED" != "$EXPECTED" ]; then + echo "FAIL: settings table documents a default baseUrl of $EXPECTED but agentgateway resolved $RESOLVED" + exit 1 +fi +echo "✓ Provider default baseUrl resolves to $EXPECTED" +{{< /doc-test >}} diff --git a/content/docs/standalone/latest/llm/providers/multiple-llms.md b/content/docs/standalone/latest/llm/providers/multiple-llms.md index d2124c4b8..c86268c41 100644 --- a/content/docs/standalone/latest/llm/providers/multiple-llms.md +++ b/content/docs/standalone/latest/llm/providers/multiple-llms.md @@ -2,8 +2,39 @@ title: Multiple LLM providers weight: 30 description: Define reusable LLM provider configurations once and reference them across multiple model definitions to avoid duplicating connection and authentication parameters. +test: + multiple-llms: + - file: ${versionRoot}/llm/providers/multiple-llms.md + path: multiple-llms --- +{{< doc-test paths="multiple-llms" >}} +# ============================================================================ +# Doc test coverage for this guide (these comments are not rendered on the page) +# ============================================================================ +# WHAT THIS TEST VALIDATES: +# * "Reusable provider configuration": the example config is accepted by +# agentgateway (--validate-only), covering `llm.providers[].name`, +# `llm.providers[].provider`, and `llm.models[].provider.reference`. +# * A reference actually resolves at runtime: with the config loaded, both the +# `fast` and `smart` models are served, which is only possible if each model +# inherited its upstream provider (and API key) from the `openai-prod` entry +# in `llm.providers[]`. A dangling reference fails to load. +# +# WHAT THIS TEST DOES NOT VALIDATE (and why): +# * That a completion request routed through `fast` or `smart` reaches OpenAI - +# external dependency; the test uses a placeholder API key and does not call +# the provider. +# * The other shared upstream settings the page mentions (host overrides, path +# overrides, other model defaults) - display-only prose with no example +# config on this page that sets them. +{{< reuse "agw-docs/snippets/install-agentgateway-binary.md" >}} + +# The example config reads the API key from the environment. --validate-only and +# model listing still resolve env vars, so a placeholder is enough here. +export OPENAI_API_KEY="${OPENAI_API_KEY:-test}" +{{< /doc-test >}} + ## Reusable provider configuration Reuse provider configuration to avoid duplicating connection and authentication parameters across multiple model definitions. Define named provider defaults once in `llm.providers[]` and reference them from multiple `llm.models[]` entries with `provider.reference`. @@ -12,7 +43,7 @@ Reuse provider configuration to avoid duplicating connection and authentication llm: providers: - name: openai-prod - provider: openai + provider: openAI params: apiKey: "$OPENAI_API_KEY" @@ -29,6 +60,64 @@ llm: model: gpt-4o ``` +{{< doc-test paths="multiple-llms" >}} +cat <<'EOF' > config.yaml +llm: + providers: + - name: openai-prod + provider: openAI + params: + apiKey: "$OPENAI_API_KEY" + + models: + - name: fast + provider: + reference: openai-prod + params: + model: gpt-4o-mini + - name: smart + provider: + reference: openai-prod + params: + model: gpt-4o +EOF +agentgateway -f config.yaml --validate-only +{{< /doc-test >}} + +{{< doc-test paths="multiple-llms" >}} +# Simplified LLM mode with no explicit port serves on 4000. +agentgateway -f config.yaml & +AGW_PID=$! +trap 'kill $AGW_PID 2>/dev/null' EXIT +sleep 3 +{{< /doc-test >}} + +{{< doc-test paths="multiple-llms" >}} +YAMLTest -f - <<'EOF' +- name: Both models that reference the shared provider are served + retries: 3 + http: + url: "http://localhost:4000" + path: /v1/models + method: GET + headers: + accept-encoding: identity + source: + type: local + expect: + statusCode: 200 + bodyJsonPath: + # Filter expressions rather than $.data[*].id, because a wildcard path + # resolves to a single match and the model order is not guaranteed. + - path: "$.data[?(@.id=='fast')].id" + comparator: equals + value: fast + - path: "$.data[?(@.id=='smart')].id" + comparator: equals + value: smart +EOF +{{< /doc-test >}} + In this example, `smart` inherits the upstream API key from `llm.providers[]` and only changes the model name. Named providers can hold shared upstream settings you want to reuse, such as authentication, host overrides, path overrides, or other model defaults. diff --git a/content/docs/standalone/latest/llm/providers/openai.md b/content/docs/standalone/latest/llm/providers/openai.md index 79b7c2cae..fda9f4104 100644 --- a/content/docs/standalone/latest/llm/providers/openai.md +++ b/content/docs/standalone/latest/llm/providers/openai.md @@ -3,10 +3,39 @@ title: OpenAI weight: 10 icon: /integrations/providers/bw/openai.svg description: Route agentgateway LLM traffic to OpenAI's GPT models. +test: + openai: + - file: ${versionRoot}/llm/providers/openai.md + path: openai --- Configure OpenAI as an LLM provider in agentgateway. +{{< doc-test paths="openai" >}} +# ============================================================================ +# Doc test coverage for this guide (these comments are not rendered on the page) +# ============================================================================ +# WHAT THIS TEST VALIDATES: +# * "Configuration": the example config is accepted by agentgateway +# (--validate-only), so `provider: openAI` is a recognized provider and the +# `name` / `params.apiKey` fields are correct. +# * With the config loaded, agentgateway serves the wildcard model from the +# example and resolves it to the `openAI` provider, which is what the `name` +# and `provider` rows of the settings table describe. +# +# WHAT THIS TEST DOES NOT VALIDATE (and why): +# * "Example request" and any example responses - external dependency; the +# request needs a real OpenAI API key and bills a live completion, so the test +# uses a placeholder key. +# * `params.model` - display-only table row; the example config omits it, so +# there is nothing on this page to run. +{{< reuse "agw-docs/snippets/install-agentgateway-binary.md" >}} + +# The example config reads the API key from the environment. --validate-only +# and the config dump still resolve env vars, so a placeholder is enough here. +export OPENAI_API_KEY="${OPENAI_API_KEY:-test}" +{{< /doc-test >}} + ## Configuration {{< reuse "agw-docs/snippets/review-configuration.md" >}} @@ -22,6 +51,20 @@ llm: apiKey: "$OPENAI_API_KEY" ``` +{{< doc-test paths="openai" >}} +cat <<'EOF' > config.yaml +# yaml-language-server: $schema=https://agentgateway.dev/schema/config + +llm: + models: + - name: "*" + provider: openAI + params: + apiKey: "$OPENAI_API_KEY" +EOF +agentgateway -f config.yaml --validate-only +{{< /doc-test >}} + {{< reuse "agw-docs/snippets/review-configuration.md" >}} | Setting | Description | @@ -36,3 +79,30 @@ llm: > [!NOTE] > To connect Codex to agentgateway, see the [Codex integration page]({{< link-hextra path="/integrations/llm-clients/codex" >}}). + +{{< doc-test paths="openai" >}} +# Confirm the config serves the model named in the example and resolves it to +# this provider. First-class providers use built-in upstream defaults, so the +# config dump reports the provider discriminant rather than a host override. +agentgateway -f config.yaml & +AGW_PID=$! +trap 'kill $AGW_PID 2>/dev/null' EXIT +sleep 3 + +SERVED=$(curl -sf --max-time 10 http://localhost:4000/v1/models | jq -r '[.data[].id] | index("*") // "missing"') +if [ "$SERVED" = "missing" ]; then + echo "FAIL: the wildcard model from the example config is not served" + exit 1 +fi +PROVIDER=$(curl -sf --max-time 10 http://localhost:15000/config_dump | jq -r ' + [ .backends[].backend.ai + | select(. != null) + | .target.providers[].active[].endpoint + | .provider | keys[0] + ] | first') +if [ "$PROVIDER" != "openAI" ]; then + echo "FAIL: expected provider openAI but agentgateway resolved $PROVIDER" + exit 1 +fi +echo "✓ The wildcard model is served and resolves to the openAI provider" +{{< /doc-test >}} diff --git a/content/docs/standalone/latest/llm/providers/openrouter.md b/content/docs/standalone/latest/llm/providers/openrouter.md index 978e74555..afe4c611e 100644 --- a/content/docs/standalone/latest/llm/providers/openrouter.md +++ b/content/docs/standalone/latest/llm/providers/openrouter.md @@ -3,10 +3,40 @@ title: OpenRouter weight: 20 icon: /integrations/providers/bw/openrouter.svg description: Route agentgateway LLM traffic to models available through OpenRouter. +test: + openrouter: + - file: ${versionRoot}/llm/providers/openrouter.md + path: openrouter --- Configure OpenRouter as an LLM provider in agentgateway. +{{< doc-test paths="openrouter" >}} +# ============================================================================ +# Doc test coverage for this guide (these comments are not rendered on the page) +# ============================================================================ +# WHAT THIS TEST VALIDATES: +# * "Configuration": the example config is accepted by agentgateway +# (--validate-only), so `provider: openrouter` is a recognized provider and the +# `name` / `params.apiKey` fields are correct. +# * The `params.baseUrl` row of the settings table: with the config loaded, +# agentgateway resolves this provider's upstream to the documented default +# (https://openrouter.ai/api/v1), so the table cannot drift from the product without +# this test failing. +# +# WHAT THIS TEST DOES NOT VALIDATE (and why): +# * "Example request" - external dependency; sending the curl request needs a +# real OpenRouter API key and bills a live completion, so the test uses a +# placeholder key and asserts on resolved config instead. +# * `params.model` - display-only table row; the example config omits it, so +# there is nothing on this page to run. +{{< reuse "agw-docs/snippets/install-agentgateway-binary.md" >}} + +# The example config reads the API key from the environment. --validate-only and +# the config dump still resolve env vars, so a placeholder is enough here. +export OPENROUTER_API_KEY="${OPENROUTER_API_KEY:-test}" +{{< /doc-test >}} + ## Configuration {{< reuse "agw-docs/snippets/review-configuration.md" >}} @@ -22,6 +52,20 @@ llm: apiKey: "$OPENROUTER_API_KEY" ``` +{{< doc-test paths="openrouter" >}} +cat <<'EOF' > config.yaml +# yaml-language-server: $schema=https://agentgateway.dev/schema/config + +llm: + models: + - name: "*" + provider: openrouter + params: + apiKey: "$OPENROUTER_API_KEY" +EOF +agentgateway -f config.yaml --validate-only +{{< /doc-test >}} + {{< reuse "agw-docs/snippets/review-configuration.md" >}} | Setting | Description | @@ -45,3 +89,25 @@ curl -X POST http://localhost:4000/v1/chat/completions \ }' ``` +{{< doc-test paths="openrouter" >}} +# Confirm the default `params.baseUrl` documented in the settings table is what +# agentgateway actually resolves. The admin config dump reports the upstream host +# and path prefix separately, so they are recombined before comparing. +agentgateway -f config.yaml & +AGW_PID=$! +trap 'kill $AGW_PID 2>/dev/null' EXIT +sleep 3 + +EXPECTED="https://openrouter.ai/api/v1" +RESOLVED=$(curl -sf --max-time 10 http://localhost:15000/config_dump | jq -r ' + [ .backends[].backend.ai + | select(. != null) + | .target.providers[].active[].endpoint + | "https://" + (.hostOverride | sub(":443$"; "")) + (.pathPrefix // "") + ] | first') +if [ "$RESOLVED" != "$EXPECTED" ]; then + echo "FAIL: settings table documents a default baseUrl of $EXPECTED but agentgateway resolved $RESOLVED" + exit 1 +fi +echo "✓ Provider default baseUrl resolves to $EXPECTED" +{{< /doc-test >}} diff --git a/content/docs/standalone/latest/llm/providers/togetherai.md b/content/docs/standalone/latest/llm/providers/togetherai.md index e4f44d1fd..cbd959969 100644 --- a/content/docs/standalone/latest/llm/providers/togetherai.md +++ b/content/docs/standalone/latest/llm/providers/togetherai.md @@ -3,10 +3,40 @@ title: Together AI weight: 20 icon: /integrations/providers/bw/togetherai.svg description: Route agentgateway LLM traffic to models hosted on Together AI. +test: + togetherai: + - file: ${versionRoot}/llm/providers/togetherai.md + path: togetherai --- Configure Together AI as an LLM provider in agentgateway. +{{< doc-test paths="togetherai" >}} +# ============================================================================ +# Doc test coverage for this guide (these comments are not rendered on the page) +# ============================================================================ +# WHAT THIS TEST VALIDATES: +# * "Configuration": the example config is accepted by agentgateway +# (--validate-only), so `provider: togetherai` is a recognized provider and the +# `name` / `params.apiKey` fields are correct. +# * The `params.baseUrl` row of the settings table: with the config loaded, +# agentgateway resolves this provider's upstream to the documented default +# (https://api.together.xyz/v1), so the table cannot drift from the product without +# this test failing. +# +# WHAT THIS TEST DOES NOT VALIDATE (and why): +# * "Example request" - external dependency; sending the curl request needs a +# real Together AI API key and bills a live completion, so the test uses a +# placeholder key and asserts on resolved config instead. +# * `params.model` - display-only table row; the example config omits it, so +# there is nothing on this page to run. +{{< reuse "agw-docs/snippets/install-agentgateway-binary.md" >}} + +# The example config reads the API key from the environment. --validate-only and +# the config dump still resolve env vars, so a placeholder is enough here. +export TOGETHER_API_KEY="${TOGETHER_API_KEY:-test}" +{{< /doc-test >}} + ## Configuration {{< reuse "agw-docs/snippets/review-configuration.md" >}} @@ -22,6 +52,20 @@ llm: apiKey: "$TOGETHER_API_KEY" ``` +{{< doc-test paths="togetherai" >}} +cat <<'EOF' > config.yaml +# yaml-language-server: $schema=https://agentgateway.dev/schema/config + +llm: + models: + - name: "*" + provider: togetherai + params: + apiKey: "$TOGETHER_API_KEY" +EOF +agentgateway -f config.yaml --validate-only +{{< /doc-test >}} + {{< reuse "agw-docs/snippets/review-configuration.md" >}} | Setting | Description | @@ -45,3 +89,25 @@ curl -X POST http://localhost:4000/v1/chat/completions \ }' ``` +{{< doc-test paths="togetherai" >}} +# Confirm the default `params.baseUrl` documented in the settings table is what +# agentgateway actually resolves. The admin config dump reports the upstream host +# and path prefix separately, so they are recombined before comparing. +agentgateway -f config.yaml & +AGW_PID=$! +trap 'kill $AGW_PID 2>/dev/null' EXIT +sleep 3 + +EXPECTED="https://api.together.xyz/v1" +RESOLVED=$(curl -sf --max-time 10 http://localhost:15000/config_dump | jq -r ' + [ .backends[].backend.ai + | select(. != null) + | .target.providers[].active[].endpoint + | "https://" + (.hostOverride | sub(":443$"; "")) + (.pathPrefix // "") + ] | first') +if [ "$RESOLVED" != "$EXPECTED" ]; then + echo "FAIL: settings table documents a default baseUrl of $EXPECTED but agentgateway resolved $RESOLVED" + exit 1 +fi +echo "✓ Provider default baseUrl resolves to $EXPECTED" +{{< /doc-test >}} diff --git a/content/docs/standalone/latest/llm/providers/vertex.md b/content/docs/standalone/latest/llm/providers/vertex.md index 85912c88b..05a68aa76 100644 --- a/content/docs/standalone/latest/llm/providers/vertex.md +++ b/content/docs/standalone/latest/llm/providers/vertex.md @@ -3,10 +3,39 @@ title: Vertex AI weight: 15 icon: /integrations/providers/bw/vertex.svg description: Route agentgateway LLM traffic to models on Google Cloud Vertex AI. +test: + vertex: + - file: ${versionRoot}/llm/providers/vertex.md + path: vertex --- Configure Google Cloud Vertex AI as an LLM provider in agentgateway. +{{< doc-test paths="vertex" >}} +# ============================================================================ +# Doc test coverage for this guide (these comments are not rendered on the page) +# ============================================================================ +# WHAT THIS TEST VALIDATES: +# * "Configuration": the example config is accepted by agentgateway +# (--validate-only), so `provider: vertex` is recognized and +# `params.model` / `params.vertexProject` / `params.vertexRegion` are correct. +# * The settings table rows for `name`, `params.model`, `params.vertexProject`, +# and `params.vertexRegion`: with the config loaded, agentgateway serves the +# client-facing model name `gemini-2.5-flash` and resolves the upstream to the +# configured model, project ID, and region. This makes the distinction between +# `name` (matched in requests) and `params.model` (sent upstream) observable +# rather than only asserted in prose. +# +# WHAT THIS TEST DOES NOT VALIDATE (and why): +# * "Authentication" - external dependency; Application Default Credentials +# require a real Google Cloud identity, which the test cannot stand up. The +# config loads without credentials because ADC is resolved per request. +# * The `auth.gcp` table row - display-only; the example config omits it and +# relies on the ADC default. +# * That a completion reaches Vertex AI - external dependency, as above. +{{< reuse "agw-docs/snippets/install-agentgateway-binary.md" >}} +{{< /doc-test >}} + ## Authentication Before you can use Vertex AI as an LLM provider, you must authenticate by using Google Cloud's [Application Default Credentials](https://docs.cloud.google.com/docs/authentication/application-default-credentials). Choose from one of the three methods: @@ -32,6 +61,22 @@ llm: vertexRegion: us-west2 ``` +{{< doc-test paths="vertex" >}} +cat <<'EOF' > config.yaml +# yaml-language-server: $schema=https://agentgateway.dev/schema/config + +llm: + models: + - name: gemini-2.5-flash + provider: vertex + params: + model: google/gemini-2.5-flash-lite-preview-06-17 + vertexProject: my-project-id + vertexRegion: us-west2 +EOF +agentgateway -f config.yaml --validate-only +{{< /doc-test >}} + {{< reuse "agw-docs/snippets/review-configuration.md" >}} | Setting | Description | @@ -42,3 +87,32 @@ llm: | `params.vertexProject` | The Google Cloud project ID. | | `params.vertexRegion` | The Google Cloud region. Defaults to `global` if not specified. | | `auth.gcp` | Google Cloud authentication configuration. Uses Application Default Credentials (ADC) by default. | + +{{< doc-test paths="vertex" >}} +# Confirm the client-facing `name` is served and that `params.model`, +# `params.vertexProject`, and `params.vertexRegion` reach the resolved provider +# config as documented in the settings table. +agentgateway -f config.yaml & +AGW_PID=$! +trap 'kill $AGW_PID 2>/dev/null' EXIT +sleep 3 + +SERVED=$(curl -sf --max-time 10 http://localhost:4000/v1/models | jq -r '[.data[].id] | index("gemini-2.5-flash") // "missing"') +if [ "$SERVED" = "missing" ]; then + echo "FAIL: the model name gemini-2.5-flash from the example config is not served" + exit 1 +fi +RESOLVED=$(curl -sf --max-time 10 http://localhost:15000/config_dump | jq -r ' + [ .backends[].backend.ai + | select(. != null) + | .target.providers[].active[].endpoint + | .provider.vertex + | "\(.model)|\(.projectId)|\(.region)" + ] | first') +EXPECTED="google/gemini-2.5-flash-lite-preview-06-17|my-project-id|us-west2" +if [ "$RESOLVED" != "$EXPECTED" ]; then + echo "FAIL: expected vertex params $EXPECTED but agentgateway resolved $RESOLVED" + exit 1 +fi +echo "✓ Vertex model, project, and region resolve to the documented values" +{{< /doc-test >}} diff --git a/content/docs/standalone/latest/llm/providers/xai.md b/content/docs/standalone/latest/llm/providers/xai.md index d61e1bd57..23025e91b 100644 --- a/content/docs/standalone/latest/llm/providers/xai.md +++ b/content/docs/standalone/latest/llm/providers/xai.md @@ -3,10 +3,40 @@ title: xAI weight: 20 icon: /integrations/providers/bw/xai.svg description: Route agentgateway LLM traffic to xAI's Grok models. +test: + xai: + - file: ${versionRoot}/llm/providers/xai.md + path: xai --- Configure xAI (Grok) as an LLM provider in agentgateway. +{{< doc-test paths="xai" >}} +# ============================================================================ +# Doc test coverage for this guide (these comments are not rendered on the page) +# ============================================================================ +# WHAT THIS TEST VALIDATES: +# * "Configuration": the example config is accepted by agentgateway +# (--validate-only), so `provider: xai` is a recognized provider and the +# `name` / `params.apiKey` fields are correct. +# * The `params.baseUrl` row of the settings table: with the config loaded, +# agentgateway resolves this provider's upstream to the documented default +# (https://api.x.ai/v1), so the table cannot drift from the product without +# this test failing. +# +# WHAT THIS TEST DOES NOT VALIDATE (and why): +# * "Example request" - external dependency; sending the curl request needs a +# real xAI API key and bills a live completion, so the test uses a +# placeholder key and asserts on resolved config instead. +# * `params.model` - display-only table row; the example config omits it, so +# there is nothing on this page to run. +{{< reuse "agw-docs/snippets/install-agentgateway-binary.md" >}} + +# The example config reads the API key from the environment. --validate-only and +# the config dump still resolve env vars, so a placeholder is enough here. +export XAI_API_KEY="${XAI_API_KEY:-test}" +{{< /doc-test >}} + ## Configuration {{< reuse "agw-docs/snippets/review-configuration.md" >}} @@ -22,6 +52,20 @@ llm: apiKey: "$XAI_API_KEY" ``` +{{< doc-test paths="xai" >}} +cat <<'EOF' > config.yaml +# yaml-language-server: $schema=https://agentgateway.dev/schema/config + +llm: + models: + - name: "*" + provider: xai + params: + apiKey: "$XAI_API_KEY" +EOF +agentgateway -f config.yaml --validate-only +{{< /doc-test >}} + {{< reuse "agw-docs/snippets/review-configuration.md" >}} | Setting | Description | @@ -45,3 +89,25 @@ curl -X POST http://localhost:4000/v1/chat/completions \ }' ``` +{{< doc-test paths="xai" >}} +# Confirm the default `params.baseUrl` documented in the settings table is what +# agentgateway actually resolves. The admin config dump reports the upstream host +# and path prefix separately, so they are recombined before comparing. +agentgateway -f config.yaml & +AGW_PID=$! +trap 'kill $AGW_PID 2>/dev/null' EXIT +sleep 3 + +EXPECTED="https://api.x.ai/v1" +RESOLVED=$(curl -sf --max-time 10 http://localhost:15000/config_dump | jq -r ' + [ .backends[].backend.ai + | select(. != null) + | .target.providers[].active[].endpoint + | "https://" + (.hostOverride | sub(":443$"; "")) + (.pathPrefix // "") + ] | first') +if [ "$RESOLVED" != "$EXPECTED" ]; then + echo "FAIL: settings table documents a default baseUrl of $EXPECTED but agentgateway resolved $RESOLVED" + exit 1 +fi +echo "✓ Provider default baseUrl resolves to $EXPECTED" +{{< /doc-test >}} diff --git a/content/docs/standalone/latest/llm/virtual-models.md b/content/docs/standalone/latest/llm/virtual-models.md index c9c0e698a..eeebacd40 100644 --- a/content/docs/standalone/latest/llm/virtual-models.md +++ b/content/docs/standalone/latest/llm/virtual-models.md @@ -2,8 +2,66 @@ title: Virtual models weight: 47 description: Configure virtual models with weighted, failover, and conditional routing in simplified LLM mode. +test: + virtual-models: + - file: ${versionRoot}/llm/virtual-models.md + path: virtual-models --- +{{< doc-test paths="virtual-models" >}} +# ============================================================================ +# Doc test coverage for this guide (these comments are not rendered on the page) +# ============================================================================ +# WHAT THIS TEST VALIDATES: +# * All three example configs are accepted by agentgateway (--validate-only), +# covering `llm.virtualModels[].routing.weighted.targets[].weight`, +# `routing.failover.targets[].priority`, and `routing.conditional.targets[].when`. +# * "Public and internal models": with each config loaded, the served model list +# contains the virtual model and any `visibility: public` model, and omits every +# `visibility: internal` model. This turns the prose description of `public` and +# `internal` into an observable assertion: +# - weighted -> gpt-4o-public, smart (2 internal targets hidden) +# - failover -> resilient (all 3 targets are internal) +# - conditional -> openai-public, adaptive (2 internal targets hidden) +# The failover case is the clearest: every target is internal, so only the +# virtual entrypoint is exposed. +# +# WHAT THIS TEST DOES NOT VALIDATE (and why): +# * That traffic is actually split 90/10 by `weight` - external dependency; +# observing the split needs many live completions against OpenAI. +# * That failover moves to a lower `priority` target on failure, and that +# same-priority targets are load balanced "based on health and latency" - +# external dependency; triggering a real upstream failure needs live providers. +# * That `when` expressions select a target by request header - requires +# config/traffic the page omits; the page shows no request example, and +# confirming which internal target served a response needs a live provider +# call. +{{< reuse "agw-docs/snippets/install-agentgateway-binary.md" >}} + +# The example configs read API keys from the environment. --validate-only and the +# model listing still resolve env vars, so placeholders are enough here. +export OPENAI_API_KEY="${OPENAI_API_KEY:-test}" +export ANTHROPIC_API_KEY="${ANTHROPIC_API_KEY:-test}" + +# Assert that a config serves exactly the expected client-facing models, which is +# what `visibility: public` / `internal` controls. +assert_models() { + local cfg="$1" expected="$2" + agentgateway -f "$cfg" & + local pid=$! + sleep 3 + local served + served=$(curl -sf --max-time 10 http://localhost:4000/v1/models | jq -cr '[.data[].id] | sort') + kill $pid 2>/dev/null + wait $pid 2>/dev/null + if [ "$served" != "$expected" ]; then + echo "FAIL: $cfg should serve $expected but served $served" + exit 1 + fi + echo "✓ $cfg serves $expected (internal targets are not exposed)" +} +{{< /doc-test >}} + Virtual models let you publish one client-facing model name and route requests across one or more internal target models. Use `llm.virtualModels[]` to define the virtual entrypoint and `llm.models[]` as the concrete upstream targets. @@ -57,6 +115,43 @@ llm: weight: 10 ``` +{{< doc-test paths="virtual-models" >}} +cat <<'EOF' > config-weighted.yaml +llm: + models: + - name: gpt-4o-public + visibility: public + provider: openAI + params: + model: gpt-4o + apiKey: "$OPENAI_API_KEY" + - name: gpt-4o-primary + visibility: internal + provider: openAI + params: + model: gpt-4o + apiKey: "$OPENAI_API_KEY" + - name: gpt-4o-fallback + visibility: internal + provider: openAI + params: + model: gpt-4o-mini + apiKey: "$OPENAI_API_KEY" + + virtualModels: + - name: smart + routing: + weighted: + targets: + - model: gpt-4o-primary + weight: 90 + - model: gpt-4o-fallback + weight: 10 +EOF +agentgateway -f config-weighted.yaml --validate-only +assert_models config-weighted.yaml '["gpt-4o-public","smart"]' +{{< /doc-test >}} + ### Failover routing Use `routing.failover.targets` and `priority` to define ordered failover targets. @@ -97,6 +192,45 @@ llm: priority: 2 ``` +{{< doc-test paths="virtual-models" >}} +cat <<'EOF' > config-failover.yaml +llm: + models: + - name: claude-primary + visibility: internal + provider: anthropic + params: + model: claude-sonnet-4-0 + apiKey: "$ANTHROPIC_API_KEY" + - name: claude-backup-a + visibility: internal + provider: anthropic + params: + model: claude-3-5-haiku-20241022 + apiKey: "$ANTHROPIC_API_KEY" + - name: claude-backup-b + visibility: internal + provider: anthropic + params: + model: claude-3-5-haiku-20241022 + apiKey: "$ANTHROPIC_API_KEY" + + virtualModels: + - name: resilient + routing: + failover: + targets: + - model: claude-primary + priority: 1 + - model: claude-backup-a + priority: 2 + - model: claude-backup-b + priority: 2 +EOF +agentgateway -f config-failover.yaml --validate-only +assert_models config-failover.yaml '["resilient"]' +{{< /doc-test >}} + ### Conditional routing Use `routing.conditional.targets` and `when` expressions to select targets by request context. @@ -134,5 +268,42 @@ llm: when: request.headers["x-tier"] == "pro" ``` +{{< doc-test paths="virtual-models" >}} +cat <<'EOF' > config-conditional.yaml +llm: + models: + - name: openai-public + visibility: public + provider: openAI + params: + model: gpt-4o-mini + apiKey: "$OPENAI_API_KEY" + - name: openai-fast + visibility: internal + provider: openAI + params: + model: gpt-4o-mini + apiKey: "$OPENAI_API_KEY" + - name: openai-smart + visibility: internal + provider: openAI + params: + model: gpt-4o + apiKey: "$OPENAI_API_KEY" + + virtualModels: + - name: adaptive + routing: + conditional: + targets: + - model: openai-fast + when: request.headers["x-tier"] == "free" + - model: openai-smart + when: request.headers["x-tier"] == "pro" +EOF +agentgateway -f config-conditional.yaml --validate-only +assert_models config-conditional.yaml '["adaptive","openai-public"]' +{{< /doc-test >}} + > [!NOTE] > For reusable provider defaults in simplified mode, see [Multiple LLM providers]({{< link-hextra path="/llm/providers/multiple-llms/" >}}). diff --git a/content/docs/standalone/latest/mcp/connect/stdio.md b/content/docs/standalone/latest/mcp/connect/stdio.md index 2894071c3..d0f092d91 100644 --- a/content/docs/standalone/latest/mcp/connect/stdio.md +++ b/content/docs/standalone/latest/mcp/connect/stdio.md @@ -2,10 +2,56 @@ title: Stdio weight: 10 description: Run a local MCP server as a subprocess and expose it through agentgateway over stdio. +test: + mcp-stdio: + - file: ${versionRoot}/mcp/connect/stdio.md + path: mcp-stdio --- An MCP backend allows exposing MCP servers through the agentgateway using {{< gloss "STDIO (Standard Input/Output)" >}}STDIO{{< /gloss >}}. +{{< doc-test paths="mcp-stdio" >}} +# ============================================================================ +# Doc test coverage for this guide (these comments are not rendered on the page) +# ============================================================================ +# WHAT THIS TEST VALIDATES: +# * "Configure the agentgateway" step 1: the documented download URL resolves and +# returns a config that agentgateway accepts (--validate-only). +# * "Verify access to tools" steps 2-3, through the MCP API rather than the UI +# playground: an MCP session initializes, tools/list includes the `echo` tool +# that the page tells you to select, and calling `echo` with the page's example +# message returns that message echoed back. These are the UI steps' scriptable +# equivalents, so the walkthrough's end state is verified even though the +# clicks are not. +# +# WHAT THIS TEST DOES NOT VALIDATE (and why): +# * The agentgateway UI itself (opening the UI, the Tool Playground, the +# "Apply CORS" button, the Result card screenshots) - UI-only steps with no +# command-line equivalent. The test drives the same MCP endpoint the playground +# drives. +# * The github-yaml rendering of the config in step 2 - display-only; it +# embeds the same file the test downloads and validates. +{{< reuse "agw-docs/snippets/install-agentgateway-binary.md" >}} + +# Open an MCP session and return its session ID. MCP responses are server-sent +# events, so `data:` lines are unwrapped before parsing. +mcp_session() { + curl -sS -D - -o /dev/null --max-time 30 -X POST http://localhost:3000/mcp \ + -H 'Content-Type: application/json' \ + -H 'Accept: application/json, text/event-stream' \ + -d '{"jsonrpc":"2.0","method":"initialize","params":{"protocolVersion":"2024-11-05","capabilities":{},"clientInfo":{"name":"doctest","version":"1.0"}},"id":1}' \ + | grep -i '^mcp-session-id:' | tr -d '\r' | awk '{print $2}' +} + +mcp_call() { + curl -sS --max-time 30 -X POST http://localhost:3000/mcp \ + -H 'Content-Type: application/json' \ + -H 'Accept: application/json, text/event-stream' \ + -H "Mcp-Session-Id: $1" \ + -d "$2" | sed -n 's/^data: //p' +} +{{< /doc-test >}} + ## Before you begin {{< reuse "agw-docs/snippets/prereq-agentgateway.md" >}} @@ -14,10 +60,14 @@ An MCP backend allows exposing MCP servers through the agentgateway using {{< gl 1. Download an MCP configuration for your agentgateway. - ```yaml + ```yaml {paths="mcp-stdio"} curl -L https://agentgateway.dev/examples/mcp-basic/config.yaml -o config.yaml ``` + {{< doc-test paths="mcp-stdio" >}} + agentgateway -f config.yaml --validate-only + {{< /doc-test >}} + 2. Review the configuration file. ``` @@ -55,3 +105,55 @@ An MCP backend allows exposing MCP servers through the agentgateway using {{< gl {{< reuse-image-light src="img/ui-playground-tool-echo.png" >}} {{< reuse-image-dark srcDark="img/ui-playground-tool-echo-dark.png" >}} + +{{< doc-test paths="mcp-stdio" >}} +# Run the gateway in the background so the MCP assertions below can drive it. The +# visible "Run the agentgateway" block is untagged because it runs in the foreground. +agentgateway -f config.yaml & +AGW_PID=$! +trap 'kill $AGW_PID 2>/dev/null' EXIT +# The stdio target launches the MCP server through npx, which downloads the package +# on first use, so allow time for the target to become ready. +for i in $(seq 1 30); do + curl -sf -o /dev/null --max-time 5 http://localhost:15021/healthz/ready && break + sleep 2 +done +{{< /doc-test >}} + +{{< doc-test paths="mcp-stdio" >}} +# The API equivalent of the "Verify access to tools" playground steps: open a +# session, confirm the `echo` tool the page tells you to select is listed, then call +# it with the page's example message and check the echoed result. +SESSION="" +for i in $(seq 1 20); do + SESSION=$(mcp_session) + [ -n "$SESSION" ] && break + sleep 3 +done +if [ -z "$SESSION" ]; then + echo "FAIL: could not open an MCP session against the configured target" + exit 1 +fi +echo "✓ MCP session initialized" + +mcp_call "$SESSION" '{"jsonrpc":"2.0","method":"notifications/initialized"}' >/dev/null + +TOOLS=$(mcp_call "$SESSION" '{"jsonrpc":"2.0","method":"tools/list","id":2}') +if [ "$(jq -r '[.result.tools[].name] | index("echo") // "missing"' <<<"$TOOLS")" = "missing" ]; then + echo "FAIL: tools/list did not include the echo tool" + jq -c '[.result.tools[].name]' <<<"$TOOLS" + exit 1 +fi +echo "✓ tools/list includes the echo tool" + +RESULT=$(mcp_call "$SESSION" '{"jsonrpc":"2.0","method":"tools/call","params":{"name":"echo","arguments":{"message":"This is my first agentgateway setup."}},"id":3}') +TEXT=$(jq -r '.result.content[0].text // ""' <<<"$RESULT") +case "$TEXT" in + *"This is my first agentgateway setup."*) + echo "✓ Calling the echo tool returned the message: $TEXT" ;; + *) + echo "FAIL: the echo tool did not echo the message back" + echo "$RESULT" + exit 1 ;; +esac +{{< /doc-test >}} diff --git a/content/docs/standalone/latest/mcp/connect/virtual.md b/content/docs/standalone/latest/mcp/connect/virtual.md index 36234a5c1..cb8389e22 100644 --- a/content/docs/standalone/latest/mcp/connect/virtual.md +++ b/content/docs/standalone/latest/mcp/connect/virtual.md @@ -2,10 +2,118 @@ title: Virtual MCP weight: 20 description: Federate multiple MCP servers into a unified virtual MCP backend +test: + mcp-virtual: + - file: ${versionRoot}/mcp/connect/virtual.md + path: mcp-virtual --- Federate tools of multiple MCP servers on the agentgateway by using MCP {{< gloss "Multiplex" >}}multiplexing{{< /gloss >}}. +{{< doc-test paths="mcp-virtual" >}} +# ============================================================================ +# Doc test coverage for this guide (these comments are not rendered on the page) +# ============================================================================ +# WHAT THIS TEST VALIDATES: +# * "Configure the agentgateway" step 1: the documented download URL resolves and +# returns a config that agentgateway accepts (--validate-only). +# * "Verify access to tools" steps 5-7, through the MCP API rather than the UI +# playground: the downloaded multiplex config runs, tools/list returns tools +# federated from both targets with target-name prefixes +# (`time_get_current_time`, `everything_echo`), calling `everything_echo` +# echoes the page's example message, and calling `time_get_current_time` with +# `America/New_York` returns a time result. +# * "Tool name prefixing": the `prefixMode: never` example config is accepted, and +# all three rows of the prefixMode table are asserted at runtime against a live +# MCP session: +# - conditional (default), two targets -> names are prefixed +# - always, one target -> names are prefixed even with one target +# - never -> names are plain (echo) +# +# WHAT THIS TEST DOES NOT VALIDATE (and why): +# * The published multiplex config's `time` target needs `uvx mcp-server-time` to +# resolve an MCP Python SDK older than 2.x, because that release renamed +# McpError and the server fails to import against it. agentgateway#2873 adds the +# `--with mcp<2` constraint to the example; until that lands and redeploys, the +# test applies the same constraint to its local copy. Once the published config +# carries it, the test runs the downloaded file verbatim. +# * The agentgateway UI steps (Tool Playground, Apply CORS, Initialize, the +# screenshots) - UI-only, no command-line equivalent. The test drives the same +# MCP endpoint the playground drives. +# * The two collapsed "details" example configs ("Example multiplexing configuration" +# and "Example load balancing configuration") - display-only structural excerpts; +# neither is a complete config (no gateways/routes, and the backends entry omits +# its required `name`), so neither can be validated as written. +# * The step 3 optional CORS config - display-only; it is an abbreviated snippet +# ending in `...`, not a complete file. +# * That load balancing distributes across backends by weight - requires +# config/traffic the page omits; the load balancing example is contrast material, +# not a walkthrough. +{{< reuse "agw-docs/snippets/install-agentgateway-binary.md" >}} + +# The stdio targets launch their MCP servers through npx and uvx. Fetch both up +# front: otherwise the first start pays a cold registry download inside every +# readiness retry loop below, which is slow enough to time the test out. +npm install -g @modelcontextprotocol/server-everything >/dev/null 2>&1 || true + +# "Before you begin" step 2 installs uv. The agentgateway install snippet already put +# $HOME/.local/bin on PATH, which is where the uv installer places its binaries. +if ! command -v uvx >/dev/null 2>&1; then + curl -LsSf https://astral.sh/uv/install.sh | sh >/dev/null 2>&1 || true +fi +uvx --with 'mcp<2' mcp-server-time --help >/dev/null 2>&1 || true + +# Open an MCP session and list the tool names it exposes. MCP responses are +# server-sent events, so `data:` lines are unwrapped before parsing. +mcp_tool_names() { + local sid + sid=$(curl -sS -D - -o /dev/null --max-time 10 -X POST http://localhost:3000/mcp \ + -H 'Content-Type: application/json' -H 'Accept: application/json, text/event-stream' \ + -d '{"jsonrpc":"2.0","method":"initialize","params":{"protocolVersion":"2024-11-05","capabilities":{},"clientInfo":{"name":"doctest","version":"1.0"}},"id":1}' \ + | grep -i '^mcp-session-id:' | tr -d '\r' | awk '{print $2}') + [ -n "$sid" ] || return 1 + curl -sS --max-time 10 -o /dev/null -X POST http://localhost:3000/mcp \ + -H 'Content-Type: application/json' -H 'Accept: application/json, text/event-stream' \ + -H "Mcp-Session-Id: $sid" -d '{"jsonrpc":"2.0","method":"notifications/initialized"}' + echo "$sid" > .mcp-session + curl -sS --max-time 15 -X POST http://localhost:3000/mcp \ + -H 'Content-Type: application/json' -H 'Accept: application/json, text/event-stream' \ + -H "Mcp-Session-Id: $sid" -d '{"jsonrpc":"2.0","method":"tools/list","id":2}' \ + | sed -n 's/^data: //p' | jq -r '[.result.tools[].name] | join(" ")' +} + +# Start a config in the background. This must NOT be called inside a command +# substitution: AGW_PID would be set in the subshell and the parent would never stop +# the gateway, leaving port 3000 held for the next config. The gateway's output goes +# to a file for the same reason -- inside `$( )` it would be captured into the value. +start_gateway() { + agentgateway -f "$1" > "agw-$1.log" 2>&1 & + AGW_PID=$! +} + +# Wait for the stdio targets to come up and echo the multiplexed tool names. Pure +# curl, no background jobs, so it is safe to call inside a command substitution. +# ~15 attempts x (10s max curl + 2s sleep) bounds this at about 3 minutes. +wait_for_tools() { + local names="" + for i in $(seq 1 15); do + names=$(mcp_tool_names 2>/dev/null || true) + [ -n "$names" ] && break + sleep 2 + done + echo "$names" +} + +stop_gateway() { + [ -n "${AGW_PID:-}" ] || return 0 + kill "$AGW_PID" 2>/dev/null || true + wait "$AGW_PID" 2>/dev/null || true + AGW_PID="" +} + +trap 'stop_gateway' EXIT +{{< /doc-test >}} + ## About multiplexing {#about} Multiplexing combines multiple MCP servers (targets) within a single backend into one unified MCP server. All targets are exposed together so that clients can access tools from all targets simultaneously. By default, when a backend has more than one target, tool names are prefixed with the target name (e.g., `time_get_current_time`, `everything_echo`) to avoid collisions. You can change this behavior with the `prefixMode` field, described in [Tool name prefixing](#tool-name-prefixing). @@ -20,7 +128,7 @@ backends: - name: time stdio: cmd: uvx - args: ["mcp-server-time"] + args: ["--with", "mcp<2", "mcp-server-time"] - name: everything stdio: cmd: npx @@ -59,10 +167,15 @@ routes: 1. Download a multiplex configuration for your agentgateway. - ```yaml + ```yaml {paths="mcp-virtual"} curl -L https://agentgateway.dev/examples/mcp-multiplex/config.yaml -o config.yaml ``` + {{< doc-test paths="mcp-virtual" >}} + # Step 1: the documented multiplex config downloads and is accepted + agentgateway -f config.yaml --validate-only + {{< /doc-test >}} + 2. Review the configuration file. ``` @@ -150,13 +263,122 @@ mcp: - name: time stdio: cmd: uvx - args: ["mcp-server-time"] + args: ["--with", "mcp<2", "mcp-server-time"] - name: everything stdio: cmd: npx args: ["@modelcontextprotocol/server-everything"] ``` +> [!NOTE] +> The `time` target pins the MCP Python SDK with `--with mcp<2` because `mcp-server-time` does not yet support version 2.x of the SDK. Without the constraint, the target fails to start. Drop the constraint after `mcp-server-time` adds support. + ## Next steps - Apply different policies to different MCP targets with [MCP target policies]({{< link-hextra path="/mcp/mcp-target-policies/" >}}). + +{{< doc-test paths="mcp-virtual" >}} +# Run the downloaded multiplex config and assert the federated tool list and both +# tool calls from "Verify access to tools" steps 5-7. +# +# The published example's `time` target needs an MCP Python SDK older than 2.x +# (agentgateway#2873). Use the downloaded file as-is once it carries that +# constraint; until then, apply the same constraint to a local copy so the target +# can start. +if grep -q 'mcp<2' config.yaml; then + cp config.yaml config-multiplex.yaml +else + sed 's/args: \["mcp-server-time"\]/args: ["--with", "mcp<2", "mcp-server-time"]/' \ + config.yaml > config-multiplex.yaml +fi +agentgateway -f config-multiplex.yaml --validate-only + +start_gateway config-multiplex.yaml +NAMES=$(wait_for_tools) +case "$NAMES" in + *time_get_current_time*) ;; + *) echo "FAIL: tools/list did not include time_get_current_time from the time target" + echo "$NAMES"; exit 1 ;; +esac +case "$NAMES" in + *everything_echo*) ;; + *) echo "FAIL: tools/list did not include everything_echo from the everything target" + echo "$NAMES"; exit 1 ;; +esac +echo "✓ Step 5: tools/list federates both targets with target-name prefixes" + +SESSION=$(cat .mcp-session) +mcp_tool_call() { + curl -sS --max-time 15 -X POST http://localhost:3000/mcp \ + -H 'Content-Type: application/json' -H 'Accept: application/json, text/event-stream' \ + -H "Mcp-Session-Id: $SESSION" -d "$1" | sed -n 's/^data: //p' +} + +RESULT=$(mcp_tool_call '{"jsonrpc":"2.0","method":"tools/call","params":{"name":"everything_echo","arguments":{"message":"hello world"}},"id":3}') +case "$(jq -r '.result.content[0].text // ""' <<<"$RESULT")" in + *"hello world"*) echo "✓ Step 6: everything_echo routes to the everything target and echoes the message" ;; + *) echo "FAIL: everything_echo did not return the message"; echo "$RESULT"; exit 1 ;; +esac + +RESULT=$(mcp_tool_call '{"jsonrpc":"2.0","method":"tools/call","params":{"name":"time_get_current_time","arguments":{"timezone":"America/New_York"}},"id":4}') +case "$(jq -r '.result.content[0].text // ""' <<<"$RESULT")" in + *America/New_York*) echo "✓ Step 7: time_get_current_time routes to the time target and returns a time" ;; + *) echo "FAIL: time_get_current_time did not return a result for America/New_York"; echo "$RESULT"; exit 1 ;; +esac +stop_gateway +echo "✓ prefixMode conditional (default): two targets produce prefixed names" +{{< /doc-test >}} + +{{< doc-test paths="mcp-virtual" >}} +# "Tool name prefixing": validate the documented prefixMode: never config, then assert +# the always and never rows of the table. Both use a single npx target, because +# `always` is only distinguishable from the default with one target and `never` +# requires names that do not collide. +cat <<'EOF' > config-prefix-never.yaml +# yaml-language-server: $schema=https://agentgateway.dev/schema/config +mcp: + port: 3000 + prefixMode: never + targets: + - name: time + stdio: + cmd: uvx + args: ["--with", "mcp<2", "mcp-server-time"] + - name: everything + stdio: + cmd: npx + args: ["@modelcontextprotocol/server-everything"] +EOF +agentgateway -f config-prefix-never.yaml --validate-only + +cat <<'EOF' > config-always.yaml +mcp: + port: 3000 + prefixMode: always + targets: + - name: alpha + stdio: + cmd: npx + args: ["@modelcontextprotocol/server-everything"] +EOF +sed 's/prefixMode: always/prefixMode: never/' config-always.yaml > config-never.yaml +agentgateway -f config-always.yaml --validate-only >/dev/null +agentgateway -f config-never.yaml --validate-only >/dev/null + +start_gateway config-always.yaml +NAMES=$(wait_for_tools) +stop_gateway +case "$NAMES" in + *alpha_echo*) echo "✓ prefixMode always: a single target still produces prefixed names" ;; + *) echo "FAIL: prefixMode always did not prefix names for a single target"; echo "$NAMES"; exit 1 ;; +esac + +start_gateway config-never.yaml +NAMES=$(wait_for_tools) +stop_gateway +case "$NAMES" in + *alpha_echo*) echo "FAIL: prefixMode never still prefixed names"; echo "$NAMES"; exit 1 ;; + *echo*) echo "✓ prefixMode never: names are unprefixed (echo)" ;; + *) echo "FAIL: prefixMode never did not expose the echo tool"; echo "$NAMES"; exit 1 ;; +esac +{{< /doc-test >}} diff --git a/content/docs/standalone/latest/mcp/mcp-target-policies.md b/content/docs/standalone/latest/mcp/mcp-target-policies.md index 982d616ce..42b1a7fa3 100644 --- a/content/docs/standalone/latest/mcp/mcp-target-policies.md +++ b/content/docs/standalone/latest/mcp/mcp-target-policies.md @@ -2,10 +2,50 @@ title: MCP target policies weight: 50 description: Scope policies to a single MCP server inside a multiplexed (virtual) MCP backend. +test: + mcp-target-policies: + - file: ${versionRoot}/mcp/mcp-target-policies.md + path: mcp-target-policies --- Apply policies at the MCP target level to control behavior for individual MCP servers within a multiplexed backend. +{{< doc-test paths="mcp-target-policies" >}} +# ============================================================================ +# Doc test coverage for this guide (these comments are not rendered on the page) +# ============================================================================ +# WHAT THIS TEST VALIDATES: +# * "Authentication per target": the example config is accepted by agentgateway +# (--validate-only), covering `mcp.targets[].policies` with `backendAuth.key` +# and `backendTLS.hostname` set per target. This example documented a +# `backendTLS.sni` field until this test was added; the schema calls it +# `hostname` ("Server name to use for TLS verification and SNI"), and `sni` was +# rejected as an unknown field. +# * "Supported policy types": each of the three listed policies +# (`backendAuth`, `backendTLS`, `requestHeaderModifier`) is accepted at the MCP +# target level. The table also listed `responseHeaderModifier` until this test +# was added; agentgateway rejects it there as an unknown field, so it moved to +# the unsupported note. +# * "Policy inheritance": a config that sets a policy at both the backend group +# level and the target level is accepted, so the documented two-level shape is +# valid. +# * All three unsupported policies from the note (`mcpAuthorization`, `ai`, `a2a`) +# are rejected as unknown fields at the target level, same as +# `responseHeaderModifier`. +# +# WHAT THIS TEST DOES NOT VALIDATE (and why): +# * That target-level policies actually override backend-level ones at request +# time - requires config/traffic the page omits; both example targets point at +# placeholder MCP servers (service-a.example.com) that the test cannot stand up, +# and the page shows no request to inspect. +# * The "Best practices" bullets - prose guidance, not runnable. +{{< reuse "agw-docs/snippets/install-agentgateway-binary.md" >}} + +export SERVICE_A_API_KEY="${SERVICE_A_API_KEY:-test}" +export SERVICE_B_API_KEY="${SERVICE_B_API_KEY:-test}" +{{< /doc-test >}} + + ## Overview MCP target policies allow you to configure policies for specific MCP backend targets, rather than applying them globally to all targets in a backend. This is useful when you have multiple MCP servers with different authentication or routing requirements. @@ -28,12 +68,12 @@ The following policies can be configured at the MCP target level. | `backendAuth` | Backend authentication (API key, passthrough, AWS, GCP, Azure) | | `backendTLS` | TLS configuration for backend connections | | `requestHeaderModifier` | Modify request headers | -| `responseHeaderModifier` | Modify response headers | > **Note:** The following policies are **not supported** at the MCP target level. They must be configured at the backend level instead: > - `mcpAuthorization`: Fine-grained authorization rules for tools, prompts, and resources. > - `ai`: LLM processing policies such as prompt guards, overrides, defaults, and model aliases. > - `a2a`: Mark traffic as agent-to-agent. +> - `responseHeaderModifier`: Modify response headers. Target-level policies apply to the connection that agentgateway opens to the target, so configure response header changes on the route or the backend instead. ### Policy inheritance @@ -68,7 +108,7 @@ mcp: backendAuth: key: "$SERVICE_A_API_KEY" backendTLS: - sni: service-a.example.com + hostname: service-a.example.com - name: service-b mcp: @@ -77,11 +117,114 @@ mcp: backendAuth: key: "$SERVICE_B_API_KEY" backendTLS: - sni: service-b.example.com + hostname: service-b.example.com ``` +{{< doc-test paths="mcp-target-policies" >}} +cat <<'EOF' > config.yaml +# yaml-language-server: $schema=https://agentgateway.dev/schema/config +mcp: + port: 3000 + targets: + - name: service-a + mcp: + host: https://service-a.example.com/mcp + policies: + backendAuth: + key: "$SERVICE_A_API_KEY" + backendTLS: + hostname: service-a.example.com + + - name: service-b + mcp: + host: https://service-b.example.com/mcp + policies: + backendAuth: + key: "$SERVICE_B_API_KEY" + backendTLS: + hostname: service-b.example.com +EOF +agentgateway -f config.yaml --validate-only +{{< /doc-test >}} + ## Learn more - [MCP Authorization]({{< link-hextra path="/mcp/mcp-authz" >}}) - [Backend Authentication]({{< link-hextra path="/configuration/security/backend-authn" >}}) - [Configuration Reference]({{< link-hextra path="/reference/configuration/schema/" >}}) + +{{< doc-test paths="mcp-target-policies" >}} +# "Supported policy types": the two header-modifier policies the table lists are also +# accepted at the target level, and "Policy inheritance": a policy set at the backend +# group level alongside a target-level override is a valid shape. +cat <<'EOF' > config-all-policies.yaml +# yaml-language-server: $schema=https://agentgateway.dev/schema/config +gateways: + default: + port: 3000 +routes: +- backends: + - mcp: + targets: + - name: service-a + mcp: + host: https://service-a.example.com/mcp + policies: + requestHeaderModifier: + add: + x-target: service-a + backendAuth: + key: "$SERVICE_A_API_KEY" + backendTLS: + hostname: service-a.example.com + policies: + backendAuth: + key: "$SERVICE_B_API_KEY" + responseHeaderModifier: + add: + x-from-backend: service-a +EOF +agentgateway -f config-all-policies.yaml --validate-only +echo "✓ All three documented target-level policies plus the backend/target inheritance shape are accepted" + +# The unsupported note says responseHeaderModifier belongs on the route or backend, +# not the target. Confirm agentgateway actually rejects it at the target level, so the +# note cannot drift back to claiming it is supported. +cat <<'EOF' > config-bad-target-policy.yaml +mcp: + port: 3000 + targets: + - name: service-a + mcp: + host: https://service-a.example.com/mcp + policies: + responseHeaderModifier: + add: + x-from-target: service-a +EOF +if agentgateway -f config-bad-target-policy.yaml --validate-only >/dev/null 2>&1; then + echo "FAIL: responseHeaderModifier was accepted at the MCP target level, so the unsupported note is now wrong" + exit 1 +fi +echo "✓ responseHeaderModifier is rejected at the MCP target level, as the note states" + +# The note also lists mcpAuthorization, ai, and a2a as unsupported at the target +# level. Confirm all three are rejected the same way responseHeaderModifier is. +for policy in mcpAuthorization ai a2a; do + cat < "config-bad-$policy.yaml" +mcp: + port: 3000 + targets: + - name: service-a + mcp: + host: https://service-a.example.com/mcp + policies: + $policy: {} +EOF + if agentgateway -f "config-bad-$policy.yaml" --validate-only >/dev/null 2>&1; then + echo "FAIL: $policy was accepted at the MCP target level, so the unsupported note is now wrong" + exit 1 + fi + echo "✓ $policy is rejected at the MCP target level, as the note states" +done +{{< /doc-test >}} diff --git a/content/docs/standalone/main/configuration/routes.md b/content/docs/standalone/main/configuration/routes.md index 2d5727988..3423994dd 100644 --- a/content/docs/standalone/main/configuration/routes.md +++ b/content/docs/standalone/main/configuration/routes.md @@ -3,8 +3,40 @@ title: Routes weight: 30 description: Match HTTP and TCP traffic on a gateway and forward it to backends. next: /configuration/traffic-management +test: + routes: + - file: ${versionRoot}/configuration/routes.md + path: routes --- +{{< doc-test paths="routes" >}} +# ============================================================================ +# Doc test coverage for this guide (these comments are not rendered on the page) +# ============================================================================ +# WHAT THIS TEST VALIDATES: +# * "HTTP routes": the example config is accepted by agentgateway +# (--validate-only), covering the `gateways`, `protocol: HTTP`, `name`, +# `gateways: [...]`, `hostnames`, `matches.path.pathPrefix`, and +# `backends[].host` / `weight` fields the route table documents. +# * "TCP routes": the `tcpRoutes` example is accepted, covering `protocol: TCP` +# and the simpler TCP route structure. +# * "Example configuration with policies": the route-with-CORS example is +# accepted, covering `policies.cors` on a route and an inline `backends[].mcp` +# backend with a `stdio` target. +# +# WHAT THIS TEST DOES NOT VALIDATE (and why): +# * That traffic is actually matched and forwarded - requires config/traffic the +# page omits; every example points at a placeholder backend +# (`http.example.com:8080`, `postgres.example.com:5432`) that the test cannot +# stand up, so only config acceptance is asserted. +# * The `matches` header, method, and query options, and the "attaches to the +# gateway named `default`" fallback - display-only table rows with no example +# config on this page. Matching is covered by the Request matching guide. +# * The CORS policy's runtime behavior - covered by the CORS guide's own test; +# here the block only proves the policy is accepted on a route. +{{< reuse "agw-docs/snippets/install-agentgateway-binary.md" >}} +{{< /doc-test >}} + {{< gloss "Route" >}}Routes{{< /gloss >}} are the entry points for traffic to your agentgateway. They attach to [gateways]({{< link-hextra path="/configuration/gateways/" >}}) and are used to route traffic to {{< gloss "Backend" >}}backends{{< /gloss >}}. ## Types of routes @@ -36,6 +68,28 @@ routes: weight: 1 ``` +{{< doc-test paths="routes" >}} +cat <<'EOF' > config-http.yaml +# yaml-language-server: $schema=https://agentgateway.dev/schema/config +gateways: + http-proxy: + port: 8080 + protocol: HTTP +routes: +- name: http-backend + gateways: [http-proxy] + hostnames: + - "example.com" + matches: + - path: + pathPrefix: / + backends: + - host: http.example.com:8080 + weight: 1 +EOF +agentgateway -f config-http.yaml --validate-only +{{< /doc-test >}} + HTTP routes support various matching options for incoming requests. For more information, see the [Request matching]({{< link-hextra path="/configuration/traffic-management/matching/" >}}) guide. ### TCP routes @@ -60,6 +114,23 @@ tcpRoutes: weight: 1 ``` +{{< doc-test paths="routes" >}} +cat <<'EOF' > config-tcp.yaml +# yaml-language-server: $schema=https://agentgateway.dev/schema/config +gateways: + postgres-proxy: + port: 5432 + protocol: TCP +tcpRoutes: +- name: postgres-backend + gateways: [postgres-proxy] + backends: + - host: postgres.example.com:5432 + weight: 1 +EOF +agentgateway -f config-tcp.yaml --validate-only +{{< /doc-test >}} + For more information, see [TCP route matching]({{< link-hextra path="/configuration/traffic-management/matching#tcp-routes" >}}). ## Route configuration @@ -115,6 +186,34 @@ routes: args: ["@modelcontextprotocol/server-everything"] ``` +{{< doc-test paths="routes" >}} +cat <<'EOF' > config-policies.yaml +# yaml-language-server: $schema=https://agentgateway.dev/schema/config +gateways: + default: + port: 3000 +routes: +- policies: + cors: + allowOrigins: + - "*" + allowHeaders: + - mcp-protocol-version + - content-type + - cache-control + exposeHeaders: + - "Mcp-Session-Id" + backends: + - mcp: + targets: + - name: everything + stdio: + cmd: npx + args: ["@modelcontextprotocol/server-everything"] +EOF +agentgateway -f config-policies.yaml --validate-only +{{< /doc-test >}} + ## Next steps After you configure routes, you might want to apply policies to them or learn more about traffic management options. diff --git a/content/docs/standalone/main/configuration/security/cors.md b/content/docs/standalone/main/configuration/security/cors.md index 75603ff74..48e3694b1 100644 --- a/content/docs/standalone/main/configuration/security/cors.md +++ b/content/docs/standalone/main/configuration/security/cors.md @@ -2,12 +2,58 @@ title: CORS weight: 11 description: Configure Cross-Origin Resource Sharing policies to control cross-domain requests. +test: + cors: + - file: ${versionRoot}/configuration/security/cors.md + path: cors --- Attaches to: {{< badge content="Route" path="/configuration/routes/">}} {{< reuse "agw-docs/snippets/config-styles-note.md" >}} +{{< doc-test paths="cors" >}} +# ============================================================================ +# Doc test coverage for this guide (these comments are not rendered on the page) +# ============================================================================ +# WHAT THIS TEST VALIDATES: +# * All three example configs (Simplified LLM, Simplified MCP, and +# Routing-based) are accepted by agentgateway (--validate-only), covering +# `allowOrigins`, `allowMethods`, `allowHeaders`, `exposeHeaders`, +# `allowCredentials`, and `maxAge` in both duration (`10m`, `100s`) forms. +# * The "Origin Allowed" branch of the CORS preflight diagram: with the +# Routing-based config loaded, an OPTIONS preflight from +# https://app.example.com returns 200 with access-control-allow-origin, +# -allow-methods, -allow-headers, -allow-credentials, -expose-headers, and +# -max-age set to the configured values (maxAge 100s is emitted as `100`). +# * The "Origin NOT Allowed" branch: an OPTIONS preflight from an origin that +# is not in `allowOrigins` still returns 200 but with no +# access-control-allow-origin header, which is what causes the browser to +# block the response. +# * The actual (non-preflight) cross-origin request: the Routing-based config is +# rerun with its placeholder backend (`api.example.com:443`) swapped for a +# local echo backend, and a GET with an `Origin` header is asserted to reach +# the backend AND come back with the CORS response headers attached - not +# just the preflight the earlier assertion covers. +# * The Simplified (MCP) config at runtime: rerun with a real npx-launched MCP +# server (the same server used by the mcp/connect guides), and an OPTIONS +# preflight against the MCP port asserts the CORS headers the settings list +# documents, including `maxAge: 10m` resolving to a `600`-second header. +# +# WHAT THIS TEST DOES NOT VALIDATE (and why): +# * That a browser enforces the policy - different layer; as the page's own tip +# notes, curl and other HTTP clients ignore CORS headers, so the test can only +# assert the headers agentgateway returns. +# * The Simplified (LLM) config at runtime - external dependency; it needs a +# real OpenAI API key to reach a provider that could return CORS headers on +# an actual completion, so it is only validated as config. +{{< reuse "agw-docs/snippets/install-agentgateway-binary.md" >}} + +# The Simplified (LLM) example reads the API key from the environment. +# --validate-only still resolves env vars, so a placeholder is enough here. +export OPENAI_API_KEY="${OPENAI_API_KEY:-test}" +{{< /doc-test >}} + ## About CORS {{< gloss "CORS (Cross-Origin Resource Sharing)" >}}Cross-origin resource sharing (CORS){{< /gloss >}} is a browser security mechanism which allows a server to control which origins can request and interact with resources that are hosted on a different domain. By default, web browsers only allow requests to resources that are hosted on the same domain as the web page that served the original request. Access to web pages or resources that are hosted on a different domain is restricted to prevent potential security vulnerabilities, such as cross-site request forgery (CRSF). @@ -136,3 +182,228 @@ routes: ``` {{< /tab >}} {{< /tabs >}} + +{{< doc-test paths="cors" >}} +cat <<'EOF' > config-llm.yaml +# yaml-language-server: $schema=https://agentgateway.dev/schema/config +llm: + policies: + cors: + allowOrigins: + - https://chat.example.com + allowMethods: + - POST + - OPTIONS + allowHeaders: + - authorization + - content-type + exposeHeaders: + - x-request-id + allowCredentials: true + maxAge: 10m + models: + - name: "*" + provider: openAI + params: + apiKey: "$OPENAI_API_KEY" +EOF +agentgateway -f config-llm.yaml --validate-only + +cat <<'EOF' > config-mcp.yaml +# yaml-language-server: $schema=https://agentgateway.dev/schema/config +mcp: + port: 3000 + policies: + cors: + allowOrigins: + - https://chat.example.com + allowMethods: + - POST + - OPTIONS + allowHeaders: + - authorization + - content-type + exposeHeaders: + - x-request-id + allowCredentials: true + maxAge: 10m + targets: + - name: everything + stdio: + cmd: npx + args: ["@modelcontextprotocol/server-everything"] +EOF +agentgateway -f config-mcp.yaml --validate-only + +cat <<'EOF' > config.yaml +# yaml-language-server: $schema=https://agentgateway.dev/schema/config +gateways: + default: + port: 3000 +routes: +- backends: + - host: api.example.com:443 + policies: + cors: + allowOrigins: + - https://app.example.com + allowMethods: + - GET + - POST + - OPTIONS + allowHeaders: + - authorization + - content-type + exposeHeaders: + - x-request-id + allowCredentials: true + maxAge: 100s +EOF +agentgateway -f config.yaml --validate-only +{{< /doc-test >}} + +{{< doc-test paths="cors" >}} +agentgateway -f config.yaml & +AGW_PID=$! +trap 'kill $AGW_PID 2>/dev/null || true' EXIT +sleep 3 +{{< /doc-test >}} + +{{< doc-test paths="cors" >}} +YAMLTest -f - <<'EOF' +- name: Preflight from an allowed origin returns the configured CORS headers + retries: 3 + http: + url: "http://localhost:3000" + path: / + method: OPTIONS + headers: + origin: "https://app.example.com" + access-control-request-method: GET + access-control-request-headers: authorization + source: + type: local + expect: + statusCode: 200 + headers: + - name: access-control-allow-origin + comparator: equals + value: "https://app.example.com" + - name: access-control-allow-methods + comparator: contains + value: GET + - name: access-control-allow-headers + comparator: contains + value: authorization + - name: access-control-expose-headers + comparator: contains + value: x-request-id + - name: access-control-allow-credentials + comparator: equals + value: "true" + - name: access-control-max-age + comparator: equals + value: "100" +EOF +{{< /doc-test >}} + +{{< doc-test paths="cors" >}} +# The "Origin NOT Allowed" branch of the diagram: agentgateway still answers the +# preflight, but omits access-control-allow-origin, which is what makes the +# browser block the response. +DISALLOWED_HEADERS=$(curl -s -i -X OPTIONS http://localhost:3000/ \ + -H "Origin: https://not-allowed.example.com" \ + -H "Access-Control-Request-Method: GET") +if grep -qi '^access-control-allow-origin' <<<"$DISALLOWED_HEADERS"; then + echo "FAIL: preflight from a disallowed origin returned access-control-allow-origin" + echo "$DISALLOWED_HEADERS" + exit 1 +fi +echo "✓ Preflight from a disallowed origin returned no access-control-allow-origin header" +{{< /doc-test >}} + +{{< doc-test paths="cors" >}} +# Confirm CORS headers reach an actual (non-preflight) cross-origin request, not +# just the preflight asserted above. Rerun the Routing-based config with its +# placeholder backend (api.example.com:443) swapped for a local echo backend, so +# a GET with an Origin header has something to forward to. +kill $AGW_PID 2>/dev/null || true +wait $AGW_PID 2>/dev/null || true + +cat <<'PYEOF' > backend.py +from http.server import BaseHTTPRequestHandler, HTTPServer + +class Echo(BaseHTTPRequestHandler): + def do_GET(self): + body = b"ok" + self.send_response(200) + self.send_header("content-type", "text/plain") + self.send_header("content-length", str(len(body))) + self.end_headers() + self.wfile.write(body) + + def log_message(self, *args): + pass + +HTTPServer(("127.0.0.1", 8081), Echo).serve_forever() +PYEOF +python3 backend.py & +BACKEND_PID=$! +trap 'kill $AGW_PID $BACKEND_PID 2>/dev/null || true' EXIT +for i in $(seq 1 30); do + curl -sf -o /dev/null http://127.0.0.1:8081/ && break + sleep 1 +done + +sed 's#api.example.com:443#localhost:8081#' config.yaml > config-cors-local.yaml +agentgateway -f config-cors-local.yaml & +AGW_PID=$! +sleep 3 + +RESPONSE=$(curl -s -i http://localhost:3000/ -H "Origin: https://app.example.com") +if ! grep -qi '^access-control-allow-origin: https://app.example.com' <<<"$RESPONSE"; then + echo "FAIL: an actual cross-origin GET did not come back with access-control-allow-origin" + echo "$RESPONSE" + exit 1 +fi +if ! grep -q '^ok$' <<<"$RESPONSE"; then + echo "FAIL: the request was not actually forwarded to the backend" + echo "$RESPONSE" + exit 1 +fi +echo "✓ An actual cross-origin request reached the backend and came back with CORS headers" + +kill $AGW_PID $BACKEND_PID 2>/dev/null || true +wait $AGW_PID $BACKEND_PID 2>/dev/null || true +{{< /doc-test >}} + +{{< doc-test paths="cors" >}} +# Confirm the Simplified (MCP) config's CORS policy works at runtime, using a +# real npx-launched MCP server (the same server the mcp/connect guides use) so +# no external dependency is needed. +agentgateway -f config-mcp.yaml & +AGW_PID=$! +trap 'kill $AGW_PID 2>/dev/null || true' EXIT +for i in $(seq 1 30); do + curl -sf -o /dev/null --max-time 5 http://localhost:15021/healthz/ready && break + sleep 2 +done + +MCP_HEADERS=$(curl -s -i -X OPTIONS http://127.0.0.1:3000/mcp \ + -H "Origin: https://chat.example.com" \ + -H "Access-Control-Request-Method: POST") +if ! grep -qi '^access-control-allow-origin: https://chat.example.com' <<<"$MCP_HEADERS"; then + echo "FAIL: MCP port preflight did not return access-control-allow-origin" + echo "$MCP_HEADERS" + exit 1 +fi +if ! grep -qi '^access-control-max-age: 600' <<<"$MCP_HEADERS"; then + echo "FAIL: MCP port preflight's access-control-max-age was not 600 (maxAge: 10m)" + echo "$MCP_HEADERS" + exit 1 +fi +echo "✓ The Simplified (MCP) CORS policy answers a real preflight against the MCP port" + +kill $AGW_PID 2>/dev/null || true +wait $AGW_PID 2>/dev/null || true +{{< /doc-test >}} diff --git a/content/docs/standalone/main/configuration/security/network-authz.md b/content/docs/standalone/main/configuration/security/network-authz.md index e98aa8e8f..a6560d452 100644 --- a/content/docs/standalone/main/configuration/security/network-authz.md +++ b/content/docs/standalone/main/configuration/security/network-authz.md @@ -2,10 +2,58 @@ title: Network authorization weight: 13 description: Enforce access control at the L4 level using CEL expressions. +test: + network-authz: + - file: ${versionRoot}/configuration/security/network-authz.md + path: network-authz --- Attaches to: {{< badge content="Frontend" path="/configuration/overview/">}} +{{< doc-test paths="network-authz" >}} +# ============================================================================ +# Doc test coverage for this guide (these comments are not rendered on the page) +# ============================================================================ +# WHAT THIS TEST VALIDATES: +# * "Configuration": the example config is accepted by agentgateway +# (--validate-only), covering `frontendPolicies.networkAuthorization.rules` +# with all three rule types (`allow`, `deny`, `require`) and the +# `source.address` / `source.port` CEL variables. +# * "Examples": all three example configs are accepted - the private-range +# allowlist (`cidr(...).containsIP(...)`), the mTLS `source.tls.identity` +# requirement, and the layered L4+L7 config that combines +# `networkAuthorization` with a route-level `authorization` policy. +# * Allowlist semantics from the "Evaluation order" list, rule 6: with the +# Configuration example loaded, a connection from localhost matches no `allow` +# rule, so the connection is rejected at L4 before any HTTP response is sent +# (the client sees a connection reset, not a status code). +# * Evaluation order rule 4 (allow match): a variant of the Configuration +# example with an `allow` rule that matches the test client's own address +# (`127.0.0.1`) lets the connection reach HTTP routing - observed as a `503` +# from the placeholder backend rather than a connection failure, confirming +# network authorization is the thing that let it through. +# +# WHAT THIS TEST DOES NOT VALIDATE (and why): +# * Evaluation order rules for `deny` match and denylist semantics - requires +# spoofing the test client's own source IP/port, which isn't controllable +# from userspace without a second host or network namespace. +# * Evaluation order rule for `require` match/no-match - the test client's +# ephemeral source port is always > 1024, so a `require: source.port > 1024` +# rule always trivially passes; forcing a low source port isn't controllable +# from userspace either. +# * Evaluation order rule 1 (no rules): trivial by definition (no +# `networkAuthorization` config at all behaves like any other page's +# unauthenticated route), so a dedicated example would add no signal beyond +# what every other doc test on this site already demonstrates. +# * `source.tls.identity` and `source.tls.subject_alt_names` at runtime - +# requires config/traffic the page omits; the page shows no TLS listener or +# client certificate setup, so the mTLS example is only validated as config. +# * The route-level `authorization` JWT requirement in the layered example - +# external dependency; enforcing it needs a JWT issuer this page does not set +# up. HTTP authorization is covered by its own guide. +{{< reuse "agw-docs/snippets/install-agentgateway-binary.md" >}} +{{< /doc-test >}} + Network authorization enforces access control at the L4 (transport) level, before HTTP processing. You can enforce policies for non-HTTP traffic such as raw TCP and TLS connections, and layer L4+L7 controls when you combine policies with [HTTP authorization]({{< link-hextra path="/configuration/security/http-authz/" >}}). Network authorization uses [CEL expressions]({{< link-hextra path="/reference/cel/" >}}) evaluated against the connection's source context. @@ -31,6 +79,85 @@ routes: - host: localhost:8080 ``` +{{< doc-test paths="network-authz" >}} +cat <<'EOF' > config.yaml +# yaml-language-server: $schema=https://agentgateway.dev/schema/config +frontendPolicies: + networkAuthorization: + rules: + - allow: 'source.address == "10.0.0.0" || source.address == "10.0.0.1"' + - deny: 'source.address == "192.168.1.100"' + - require: 'source.port > 1024' + +gateways: + default: + port: 3000 +routes: +- backends: + - host: localhost:8080 +EOF +agentgateway -f config.yaml --validate-only +{{< /doc-test >}} + +{{< doc-test paths="network-authz" >}} +# Load the Configuration example and confirm allowlist semantics (evaluation +# order rule 6): the test client connects from localhost, which matches none of +# the `allow` rules, so the connection must be rejected at L4. A rejected L4 +# connection produces a transport error rather than an HTTP status, so this is +# asserted with curl rather than YAMLTest. +agentgateway -f config.yaml & +AGW_PID=$! +trap 'kill $AGW_PID 2>/dev/null || true' EXIT +sleep 3 + +if curl -s -o /dev/null --max-time 5 http://localhost:3000/; then + echo "FAIL: connection from a non-allowlisted source address was not rejected" + exit 1 +fi +echo "✓ Network authorization rejected a connection from a non-allowlisted source address" + +kill $AGW_PID 2>/dev/null || true +wait $AGW_PID 2>/dev/null || true +{{< /doc-test >}} + +{{< doc-test paths="network-authz" >}} +# Evaluation order rule 4 (allow match): the same shape as the Configuration +# example, but with an allow rule that matches the test client's own address +# (127.0.0.1) instead of the page's example addresses. If network authorization +# is what's gating the connection, it should now reach HTTP routing -- observed +# as a 503 from the placeholder backend at localhost:8080, not a connection +# failure. +cat <<'EOF' > config-allow-match.yaml +frontendPolicies: + networkAuthorization: + rules: + - allow: 'source.address == "127.0.0.1"' + - deny: 'source.address == "192.168.1.100"' + - require: 'source.port > 1024' + +gateways: + default: + port: 3000 +routes: +- backends: + - host: localhost:8080 +EOF +agentgateway -f config-allow-match.yaml --validate-only + +agentgateway -f config-allow-match.yaml & +AGW_PID=$! +sleep 3 + +STATUS=$(curl -s -o /dev/null -w '%{http_code}' --max-time 5 http://127.0.0.1:3000/) +kill $AGW_PID 2>/dev/null || true +wait $AGW_PID 2>/dev/null || true +if [ "$STATUS" != "503" ]; then + echo "FAIL: expected a 503 from the placeholder backend once network authorization allowed the connection, got $STATUS" + exit 1 +fi +echo "✓ Network authorization allowed a connection matching an allow rule through to HTTP routing" +{{< /doc-test >}} + ## Rules Network authorization supports the same rule types as HTTP authorization: @@ -71,6 +198,16 @@ frontendPolicies: - allow: 'cidr("10.0.0.0/8").containsIP(source.address) || cidr("172.16.0.0/12").containsIP(source.address) || cidr("192.168.0.0/16").containsIP(source.address)' ``` +{{< doc-test paths="network-authz" >}} +cat <<'EOF' > config-private.yaml +frontendPolicies: + networkAuthorization: + rules: + - allow: 'cidr("10.0.0.0/8").containsIP(source.address) || cidr("172.16.0.0/12").containsIP(source.address) || cidr("192.168.0.0/16").containsIP(source.address)' +EOF +agentgateway -f config-private.yaml --validate-only +{{< /doc-test >}} + ### Require mTLS client identity ```yaml @@ -80,6 +217,16 @@ frontendPolicies: - require: 'source.tls.identity == "spiffe://cluster.local/ns/default/sa/my-service"' ``` +{{< doc-test paths="network-authz" >}} +cat <<'EOF' > config-mtls.yaml +frontendPolicies: + networkAuthorization: + rules: + - require: 'source.tls.identity == "spiffe://cluster.local/ns/default/sa/my-service"' +EOF +agentgateway -f config-mtls.yaml --validate-only +{{< /doc-test >}} + ### Layered L4+L7 controls Combine network authorization with HTTP authorization for defense in depth. @@ -102,4 +249,25 @@ routes: - require: 'jwt.aud == "my-service"' ``` +{{< doc-test paths="network-authz" >}} +cat <<'EOF' > config-layered.yaml +frontendPolicies: + networkAuthorization: + rules: + - allow: 'cidr("10.0.0.0/8").containsIP(source.address)' + +gateways: + default: + port: 3000 +routes: +- backends: + - host: localhost:8080 + policies: + authorization: + rules: + - require: 'jwt.aud == "my-service"' +EOF +agentgateway -f config-layered.yaml --validate-only +{{< /doc-test >}} + In this example, only connections from the `10.0.0.0/8` range are accepted at the network level, and those connections must also present a valid JWT with the correct audience claim. diff --git a/content/docs/standalone/main/configuration/traffic-management/buffer.md b/content/docs/standalone/main/configuration/traffic-management/buffer.md index b2f53e41e..f8f0423cf 100644 --- a/content/docs/standalone/main/configuration/traffic-management/buffer.md +++ b/content/docs/standalone/main/configuration/traffic-management/buffer.md @@ -2,10 +2,41 @@ title: Body buffering weight: 17 description: Buffer request and response bodies before forwarding them. +test: + buffer: + - file: ${versionRoot}/configuration/traffic-management/buffer.md + path: buffer --- Attaches to: {{< badge content="Route" path="/configuration/routes/" >}} +{{< doc-test paths="buffer" >}} +# ============================================================================ +# Doc test coverage for this guide (these comments are not rendered on the page) +# ============================================================================ +# WHAT THIS TEST VALIDATES: +# * "Buffer request and response bodies": the example config is accepted by +# agentgateway (--validate-only), so the `policies.buffer.request.maxBytes` +# and `policies.buffer.response.maxBytes` field names and nesting are correct. +# * The same config serves live traffic: with the policy applied, a GET request +# reaches the backend and returns 200, and a POST request with a body inside +# the `maxBytes` limit is buffered and forwarded to the backend with all of +# its bytes intact (the backend echoes the body back). +# +# WHAT THIS TEST DOES NOT VALIDATE (and why): +# * `failureMode` (`failClosed` / `failOpen`) behavior when a body exceeds +# `maxBytes` - requires config/traffic the page omits; the page documents the +# fields in a table but shows no example that sets `failureMode` or sends an +# oversized body. +# * That bodies are actually accumulated in memory rather than streamed - a +# different layer; the proxy exposes no per-request signal that this page +# documents, so only the end-to-end result is asserted. +# * The `frontendPolicies.http.maxBufferSize` gateway-level limit mentioned in +# the note - display-only reference to a separate setting, with no example on +# this page. +{{< reuse "agw-docs/snippets/install-agentgateway-binary.md" >}} +{{< /doc-test >}} + Use the `policies.buffer` policy to buffer request or response bodies in the proxy before the bodies are forwarded. By default, agentgateway streams bodies. When you configure `policies.buffer`, the proxy accumulates the configured body direction in memory until the body is complete, and then forwards it. > [!NOTE] @@ -43,3 +74,107 @@ routes: response: maxBytes: 262144 ``` + +{{< doc-test paths="buffer" >}} +cat <<'EOF' > config.yaml +# yaml-language-server: $schema=https://agentgateway.dev/schema/config +gateways: + default: + port: 3000 +routes: +- backends: + - host: localhost:8080 + policies: + buffer: + request: + maxBytes: 65536 + response: + maxBytes: 262144 +EOF +agentgateway -f config.yaml --validate-only +{{< /doc-test >}} + +{{< doc-test paths="buffer" >}} +# Stand up an HTTP backend on localhost:8080 so the route in the example config +# has something to forward to. The backend echoes the request body back so the +# test can confirm a buffered POST body arrives intact, then wait for it to +# accept connections. +cat <<'EOF' > backend.py +from http.server import BaseHTTPRequestHandler, HTTPServer + +class Echo(BaseHTTPRequestHandler): + def _reply(self, body=b""): + self.send_response(200) + self.send_header("content-type", "text/plain") + self.send_header("content-length", str(len(body))) + self.end_headers() + self.wfile.write(body) + + def do_GET(self): + self._reply(b"ok") + + def do_POST(self): + length = int(self.headers.get("content-length") or 0) + self._reply(self.rfile.read(length)) + + def log_message(self, *args): + pass + +HTTPServer(("127.0.0.1", 8080), Echo).serve_forever() +EOF +python3 backend.py & +BACKEND_PID=$! +trap 'kill $BACKEND_PID 2>/dev/null' EXIT +# Wait for the echo backend, and confirm the responder is actually this backend +# rather than some other process already holding 8080 -- otherwise the POST +# assertion below fails in a way that looks like a buffering bug. +for i in $(seq 1 30); do + [ "$(curl -sf --max-time 5 -X POST -d probe http://127.0.0.1:8080/ 2>/dev/null)" = "probe" ] && break + sleep 1 +done +if [ "$(curl -sf --max-time 5 -X POST -d probe http://127.0.0.1:8080/ 2>/dev/null)" != "probe" ]; then + echo "FAIL: the echo backend did not come up on 127.0.0.1:8080 (is the port already in use?)" + exit 1 +fi +{{< /doc-test >}} + +{{< doc-test paths="buffer" >}} +agentgateway -f config.yaml & +AGW_PID=$! +trap 'kill $AGW_PID $BACKEND_PID 2>/dev/null' EXIT +sleep 3 +{{< /doc-test >}} + +{{< doc-test paths="buffer" >}} +YAMLTest -f - <<'EOF' +- name: Buffered route forwards a GET request to the backend + retries: 3 + http: + url: "http://localhost:3000" + path: / + method: GET + source: + type: local + expect: + statusCode: 200 +- name: Buffered route forwards a POST request body under maxBytes + http: + url: "http://localhost:3000" + path: / + method: POST + headers: + content-type: text/plain + accept-encoding: identity + body: "buffered request body" + source: + type: local + expect: + statusCode: 200 + headers: + # The backend echoes the request body, so a content-length of 21 confirms + # all 21 bytes of "buffered request body" survived buffering. + - name: content-length + comparator: equals + value: "21" +EOF +{{< /doc-test >}} diff --git a/content/docs/standalone/main/configuration/traffic-management/route-delegation.md b/content/docs/standalone/main/configuration/traffic-management/route-delegation.md index 2dd8e931c..2a2787a06 100644 --- a/content/docs/standalone/main/configuration/traffic-management/route-delegation.md +++ b/content/docs/standalone/main/configuration/traffic-management/route-delegation.md @@ -2,8 +2,106 @@ title: Route delegation weight: 15 description: Delegate routing decisions to route groups for independent team management. +test: + route-delegation: + - file: ${versionRoot}/configuration/traffic-management/route-delegation.md + path: route-delegation --- +{{< doc-test paths="route-delegation" >}} +# ============================================================================ +# Doc test coverage for this guide (these comments are not rendered on the page) +# ============================================================================ +# WHAT THIS TEST VALIDATES: +# * All six example configs are accepted by agentgateway (--validate-only), +# covering `backends[].routeGroup`, top-level `routeGroups[].routes[]`, nested +# route groups, `policies` on both a parent and a child route, a cyclic +# `routeGroup` reference, and a dangling `routeGroup` reference. +# * "Basic delegation", "Header and query matching", and "Multi-level +# delegation" each get two passes: first with the page's own config (so the +# documented `503`/`404` outcomes for a placeholder backend are verified as +# written), then again with the placeholder hosts swapped for a local echo +# backend, asserting a real `200` for every path that should be delegated. +# This proves a delegated request actually reaches a backend, not just that +# it isn't a 404. +# * "Policy inheritance" step 3: a child with no policy of its own receives the +# parent's `x-parent` request header, and the child that defines its own +# `requestHeaderModifier` receives `x-child` and NOT `x-parent`. This confirms +# the documented precedence rule ("the child's policy takes precedence"). +# * "Cyclic delegation": the two-route-group cycle is accepted by +# --validate-only (the cycle is only caught at request time), and a request +# that walks into it gets the documented `500`. +# * "Missing route group": a route referencing a nonexistent `routeGroup` is +# accepted by --validate-only, and a request to it returns `404`. The details +# table documented `500` for this case until this test was added; the +# product actually returns `404` (`error="route not found" reason=NotFound`, +# the same as an unmatched path) - the table was corrected to match. +# +# WHAT THIS TEST DOES NOT VALIDATE (and why): +# * None of the six configs use TLS, so the exact wording of "the connection +# is reset" vs. an HTTP-level error for non-HTTP failure modes elsewhere in +# agentgateway isn't exercised here - out of scope for this page. +{{< reuse "agw-docs/snippets/install-agentgateway-binary.md" >}} + +# Assert the HTTP status of one documented request. Extra args are passed to curl. +assert_status() { + local desc="$1" expected="$2"; shift 2 + local got + got=$(curl -s -o /dev/null -w '%{http_code}' --max-time 10 "$@") + if [ "$got" != "$expected" ]; then + echo "FAIL: $desc -- expected HTTP $expected but got $got" + exit 1 + fi + echo "✓ $desc -> $expected" +} + +start_gateway() { + agentgateway -f "${1:-config.yaml}" & + AGW_PID=$! + sleep 3 +} + +stop_gateway() { + [ -n "${AGW_PID:-}" ] || return 0 + kill "$AGW_PID" 2>/dev/null || true + wait "$AGW_PID" 2>/dev/null || true + AGW_PID="" +} + +trap 'stop_gateway; [ -n "${BACKEND_PID:-}" ] && kill "$BACKEND_PID" 2>/dev/null || true' EXIT + +# Every example on this page points at a placeholder host (team1-foo.example.com +# and friends). Stand up one local echo backend that later sections point +# swapped-host copies of the page's configs at, so a delegated request can be +# observed reaching a real backend (200) instead of only ever seeing the 503 a +# placeholder host produces. Port 8081, not 8080, so it doesn't collide with +# another page's documented config. +cat <<'PYEOF' > backend.py +from http.server import BaseHTTPRequestHandler, HTTPServer +import json + +class Echo(BaseHTTPRequestHandler): + def do_GET(self): + body = json.dumps({k.lower(): v for k, v in self.headers.items()}).encode() + self.send_response(200) + self.send_header("content-type", "application/json") + self.send_header("content-length", str(len(body))) + self.end_headers() + self.wfile.write(body) + + def log_message(self, *args): + pass + +HTTPServer(("127.0.0.1", 8081), Echo).serve_forever() +PYEOF +python3 backend.py & +BACKEND_PID=$! +for i in $(seq 1 30); do + curl -sf -o /dev/null http://127.0.0.1:8081/ && break + sleep 1 +done +{{< /doc-test >}} + Delegate routing decisions from a parent route to a set of child routes defined in a route group. Route delegation lets you break up large routing configurations into smaller, independently managed pieces. ## About @@ -46,8 +144,8 @@ Review more details about how route delegation works in standalone mode. |---|---| | Parent path matcher | A parent route that delegates to a route group must use a `pathPrefix` matcher. | | Child path scope | Child routes must match a path that falls within the parent's prefix. For example, if the parent matches `/api`, a child must match a path starting with `/api`. | -| Cyclic delegation | Agentgateway does not allow cyclic delegation. If route group A delegates to B, and B delegates back to A, agentgateway detects the cycle at runtime and returns an error. | -| Missing route group | If a route references a `routeGroup` that does not exist, the route is replaced with a 500 HTTP response. | +| Cyclic delegation | Agentgateway does not allow cyclic delegation. If route group A delegates to B, and B delegates back to A, agentgateway detects the cycle at runtime and returns a `500` response. See [Error responses](#error-responses). | +| Missing route group | If a route references a `routeGroup` that does not exist, agentgateway returns a `404` response for that route, the same as a path with no match. See [Error responses](#error-responses). | ## Before you begin @@ -61,7 +159,7 @@ In this example, a parent route matches the `/anything/team1` prefix and delegat 1. Create the configuration file. - ```sh + ```sh {paths="route-delegation"} cat > config.yaml <<'EOF' # yaml-language-server: $schema=https://agentgateway.dev/schema/config gateways: @@ -94,6 +192,11 @@ In this example, a parent route matches the `/anything/team1` prefix and delegat EOF ``` + {{< doc-test paths="route-delegation" >}} + # Basic delegation: validate the config written by step 1 + agentgateway -f config.yaml --validate-only + {{< /doc-test >}} + 2. Run the gateway. ```sh @@ -116,6 +219,24 @@ In this example, a parent route matches the `/anything/team1` prefix and delegat curl -i 127.0.0.1:3000/other ``` +{{< doc-test paths="route-delegation" >}} +start_gateway +assert_status "Basic: /anything/team1/foo is delegated to child-foo" 503 127.0.0.1:3000/anything/team1/foo +assert_status "Basic: /anything/team1/bar is delegated to child-bar" 503 127.0.0.1:3000/anything/team1/bar +assert_status "Basic: parent prefix with no matching child" 404 127.0.0.1:3000/anything/team1/other +assert_status "Basic: path outside the parent prefix" 404 127.0.0.1:3000/other +stop_gateway + +# Confirm a delegated request actually reaches a backend: rerun with the +# placeholder hosts swapped for the local echo backend and expect a real 200. +sed 's#team1-foo.example.com:8080#localhost:8081#; s#team1-bar.example.com:8080#localhost:8081#' \ + config.yaml > config-basic-local.yaml +start_gateway config-basic-local.yaml +assert_status "Basic: /anything/team1/foo reaches the backend" 200 127.0.0.1:3000/anything/team1/foo +assert_status "Basic: /anything/team1/bar reaches the backend" 200 127.0.0.1:3000/anything/team1/bar +stop_gateway +{{< /doc-test >}} + ## Header and query matching Parent routes can include header and query parameter matchers that control which requests are delegated. Child routes can independently define their own matchers. A request must satisfy both the parent's and the child's matchers to reach a backend. @@ -127,7 +248,7 @@ In this example, a parent route matches `/anything/team1` only when the `x-team` 1. Create the configuration file. - ```sh + ```sh {paths="route-delegation"} cat > config.yaml <<'EOF' # yaml-language-server: $schema=https://agentgateway.dev/schema/config gateways: @@ -172,6 +293,11 @@ In this example, a parent route matches `/anything/team1` only when the `x-team` EOF ``` + {{< doc-test paths="route-delegation" >}} + # Header and query matching: validate the config written by step 1 + agentgateway -f config.yaml --validate-only + {{< /doc-test >}} + 2. Run the gateway. ```sh @@ -181,7 +307,7 @@ In this example, a parent route matches `/anything/team1` only when the `x-team` 3. Test the routes. ```sh - # child-foo: parent matchers + child's x-role header -> 200 + # child-foo: parent matchers + child's x-role header -> routed to child-foo curl -i "127.0.0.1:3000/anything/team1/foo?env=prod" \ -H "x-team: team1" -H "x-role: admin" @@ -189,7 +315,7 @@ In this example, a parent route matches `/anything/team1` only when the `x-team` curl -i "127.0.0.1:3000/anything/team1/foo?env=prod" \ -H "x-team: team1" - # child-bar: parent matchers, child matches on path only -> 200 + # child-bar: parent matchers, child matches on path only -> routed to child-bar curl -i "127.0.0.1:3000/anything/team1/bar?env=prod" \ -H "x-team: team1" @@ -197,6 +323,34 @@ In this example, a parent route matches `/anything/team1` only when the `x-team` curl -i 127.0.0.1:3000/anything/team1/bar ``` +{{< doc-test paths="route-delegation" >}} +start_gateway +assert_status "Header/query: parent matchers plus the child's x-role is delegated" 503 \ + "127.0.0.1:3000/anything/team1/foo?env=prod" -H "x-team: team1" -H "x-role: admin" +assert_status "Header/query: parent matchers but missing the child's x-role" 404 \ + "127.0.0.1:3000/anything/team1/foo?env=prod" -H "x-team: team1" +assert_status "Header/query: child-bar matches on path only" 503 \ + "127.0.0.1:3000/anything/team1/bar?env=prod" -H "x-team: team1" +assert_status "Header/query: missing the parent's matchers is not delegated" 404 \ + 127.0.0.1:3000/anything/team1/bar +stop_gateway + +# Confirm a delegated request actually reaches a backend: rerun with the +# placeholder hosts swapped for the local echo backend and expect a real 200. +sed 's#team1-foo.example.com:8080#localhost:8081#; s#team1-bar.example.com:8080#localhost:8081#' \ + config.yaml > config-headerquery-local.yaml +start_gateway config-headerquery-local.yaml +assert_status "Header/query: child-foo reaches the backend" 200 \ + "127.0.0.1:3000/anything/team1/foo?env=prod" -H "x-team: team1" -H "x-role: admin" +assert_status "Header/query: child-bar reaches the backend" 200 \ + "127.0.0.1:3000/anything/team1/bar?env=prod" -H "x-team: team1" +stop_gateway +{{< /doc-test >}} + + The backend hosts in these examples are placeholders, so a request that is + routed to a child returns `503` instead of a response from the backend. The + `404` responses are the ones that show a request was not delegated. + ## Multi-level delegation Child routes inside a route group can delegate to other route groups, creating a multi-level delegation hierarchy. Agentgateway detects cycles at runtime and returns an error if a delegation chain loops back to a previously visited route group. @@ -205,7 +359,7 @@ In this example, a parent route delegates `/api` to a route group. One child han 1. Create the configuration file. - ```sh + ```sh {paths="route-delegation"} cat > config.yaml <<'EOF' # yaml-language-server: $schema=https://agentgateway.dev/schema/config gateways: @@ -252,6 +406,11 @@ In this example, a parent route delegates `/api` to a route group. One child han EOF ``` + {{< doc-test paths="route-delegation" >}} + # Multi-level delegation: validate the config written by step 1 + agentgateway -f config.yaml --validate-only + {{< /doc-test >}} + 2. Run the gateway. ```sh @@ -274,6 +433,26 @@ In this example, a parent route delegates `/api` to a route group. One child han curl -i 127.0.0.1:3000/api/orders/other ``` +{{< doc-test paths="route-delegation" >}} +start_gateway +assert_status "Multi-level: /api/users resolves through api-routes" 503 127.0.0.1:3000/api/users +assert_status "Multi-level: /api/orders/list resolves through two route groups" 503 127.0.0.1:3000/api/orders/list +assert_status "Multi-level: /api/orders/detail resolves through two route groups" 503 127.0.0.1:3000/api/orders/detail +assert_status "Multi-level: child-orders prefix with no matching grandchild" 404 127.0.0.1:3000/api/orders/other +stop_gateway + +# Confirm a delegated request actually reaches a backend at every level of the +# chain: rerun with the three placeholder hosts swapped for the local echo +# backend and expect a real 200. +sed 's#users-service.example.com:8080#localhost:8081#; s#orders-list.example.com:8080#localhost:8081#; s#orders-detail.example.com:8080#localhost:8081#' \ + config.yaml > config-multilevel-local.yaml +start_gateway config-multilevel-local.yaml +assert_status "Multi-level: /api/users reaches the backend" 200 127.0.0.1:3000/api/users +assert_status "Multi-level: /api/orders/list reaches the backend through two route groups" 200 127.0.0.1:3000/api/orders/list +assert_status "Multi-level: /api/orders/detail reaches the backend through two route groups" 200 127.0.0.1:3000/api/orders/detail +stop_gateway +{{< /doc-test >}} + ## Policy inheritance Policies defined on a parent route are inherited by child routes in the delegation chain. If a child route defines the same type of policy, the child's policy takes precedence. @@ -282,7 +461,7 @@ In this example, a parent route sets a `requestHeaderModifier` policy that adds 1. Create the configuration file. - ```sh + ```sh {paths="route-delegation"} cat > config.yaml <<'EOF' # yaml-language-server: $schema=https://agentgateway.dev/schema/config gateways: @@ -323,6 +502,11 @@ In this example, a parent route sets a `requestHeaderModifier` policy that adds EOF ``` + {{< doc-test paths="route-delegation" >}} + # Policy inheritance: validate the config written by step 1 + agentgateway -f config.yaml --validate-only + {{< /doc-test >}} + 2. Run the gateway. ```sh @@ -338,3 +522,153 @@ In this example, a parent route sets a `requestHeaderModifier` policy that adds # child-overrides: receives x-child header; parent's requestHeaderModifier is overridden curl -i 127.0.0.1:3000/anything/team1/bar ``` + +{{< doc-test paths="route-delegation" >}} +# Inherited request headers are only observable at the backend, so this assertion +# runs the page's config with the two placeholder hosts swapped for the shared +# local echo backend (started once, at the top of this test) that echoes the +# request headers it received. +sed 's#team1-foo.example.com:8080#localhost:8081#; s#team1-bar.example.com:8080#localhost:8081#' \ + config.yaml > config-policy-local.yaml +start_gateway config-policy-local.yaml + +INHERITS=$(curl -sf --max-time 10 127.0.0.1:3000/anything/team1/foo) +if [ "$(jq -r '."x-parent" // "absent"' <<<"$INHERITS")" != "from-parent" ]; then + echo "FAIL: child-inherits did not receive the parent's x-parent header" + echo "$INHERITS" + exit 1 +fi +echo "✓ Policy inheritance: child-inherits received x-parent from the parent route" + +OVERRIDES=$(curl -sf --max-time 10 127.0.0.1:3000/anything/team1/bar) +if [ "$(jq -r '."x-child" // "absent"' <<<"$OVERRIDES")" != "from-child" ]; then + echo "FAIL: child-overrides did not receive its own x-child header" + echo "$OVERRIDES" + exit 1 +fi +if [ "$(jq -r '."x-parent" // "absent"' <<<"$OVERRIDES")" != "absent" ]; then + echo "FAIL: child-overrides should override the parent policy, but x-parent was still added" + echo "$OVERRIDES" + exit 1 +fi +echo "✓ Policy inheritance: child-overrides received x-child and not x-parent" +stop_gateway +{{< /doc-test >}} + +## Error responses + +Two invalid delegation configurations produce specific error responses, rather than being rejected at validation time. + +### Cyclic delegation + +Agentgateway does not allow cyclic delegation. If route group A delegates to B, and B delegates back to A, agentgateway detects the cycle at runtime and returns a `500` response. The cycle is not caught by `--validate-only`, because static validation does not follow `routeGroup` references. + +1. Create the configuration file. + + ```sh {paths="route-delegation"} + cat > config-cycle.yaml <<'EOF' + # yaml-language-server: $schema=https://agentgateway.dev/schema/config + gateways: + default: + port: 3000 + protocol: HTTP + routes: + - name: parent-a + matches: + - path: + pathPrefix: /a + backends: + - routeGroup: group-a + + routeGroups: + - name: group-a + routes: + - name: to-b + matches: + - path: + pathPrefix: /a + backends: + - routeGroup: group-b + - name: group-b + routes: + - name: to-a + matches: + - path: + pathPrefix: /a + backends: + - routeGroup: group-a + EOF + ``` + + {{< doc-test paths="route-delegation" >}} + # Cyclic delegation: validate the config written by step 1. --validate-only + # succeeds because the cycle is only detected at request time. + agentgateway -f config-cycle.yaml --validate-only + {{< /doc-test >}} + +2. Run the gateway. + + ```sh + agentgateway -f config-cycle.yaml + ``` + +3. Test the route. + + ```sh + # group-a -> group-b -> group-a is a cycle -> 500 + curl -i 127.0.0.1:3000/a + ``` + +{{< doc-test paths="route-delegation" >}} +start_gateway config-cycle.yaml +assert_status "Cyclic delegation is detected at runtime and returns 500" 500 127.0.0.1:3000/a +stop_gateway +{{< /doc-test >}} + +### Missing route group + +If a route references a `routeGroup` that does not exist, agentgateway returns a `404` response for that route, the same as a path with no match. + +1. Create the configuration file. + + ```sh {paths="route-delegation"} + cat > config-missing-group.yaml <<'EOF' + # yaml-language-server: $schema=https://agentgateway.dev/schema/config + gateways: + default: + port: 3000 + protocol: HTTP + routes: + - name: parent-missing + matches: + - path: + pathPrefix: /missing + backends: + - routeGroup: does-not-exist + EOF + ``` + + {{< doc-test paths="route-delegation" >}} + # Missing route group: validate the config written by step 1. --validate-only + # succeeds because the dangling reference is only resolved at request time. + agentgateway -f config-missing-group.yaml --validate-only + {{< /doc-test >}} + +2. Run the gateway. + + ```sh + agentgateway -f config-missing-group.yaml + ``` + +3. Test the route. + + ```sh + # does-not-exist is not a defined routeGroup -> 404 + curl -i 127.0.0.1:3000/missing + ``` + +{{< doc-test paths="route-delegation" >}} +start_gateway config-missing-group.yaml +assert_status "A route referencing a nonexistent route group returns 404" 404 127.0.0.1:3000/missing +stop_gateway +{{< /doc-test >}} diff --git a/content/docs/standalone/main/integrations/llm-providers/anthropic.md b/content/docs/standalone/main/integrations/llm-providers/anthropic.md index 34114093d..ee8b25b13 100644 --- a/content/docs/standalone/main/integrations/llm-providers/anthropic.md +++ b/content/docs/standalone/main/integrations/llm-providers/anthropic.md @@ -2,6 +2,7 @@ title: Anthropic weight: 20 description: Connect agentgateway to Anthropic's Claude models +test: skip --- {{< redirect path="/llm/providers/anthropic/" >}} diff --git a/content/docs/standalone/main/integrations/llm-providers/azure-openai.md b/content/docs/standalone/main/integrations/llm-providers/azure-openai.md index af34eab2c..ef48a6d3c 100644 --- a/content/docs/standalone/main/integrations/llm-providers/azure-openai.md +++ b/content/docs/standalone/main/integrations/llm-providers/azure-openai.md @@ -2,6 +2,7 @@ title: Azure OpenAI weight: 30 description: Connect agentgateway to Azure-hosted OpenAI models +test: skip --- {{< redirect path="/llm/providers/azure/" >}} diff --git a/content/docs/standalone/main/integrations/llm-providers/bedrock.md b/content/docs/standalone/main/integrations/llm-providers/bedrock.md index 78841572f..b93c2e876 100644 --- a/content/docs/standalone/main/integrations/llm-providers/bedrock.md +++ b/content/docs/standalone/main/integrations/llm-providers/bedrock.md @@ -2,6 +2,7 @@ title: Amazon Bedrock weight: 40 description: Connect agentgateway to AWS foundation models via Amazon Bedrock +test: skip --- {{< redirect path="/llm/providers/bedrock/" >}} diff --git a/content/docs/standalone/main/integrations/llm-providers/gemini.md b/content/docs/standalone/main/integrations/llm-providers/gemini.md index 0a9e0fd61..5e332ccc4 100644 --- a/content/docs/standalone/main/integrations/llm-providers/gemini.md +++ b/content/docs/standalone/main/integrations/llm-providers/gemini.md @@ -2,6 +2,7 @@ title: Google Gemini weight: 50 description: Connect agentgateway to Google's Gemini models +test: skip --- {{< redirect path="/llm/providers/gemini/" >}} diff --git a/content/docs/standalone/main/integrations/llm-providers/openai-compatible.md b/content/docs/standalone/main/integrations/llm-providers/openai-compatible.md index 56198a729..877a4ef8b 100644 --- a/content/docs/standalone/main/integrations/llm-providers/openai-compatible.md +++ b/content/docs/standalone/main/integrations/llm-providers/openai-compatible.md @@ -2,6 +2,7 @@ title: OpenAI-Compatible Providers weight: 70 description: Connect agentgateway to any OpenAI-compatible API (xAI, Cohere, Ollama, etc.) +test: skip --- {{< redirect path="/llm/providers/custom/" >}} diff --git a/content/docs/standalone/main/integrations/llm-providers/openai.md b/content/docs/standalone/main/integrations/llm-providers/openai.md index 95df503c2..2b9f14a60 100644 --- a/content/docs/standalone/main/integrations/llm-providers/openai.md +++ b/content/docs/standalone/main/integrations/llm-providers/openai.md @@ -2,6 +2,7 @@ title: OpenAI weight: 10 description: Connect agentgateway to OpenAI's GPT models +test: skip --- {{< redirect path="/llm/providers/openai/" >}} diff --git a/content/docs/standalone/main/integrations/llm-providers/vertex.md b/content/docs/standalone/main/integrations/llm-providers/vertex.md index 2950ce526..9907418ff 100644 --- a/content/docs/standalone/main/integrations/llm-providers/vertex.md +++ b/content/docs/standalone/main/integrations/llm-providers/vertex.md @@ -2,6 +2,7 @@ title: Vertex AI weight: 60 description: Connect agentgateway to Google Cloud's Vertex AI platform +test: skip --- {{< redirect path="/llm/providers/vertex/" >}} diff --git a/content/docs/standalone/main/integrations/llm-providers/xai.md b/content/docs/standalone/main/integrations/llm-providers/xai.md index b7a45f2bc..b816786a0 100644 --- a/content/docs/standalone/main/integrations/llm-providers/xai.md +++ b/content/docs/standalone/main/integrations/llm-providers/xai.md @@ -2,6 +2,7 @@ title: xAI (Grok) weight: 75 description: Connect agentgateway to xAI's Grok models +test: skip --- {{< redirect path="/llm/providers/xai/" >}} diff --git a/content/docs/standalone/main/llm/about.md b/content/docs/standalone/main/llm/about.md index fe16d6fb2..7f047a8a8 100644 --- a/content/docs/standalone/main/llm/about.md +++ b/content/docs/standalone/main/llm/about.md @@ -150,7 +150,7 @@ Use `name: "*"` without setting `params.model` to accept any model name and pass llm: models: - name: "*" - provider: openai + provider: openAI params: apiKey: "$OPENAI_API_KEY" ``` @@ -166,7 +166,7 @@ This is the recommended approach when you want to expose all models from multipl llm: models: - name: "*" - provider: openai + provider: openAI params: apiKey: "$OPENAI_API_KEY" transformation: diff --git a/content/docs/standalone/main/llm/configuration-modes.md b/content/docs/standalone/main/llm/configuration-modes.md index 6ccc3ac4c..61f31a629 100644 --- a/content/docs/standalone/main/llm/configuration-modes.md +++ b/content/docs/standalone/main/llm/configuration-modes.md @@ -98,6 +98,11 @@ llm: To set the port and TLS settings for LLM traffic, define a gateway and attach the `llm` section to it. When you omit the `gateways` field, the `llm` section attaches to the gateway named `default`. The `mcp` and `ui` sections attach the same way, so all three can share one port. +When your configuration file defines no gateway at all, such as the earlier basic example, the implied `default` gateway serves LLM traffic on port `4000` and MCP traffic on port `3000`. Requests use the OpenAI-compatible paths, such as `http://localhost:4000/v1/chat/completions`. + +> [!NOTE] +> The `llm.port`, `llm.tls`, and `mcp.port` fields are deprecated in favor of gateways. They still work, and setting them overrides these defaults. + Use the gateway's `tls` field to serve LLM traffic over TLS. - Most deployments only need `cert` and `key`. - Use `root` for a custom trust bundle or mTLS. @@ -125,9 +130,6 @@ mcp: args: ["@modelcontextprotocol/server-everything"] ``` -> [!NOTE] -> The `llm.port`, `llm.tls`, and `mcp.port` fields are deprecated in favor of gateways. They still work, and when you set them without a gateway, LLM traffic defaults to port `4000` and MCP traffic to port `3000`. - For more MCP listener context, see [MCP overview]({{< link-hextra path="/mcp/" >}}). ## Routing-based configuration diff --git a/content/docs/standalone/main/llm/prompt-guards/regex.md b/content/docs/standalone/main/llm/prompt-guards/regex.md index 106a4cb6e..2b23a5454 100644 --- a/content/docs/standalone/main/llm/prompt-guards/regex.md +++ b/content/docs/standalone/main/llm/prompt-guards/regex.md @@ -2,10 +2,59 @@ title: Regex filters weight: 10 description: Match and redact prompt content with custom regex patterns or agentgateway's built-in PII detectors. +test: + regex: + - file: ${versionRoot}/llm/prompt-guards/regex.md + path: regex --- Use custom regex patterns and built-in PII detectors to filter LLM requests and responses. +{{< doc-test paths="regex" >}} +# ============================================================================ +# Doc test coverage for this guide (these comments are not rendered on the page) +# ============================================================================ +# WHAT THIS TEST VALIDATES: +# * "Custom regex patterns": the credential-matching example config is accepted +# by agentgateway (--validate-only), covering `guardrails.request[].regex` +# with `action: reject`, `rules[].pattern`, and a `rejection` block that sets +# a status, headers, and body. +# * "PII detection" step 1: the config with both a custom-pattern rule and a +# `builtin: email` rule is accepted. +# * "PII detection" step 4: a request containing the SSN keyword is rejected +# with the documented status (400) and the exact documented error body +# (`content_policy_violation`). The `Social Security` pattern from the same +# rule is checked too, which the page describes but does not demonstrate. +# * "PII detection" step 5: a request containing an email address is rejected by +# the built-in `email` pattern with the documented `pii_detected` body, +# confirming the built-in patterns table is wired up and that the second +# guardrail is evaluated independently of the first. +# * "PII detection" step 3, partially: a prompt that matches no rule is NOT +# blocked by the guard. The test asserts the response is not a guard rejection +# rather than asserting success, so it holds whether or not a real API key is +# present. +# * "Mask PII in responses" step 1: the `action: mask` config with +# `builtin: phoneNumber` is accepted. +# +# WHAT THIS TEST DOES NOT VALIDATE (and why): +# * The successful completion in "PII detection" step 3 and its example output - +# external dependency; a real response needs a live OpenAI key and bills a +# completion. Only that the guard does not block the request is asserted. +# * "Mask PII in responses" steps 2-3, including the `` +# replacement - external dependency; masking operates on a real LLM response +# body, so there is nothing to redact without a live provider call. The config +# is validated but the mask behavior is not. +# * The other built-in patterns (`phoneNumber`, `ssn`, `creditCard`, `caSin`) as +# request filters - display-only table rows; only `email` appears in a runnable +# example on this page. +{{< reuse "agw-docs/snippets/install-agentgateway-binary.md" >}} + +# The example configs read the API key from the environment. Guard rejections +# happen before any upstream call, so a placeholder is enough for the assertions +# below; CI supplies a real key when one is available. +export OPENAI_API_KEY="${OPENAI_API_KEY:-test}" +{{< /doc-test >}} + ## About regex prompt templating Regex-based prompt guards let you inspect LLM requests and responses against custom regex patterns or built-in PII detectors. Use the `reject` action to block requests that match a pattern, or the `mask` action to redact sensitive data in responses before they reach the client. @@ -57,6 +106,40 @@ llm: } ``` +{{< doc-test paths="regex" >}} +cat <<'EOF' > config-custom.yaml +llm: + models: + - name: "*" + provider: openAI + params: + model: gpt-4o-mini + apiKey: "$OPENAI_API_KEY" + guardrails: + request: + - regex: + action: reject + rules: + - pattern: "password[=:]\\s*\\S+" + - pattern: "api[_-]?key[=:]\\s*\\S+" + - pattern: "secret[=:]\\s*\\S+" + rejection: + status: 400 + headers: + set: + content-type: "application/json" + body: | + { + "error": { + "message": "Request contains credentials", + "type": "invalid_request_error", + "code": "credentials_detected" + } + } +EOF +agentgateway -f config-custom.yaml --validate-only +{{< /doc-test >}} + ## Before you begin {{< reuse "agw-docs/snippets/prereq-agentgateway.md" >}} @@ -66,7 +149,7 @@ llm: The following example rejects requests that contain PII data, such as Social Security Numbers (using a custom keyword pattern) or email addresses (using the built-in `email` pattern). When a request is blocked, agentgateway returns a custom error response. 1. Create a configuration file with regex prompt guard policies. - ```yaml + ```yaml {paths="regex"} cat <<'EOF' > config.yaml # yaml-language-server: $schema=https://agentgateway.dev/schema/config llm: @@ -195,6 +278,108 @@ The following example rejects requests that contain PII data, such as Social Sec } ``` +{{< doc-test paths="regex" >}} +# Validate the config written by step 1, then run it in the background so the +# step 4 and step 5 requests can be asserted. The visible "Start the agentgateway" +# block is untagged because it runs in the foreground. +agentgateway -f config.yaml --validate-only + +agentgateway -f config.yaml & +AGW_PID=$! +trap 'kill $AGW_PID 2>/dev/null' EXIT +sleep 3 +{{< /doc-test >}} + +{{< doc-test paths="regex" >}} +YAMLTest -f - <<'EOF' +# Guard rejections are produced by agentgateway before the request reaches the +# provider, so these assertions hold with a placeholder API key. +- name: Step 4 - a request containing the SSN keyword is rejected + retries: 3 + http: + url: "http://localhost:4000" + path: /v1/chat/completions + method: POST + headers: + content-type: application/json + accept-encoding: identity + body: | + {"model":"gpt-4o-mini","messages":[{"role":"user","content":"My SSN is 123-45-6789"}]} + source: + type: local + expect: + statusCode: 400 + headers: + - name: content-type + comparator: contains + value: application/json + bodyJsonPath: + - path: "$.error.code" + comparator: equals + value: content_policy_violation + - path: "$.error.message" + comparator: equals + value: "Request rejected: Content contains sensitive information" + - path: "$.error.type" + comparator: equals + value: invalid_request_error +- name: Step 4 rule - the Social Security pattern in the same rule also rejects + http: + url: "http://localhost:4000" + path: /v1/chat/completions + method: POST + headers: + content-type: application/json + accept-encoding: identity + body: | + {"model":"gpt-4o-mini","messages":[{"role":"user","content":"my Social Security number"}]} + source: + type: local + expect: + statusCode: 400 + bodyJsonPath: + - path: "$.error.code" + comparator: equals + value: content_policy_violation +- name: Step 5 - a request containing an email is rejected by the builtin pattern + http: + url: "http://localhost:4000" + path: /v1/chat/completions + method: POST + headers: + content-type: application/json + accept-encoding: identity + body: | + {"model":"gpt-4o-mini","messages":[{"role":"user","content":"Contact me at test@example.com"}]} + source: + type: local + expect: + statusCode: 400 + bodyJsonPath: + - path: "$.error.code" + comparator: equals + value: pii_detected + - path: "$.error.message" + comparator: equals + value: "Request blocked: Contains email address" +EOF +{{< /doc-test >}} + +{{< doc-test paths="regex" >}} +# Step 3: confirm a prompt that matches no rule is not blocked by the guard. The +# assertion is negative rather than a 200 check, because without a real API key the +# upstream returns an auth error -- either way the guard must not have rejected it. +CLEAN=$(curl -s --max-time 15 http://localhost:4000/v1/chat/completions \ + -H "Content-Type: application/json" \ + -d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"Hello, how are you?"}]}') +if grep -qE 'content_policy_violation|pii_detected' <<<"$CLEAN"; then + echo "FAIL: a prompt matching no regex rule was blocked by the prompt guard" + echo "$CLEAN" + exit 1 +fi +echo "✓ A prompt matching no regex rule was not blocked by the prompt guard" +{{< /doc-test >}} + ## Mask PII in responses You can also filter LLM responses to redact sensitive data before it reaches the client. When a match is found, agentgateway replaces built-in pattern matches with `` (for example, ``) and custom pattern matches with ``. The following example masks credit card numbers in responses. @@ -256,3 +441,25 @@ You can also filter LLM responses to redact sensitive data before it reaches the "system_fingerprint":"fp_a1ddba3226"}% ``` + +{{< doc-test paths="regex" >}} +# The mask config is written to its own file so it does not overwrite the config.yaml +# that the running gateway (and the assertions above) depend on. +cat <<'EOF' > config-mask.yaml +# yaml-language-server: $schema=https://agentgateway.dev/schema/config +llm: + models: + - name: "*" + provider: openAI + params: + model: gpt-4o-mini + apiKey: "$OPENAI_API_KEY" + guardrails: + response: + - regex: + action: mask + rules: + - builtin: phoneNumber +EOF +agentgateway -f config-mask.yaml --validate-only +{{< /doc-test >}} diff --git a/content/docs/standalone/main/llm/providers/anthropic.md b/content/docs/standalone/main/llm/providers/anthropic.md index 427b428b6..757f9c94a 100644 --- a/content/docs/standalone/main/llm/providers/anthropic.md +++ b/content/docs/standalone/main/llm/providers/anthropic.md @@ -3,10 +3,43 @@ title: Anthropic weight: 15 icon: /integrations/providers/bw/anthropic.svg description: Route agentgateway LLM traffic to Anthropic's Claude models. +test: + anthropic: + - file: ${versionRoot}/llm/providers/anthropic.md + path: anthropic --- Configure Anthropic (Claude models) as an LLM provider in agentgateway. +{{< doc-test paths="anthropic" >}} +# ============================================================================ +# Doc test coverage for this guide (these comments are not rendered on the page) +# ============================================================================ +# WHAT THIS TEST VALIDATES: +# * "Configuration": the API key example config is accepted by agentgateway +# (--validate-only), so `provider: anthropic` is recognized. +# * "Use Claude Platform on AWS", both tabs: the API-key config (with +# `requestHeaders.set` and a `params.baseUrl` override) and the AWS SigV4 +# config (with `params.awsRegion` and `auth.aws.serviceName`) are both +# accepted. +# * With the base config loaded, agentgateway serves the wildcard model and +# resolves it to the `anthropic` provider. +# +# WHAT THIS TEST DOES NOT VALIDATE (and why): +# * "Example request", "Token counting", "Extended thinking and reasoning", and +# "Structured outputs" - external dependency; each needs a real Anthropic API +# key and bills live completions. Their example responses are display-only. +# * Claude Platform on AWS at runtime - external dependency; reaching +# aws-external-anthropic needs real AWS credentials and an Anthropic +# workspace. +# * The `thinking` and `output_config` field tables - display-only table rows +# describing request bodies, with no runnable config on this page. +{{< reuse "agw-docs/snippets/install-agentgateway-binary.md" >}} + +export ANTHROPIC_API_KEY="${ANTHROPIC_API_KEY:-test}" +export ANTHROPIC_AWS_API_KEY="${ANTHROPIC_AWS_API_KEY:-test}" +{{< /doc-test >}} + ## Configuration For the common API key case, use the following config. Use the AWS SigV4 section later in the page only when you need Claude Platform on AWS or custom signing behavior. @@ -24,6 +57,20 @@ llm: apiKey: "$ANTHROPIC_API_KEY" ``` +{{< doc-test paths="anthropic" >}} +cat <<'EOF' > config.yaml +# yaml-language-server: $schema=https://agentgateway.dev/schema/config + +llm: + models: + - name: "*" + provider: anthropic + params: + apiKey: "$ANTHROPIC_API_KEY" +EOF +agentgateway -f config.yaml --validate-only +{{< /doc-test >}} + {{< reuse "agw-docs/snippets/review-configuration.md" >}} | Setting | Description | @@ -257,6 +304,26 @@ llm: baseUrl: https://aws-external-anthropic.us-west-2.api.aws/v1 ``` +{{< doc-test paths="anthropic" >}} +cat <<'EOF' > config-claude-platform-apikey.yaml +# yaml-language-server: $schema=https://agentgateway.dev/schema/config + +llm: + models: + - name: "*" + provider: anthropic + requestHeaders: + set: + # Replace with your workspace ID + anthropic-workspace-id: wrkspc_XXXXX + params: + apiKey: $ANTHROPIC_AWS_API_KEY + # Replace with your region + baseUrl: https://aws-external-anthropic.us-west-2.api.aws/v1 +EOF +agentgateway -f config-claude-platform-apikey.yaml --validate-only +{{< /doc-test >}} + | Setting | Description | |---------------------------------------------|-------------| | `requestHeaders.set.anthropic-workspace-id` | The Anthropic workspace ID that scopes the request. Replace `wrkspc_XXXXX` with your workspace ID. | @@ -287,6 +354,27 @@ llm: serviceName: aws-external-anthropic ``` +{{< doc-test paths="anthropic" >}} +cat <<'EOF' > config-claude-platform-sigv4.yaml +# yaml-language-server: $schema=https://agentgateway.dev/schema/config + +llm: + models: + - name: "claude-platform/*" + provider: anthropic + requestHeaders: + set: + anthropic-workspace-id: wrkspc_XXXXX + params: + awsRegion: us-west-2 + baseUrl: https://aws-external-anthropic.us-west-2.api.aws/v1 + auth: + aws: + serviceName: aws-external-anthropic +EOF +agentgateway -f config-claude-platform-sigv4.yaml --validate-only +{{< /doc-test >}} + | Setting | Description | |---------|-------------| | `name` | Matches model names that start with `claude-platform/`, so you can route Claude Platform traffic alongside other Anthropic models. | @@ -306,3 +394,29 @@ For setup instructions, see [Use Claude models on Azure AI Foundry]({{< link-hex ## Connect to Claude Code To route Claude Code CLI traffic through agentgateway, see the [Claude Code integration guide]({{< link-hextra path="/integrations/llm-clients/claude-code" >}}). + +{{< doc-test paths="anthropic" >}} +# Confirm the base API key config serves the wildcard model and resolves it to +# the anthropic provider. +agentgateway -f config.yaml & +AGW_PID=$! +trap 'kill $AGW_PID 2>/dev/null' EXIT +sleep 3 + +SERVED=$(curl -sf --max-time 10 http://localhost:4000/v1/models | jq -r '[.data[].id] | index("*") // "missing"') +if [ "$SERVED" = "missing" ]; then + echo "FAIL: the wildcard model from the example config is not served" + exit 1 +fi +PROVIDER=$(curl -sf --max-time 10 http://localhost:15000/config_dump | jq -r ' + [ .backends[].backend.ai + | select(. != null) + | .target.providers[].active[].endpoint + | .provider | keys[0] + ] | first') +if [ "$PROVIDER" != "anthropic" ]; then + echo "FAIL: expected provider anthropic but agentgateway resolved $PROVIDER" + exit 1 +fi +echo "✓ The wildcard model is served and resolves to the anthropic provider" +{{< /doc-test >}} diff --git a/content/docs/standalone/main/llm/providers/azure.md b/content/docs/standalone/main/llm/providers/azure.md index 4e1b3a023..d19d85a09 100644 --- a/content/docs/standalone/main/llm/providers/azure.md +++ b/content/docs/standalone/main/llm/providers/azure.md @@ -3,10 +3,48 @@ title: Azure weight: 15 icon: /integrations/providers/bw/azure.svg description: Route agentgateway LLM traffic to models hosted on Microsoft Azure AI. +test: + azure: + - file: ${versionRoot}/llm/providers/azure.md + path: azure --- Configure Microsoft Azure AI as an LLM provider in agentgateway. +{{< doc-test paths="azure" >}} +# ============================================================================ +# Doc test coverage for this guide (these comments are not rendered on the page) +# ============================================================================ +# WHAT THIS TEST VALIDATES: +# * "Configuration", all three tabs: the Foundry implicit-auth, Foundry API-key +# (`auth.key.location.header`), and Azure OpenAI configs are accepted by +# agentgateway (--validate-only), covering `params.azureResourceName`, +# `params.azureResourceType`, and `params.azureProjectName`. +# * "Advanced configuration", all six tabs: the routing-based configs for +# implicit auth, client secret (Foundry and Azure OpenAI), system-assigned and +# user-assigned managed identity, and workload identity are all accepted, +# covering every `policies.backendAuth.azure.explicitConfig` variant the page +# documents. +# * "Use Claude models on Azure AI Foundry": the routing-based Claude config is +# accepted. This example was missing its `gateways` and `routes` keys until +# this test was added, so it could not have been run as written. +# * With the Foundry implicit-auth config loaded, agentgateway serves the +# wildcard model and resolves it to the `azure` provider. +# +# WHAT THIS TEST DOES NOT VALIDATE (and why): +# * Any authentication method at runtime - external dependency; each needs a +# real Azure tenant, resource, and identity (Entra ID, service principal, +# managed identity, or workload identity), none of which the test can stand +# up. Only that agentgateway accepts each config shape is asserted. +# * The verification curl at the end of the Claude Foundry section - external +# dependency, as above. +# * `params.azureApiVersion` - display-only table row; no example on this page +# sets it. +{{< reuse "agw-docs/snippets/install-agentgateway-binary.md" >}} + +export AZURE_API_KEY="${AZURE_API_KEY:-test}" +{{< /doc-test >}} + ## Authentication Before you can use Azure as an LLM provider, you must authenticate by using one of the standard [Azure authentication methods](https://learn.microsoft.com/en-us/azure/ai-services/authentication). In standalone mode, this authentication is configured with `llm.models[]` fields (for example, `params.apiKey` or `auth.azure`). In routing-based configurations, use `policies.backendAuth.azure`. @@ -36,6 +74,21 @@ llm: azureProjectName: "your-project-name" ``` +{{< doc-test paths="azure" >}} +cat <<'EOF' > config-foundry-implicit.yaml +# yaml-language-server: $schema=https://agentgateway.dev/schema/config +llm: + models: + - name: "*" + provider: azure + params: + azureResourceName: "your-resource-name" + azureResourceType: foundry + azureProjectName: "your-project-name" +EOF +agentgateway -f config-foundry-implicit.yaml --validate-only +{{< /doc-test >}} + {{% /tab %}} {{% tab name="Foundry (API key)" %}} @@ -57,6 +110,27 @@ llm: azureProjectName: "your-project-name" ``` +{{< doc-test paths="azure" >}} +cat <<'EOF' > config-foundry-apikey.yaml +# yaml-language-server: $schema=https://agentgateway.dev/schema/config +llm: + models: + - name: "gpt-4.1" + provider: azure + auth: + key: + value: "$AZURE_API_KEY" + location: + header: + name: api-key + params: + azureResourceName: "your-resource-name" + azureResourceType: foundry + azureProjectName: "your-project-name" +EOF +agentgateway -f config-foundry-apikey.yaml --validate-only +{{< /doc-test >}} + {{% /tab %}} {{% tab name="Azure OpenAI (implicit auth)" %}} @@ -71,6 +145,20 @@ llm: azureResourceType: openAI ``` +{{< doc-test paths="azure" >}} +cat <<'EOF' > config-azure-openai.yaml +# yaml-language-server: $schema=https://agentgateway.dev/schema/config +llm: + models: + - name: "gpt-4.1" + provider: azure + params: + azureResourceName: "your-resource-name" + azureResourceType: openAI +EOF +agentgateway -f config-azure-openai.yaml --validate-only +{{< /doc-test >}} + {{% /tab %}} {{< /tabs >}} @@ -116,6 +204,29 @@ routes: model: gpt-4.1 ``` +{{< doc-test paths="azure" >}} +cat <<'EOF' > config-adv-foundry-implicit.yaml +# yaml-language-server: $schema=https://agentgateway.dev/schema/config +gateways: + default: + port: 3000 +routes: +- matches: + - path: + pathPrefix: /azure + backends: + - ai: + name: azure + provider: + azure: + resourceName: "your-resource-name" + projectName: "your-project-name" + resourceType: foundry + model: gpt-4.1 +EOF +agentgateway -f config-adv-foundry-implicit.yaml --validate-only +{{< /doc-test >}} + {{< reuse "agw-docs/snippets/review-configuration.md" >}} {{< reuse-append "agw-docs/snippets/provider-azure-base-configuration.md" >}} | `backendAuth.azure.implicit` | Use implicit authentication via `DefaultAzureCredential`, which automatically detects credentials from the environment. | @@ -153,6 +264,37 @@ routes: model: gpt-4.1 ``` +{{< doc-test paths="azure" >}} +cat <<'EOF' > config-adv-foundry-client-secret.yaml +# yaml-language-server: $schema=https://agentgateway.dev/schema/config +gateways: + default: + port: 3000 +routes: +- matches: + - path: + pathPrefix: /azure + policies: + backendAuth: + azure: + explicitConfig: + clientSecret: + tenant_id: "" + client_id: "" + client_secret: "" + backends: + - ai: + name: azure + provider: + azure: + resourceName: "your-resource-name" + projectName: "your-project-name" + resourceType: foundry + model: gpt-4.1 +EOF +agentgateway -f config-adv-foundry-client-secret.yaml --validate-only +{{< /doc-test >}} + {{< reuse "agw-docs/snippets/review-configuration.md" >}} {{< reuse-append "agw-docs/snippets/provider-azure-base-configuration.md" >}} | `backendAuth.azure.explicitConfig.clientSecret` | Use Azure service principal authentication with tenant ID, client ID, and client secret. | @@ -185,6 +327,33 @@ routes: client_secret: "" ``` +{{< doc-test paths="azure" >}} +cat <<'EOF' > config-adv-client-secret.yaml +# yaml-language-server: $schema=https://agentgateway.dev/schema/config +gateways: + default: + port: 3000 +routes: +- backends: + - ai: + name: azure + provider: + azure: + resourceName: "your-resource-name" + resourceType: openAI + model: gpt-4.1 + policies: + backendAuth: + azure: + explicitConfig: + clientSecret: + tenant_id: "" + client_id: "" + client_secret: "" +EOF +agentgateway -f config-adv-client-secret.yaml --validate-only +{{< /doc-test >}} + {{< reuse "agw-docs/snippets/review-configuration.md" >}} {{< reuse-append "agw-docs/snippets/provider-azure-base-configuration.md" >}} | `backendAuth.azure.explicitConfig.clientSecret` | Use Azure service principal authentication with tenant ID, client ID, and client secret. | @@ -221,6 +390,30 @@ routes: managedIdentity: {} ``` +{{< doc-test paths="azure" >}} +cat <<'EOF' > config-adv-system-managed-identity.yaml +# yaml-language-server: $schema=https://agentgateway.dev/schema/config +gateways: + default: + port: 3000 +routes: +- backends: + - ai: + name: azure + provider: + azure: + resourceName: "your-resource-name" + resourceType: openAI + model: gpt-4.1 + policies: + backendAuth: + azure: + explicitConfig: + managedIdentity: {} +EOF +agentgateway -f config-adv-system-managed-identity.yaml --validate-only +{{< /doc-test >}} + {{< reuse "agw-docs/snippets/review-configuration.md" >}} {{< reuse-append "agw-docs/snippets/provider-azure-base-configuration.md" >}} | `backendAuth.azure.explicitConfig.managedIdentity` | Use Azure managed identity. Leave empty for system-assigned, or specify `userAssignedIdentity` with `clientId`, `objectId`, or `resourceId`. | @@ -263,6 +456,35 @@ routes: # resourceId: "/subscriptions/.../resourceGroups/.../providers/Microsoft.ManagedIdentity/userAssignedIdentities/..." ``` +{{< doc-test paths="azure" >}} +cat <<'EOF' > config-adv-user-managed-identity.yaml +# yaml-language-server: $schema=https://agentgateway.dev/schema/config +gateways: + default: + port: 3000 +routes: +- backends: + - ai: + name: azure + provider: + azure: + resourceName: "your-resource-name" + resourceType: openAI + model: gpt-4.1 + policies: + backendAuth: + azure: + explicitConfig: + managedIdentity: + userAssignedIdentity: + clientId: "" + # OR use objectId or resourceId instead + # objectId: "your-managed-identity-object-id" + # resourceId: "/subscriptions/.../resourceGroups/.../providers/Microsoft.ManagedIdentity/userAssignedIdentities/..." +EOF +agentgateway -f config-adv-user-managed-identity.yaml --validate-only +{{< /doc-test >}} + {{< reuse "agw-docs/snippets/review-configuration.md" >}} {{< reuse-append "agw-docs/snippets/provider-azure-base-configuration.md" >}} | `backendAuth.azure.explicitConfig.managedIdentity` | Use Azure managed identity. Leave empty for system-assigned, or specify `userAssignedIdentity` with `clientId`, `objectId`, or `resourceId`. | @@ -299,6 +521,31 @@ routes: backendTLS: {} ``` +{{< doc-test paths="azure" >}} +cat <<'EOF' > config-adv-workload-identity.yaml +# yaml-language-server: $schema=https://agentgateway.dev/schema/config +gateways: + default: + port: 3000 +routes: +- backends: + - ai: + name: azure + provider: + azure: + resourceName: "your-resource-name" + resourceType: openAI + model: gpt-4.1 + policies: + backendAuth: + azure: + explicitConfig: + workloadIdentity: {} + backendTLS: {} +EOF +agentgateway -f config-adv-workload-identity.yaml --validate-only +{{< /doc-test >}} + {{< reuse "agw-docs/snippets/review-configuration.md" >}} {{< reuse-append "agw-docs/snippets/provider-azure-base-configuration.md" >}} | `backendAuth.azure.explicitConfig.workloadIdentity` | Use Azure workload identity for Kubernetes environments. | @@ -318,6 +565,10 @@ routes: ```yaml # yaml-language-server: $schema=https://agentgateway.dev/schema/config +gateways: + default: + port: 3000 +routes: - name: azure matches: - path: @@ -330,13 +581,41 @@ routes: resourceName: your-foundry-resource projectName: your-project-name resourceType: foundry - model: claude-sonnet-4-6 + model: claude-sonnet-4-6 policies: backendAuth: key: value: your-api-key ``` +{{< doc-test paths="azure" >}} +cat <<'EOF' > config-claude-foundry.yaml +# yaml-language-server: $schema=https://agentgateway.dev/schema/config +gateways: + default: + port: 3000 +routes: +- name: azure + matches: + - path: + pathPrefix: /azure-anthropic #prefix example + backends: + - ai: + name: azure + provider: + azure: + resourceName: your-foundry-resource + projectName: your-project-name + resourceType: foundry + model: claude-sonnet-4-6 + policies: + backendAuth: + key: + value: your-api-key +EOF +agentgateway -f config-claude-foundry.yaml --validate-only +{{< /doc-test >}} + {{< reuse "agw-docs/snippets/review-table.md" >}} | Setting | Description | @@ -358,3 +637,29 @@ curl -X POST http://localhost:4000/azure-anthropic \ "messages": [{"role": "user", "content": "Hello!"}] }' ``` + +{{< doc-test paths="azure" >}} +# Confirm the Foundry implicit-auth config serves the wildcard model and resolves +# it to the azure provider. +agentgateway -f config-foundry-implicit.yaml & +AGW_PID=$! +trap 'kill $AGW_PID 2>/dev/null' EXIT +sleep 3 + +SERVED=$(curl -sf --max-time 10 http://localhost:4000/v1/models | jq -r '[.data[].id] | index("*") // "missing"') +if [ "$SERVED" = "missing" ]; then + echo "FAIL: the wildcard model from the example config is not served" + exit 1 +fi +PROVIDER=$(curl -sf --max-time 10 http://localhost:15000/config_dump | jq -r ' + [ .backends[].backend.ai + | select(. != null) + | .target.providers[].active[].endpoint + | .provider | keys[0] + ] | first') +if [ "$PROVIDER" != "azure" ]; then + echo "FAIL: expected provider azure but agentgateway resolved $PROVIDER" + exit 1 +fi +echo "✓ The wildcard model is served and resolves to the azure provider" +{{< /doc-test >}} diff --git a/content/docs/standalone/main/llm/providers/baseten.md b/content/docs/standalone/main/llm/providers/baseten.md index e69927e27..2c55a06a0 100644 --- a/content/docs/standalone/main/llm/providers/baseten.md +++ b/content/docs/standalone/main/llm/providers/baseten.md @@ -3,10 +3,40 @@ title: Baseten weight: 20 icon: /integrations/providers/bw/baseten.svg description: Route agentgateway LLM traffic to models hosted on Baseten. +test: + baseten: + - file: ${versionRoot}/llm/providers/baseten.md + path: baseten --- Configure Baseten as an LLM provider in agentgateway. +{{< doc-test paths="baseten" >}} +# ============================================================================ +# Doc test coverage for this guide (these comments are not rendered on the page) +# ============================================================================ +# WHAT THIS TEST VALIDATES: +# * "Configuration": the example config is accepted by agentgateway +# (--validate-only), so `provider: baseten` is a recognized provider and the +# `name` / `params.apiKey` fields are correct. +# * The `params.baseUrl` row of the settings table: with the config loaded, +# agentgateway resolves this provider's upstream to the documented default +# (https://inference.baseten.co/v1), so the table cannot drift from the product without +# this test failing. +# +# WHAT THIS TEST DOES NOT VALIDATE (and why): +# * "Example request" - external dependency; sending the curl request needs a +# real Baseten API key and bills a live completion, so the test uses a +# placeholder key and asserts on resolved config instead. +# * `params.model` - display-only table row; the example config omits it, so +# there is nothing on this page to run. +{{< reuse "agw-docs/snippets/install-agentgateway-binary.md" >}} + +# The example config reads the API key from the environment. --validate-only and +# the config dump still resolve env vars, so a placeholder is enough here. +export BASETEN_API_KEY="${BASETEN_API_KEY:-test}" +{{< /doc-test >}} + ## Configuration {{< reuse "agw-docs/snippets/review-configuration.md" >}} @@ -22,6 +52,20 @@ llm: apiKey: "$BASETEN_API_KEY" ``` +{{< doc-test paths="baseten" >}} +cat <<'EOF' > config.yaml +# yaml-language-server: $schema=https://agentgateway.dev/schema/config + +llm: + models: + - name: "*" + provider: baseten + params: + apiKey: "$BASETEN_API_KEY" +EOF +agentgateway -f config.yaml --validate-only +{{< /doc-test >}} + {{< reuse "agw-docs/snippets/review-configuration.md" >}} | Setting | Description | @@ -44,3 +88,26 @@ curl -X POST http://localhost:4000/v1/chat/completions \ "messages": [{"role": "user", "content": "Hello from Baseten!"}] }' ``` + +{{< doc-test paths="baseten" >}} +# Confirm the default `params.baseUrl` documented in the settings table is what +# agentgateway actually resolves. The admin config dump reports the upstream host +# and path prefix separately, so they are recombined before comparing. +agentgateway -f config.yaml & +AGW_PID=$! +trap 'kill $AGW_PID 2>/dev/null' EXIT +sleep 3 + +EXPECTED="https://inference.baseten.co/v1" +RESOLVED=$(curl -sf --max-time 10 http://localhost:15000/config_dump | jq -r ' + [ .backends[].backend.ai + | select(. != null) + | .target.providers[].active[].endpoint + | "https://" + (.hostOverride | sub(":443$"; "")) + (.pathPrefix // "") + ] | first') +if [ "$RESOLVED" != "$EXPECTED" ]; then + echo "FAIL: settings table documents a default baseUrl of $EXPECTED but agentgateway resolved $RESOLVED" + exit 1 +fi +echo "✓ Provider default baseUrl resolves to $EXPECTED" +{{< /doc-test >}} diff --git a/content/docs/standalone/main/llm/providers/bedrock.md b/content/docs/standalone/main/llm/providers/bedrock.md index e5fa0a83b..8592d77bc 100644 --- a/content/docs/standalone/main/llm/providers/bedrock.md +++ b/content/docs/standalone/main/llm/providers/bedrock.md @@ -3,10 +3,40 @@ title: Amazon Bedrock weight: 15 icon: /integrations/providers/bw/bedrock.svg description: Route agentgateway LLM traffic to foundation models on Amazon Bedrock. +test: + bedrock: + - file: ${versionRoot}/llm/providers/bedrock.md + path: bedrock --- Configure Amazon Bedrock as an LLM provider in agentgateway. +{{< doc-test paths="bedrock" >}} +# ============================================================================ +# Doc test coverage for this guide (these comments are not rendered on the page) +# ============================================================================ +# WHAT THIS TEST VALIDATES: +# * "Configuration": the example config is accepted by agentgateway +# (--validate-only), so `provider: bedrock` is recognized and +# `params.awsRegion` is correct. +# * "Passthrough": the `passthrough: detect` config is accepted, including the +# `name: us.anthropic*` prefix match. +# * With the base config loaded, agentgateway serves the wildcard model and +# resolves it to the `bedrock` provider in the configured AWS region. +# +# WHAT THIS TEST DOES NOT VALIDATE (and why): +# * "Authentication" - external dependency; AWS credentials are resolved per +# request from the ambient environment, which the test cannot provide. +# * The Converse and Invoke boto3 examples - display-only Python snippets that +# need real AWS credentials and a Bedrock model grant. +# * "Token counting", "Extended thinking and reasoning", and "Structured +# outputs" - external dependency; each bills a live Bedrock completion. Their +# example responses and the `reasoning_effort` budget table are display-only. +# * That format translation to Bedrock's Converse API is correct - a different +# layer; verifying the translation needs a live Bedrock upstream. +{{< reuse "agw-docs/snippets/install-agentgateway-binary.md" >}} +{{< /doc-test >}} + > [!NOTE] > Agentgateway accepts requests in one of the supported [API formats](../../api-types) (such as the `/v1/chat/completions` request body shape) and returns responses in that format. > Agentgateway translates between these formats and Bedrock formats internally using Bedrock's [Converse API](https://docs.aws.amazon.com/bedrock/latest/userguide/conversation-inference-call.html). @@ -32,6 +62,20 @@ llm: awsRegion: us-west-2 ``` +{{< doc-test paths="bedrock" >}} +cat <<'EOF' > config.yaml +# yaml-language-server: $schema=https://agentgateway.dev/schema/config + +llm: + models: + - name: "*" + provider: bedrock + params: + awsRegion: us-west-2 +EOF +agentgateway -f config.yaml --validate-only +{{< /doc-test >}} + {{< reuse "agw-docs/snippets/review-configuration.md" >}} | Setting | Description | @@ -61,6 +105,20 @@ llm: passthrough: detect ``` +{{< doc-test paths="bedrock" >}} +cat <<'EOF' > config-passthrough.yaml +# yaml-language-server: $schema=https://agentgateway.dev/schema/config +llm: + models: + - name: us.anthropic* + provider: bedrock + params: + awsRegion: us-west-2 + passthrough: detect +EOF +agentgateway -f config-passthrough.yaml --validate-only +{{< /doc-test >}} + Then, you can send native Converse and Invoke requests: {{< tabs >}} @@ -222,3 +280,29 @@ curl "localhost:4000/v1/chat/completions" -H content-type:application/json -d '{ ] }' | jq ``` + +{{< doc-test paths="bedrock" >}} +# Confirm the base config serves the wildcard model and that `params.awsRegion` +# reaches the resolved provider config. +agentgateway -f config.yaml & +AGW_PID=$! +trap 'kill $AGW_PID 2>/dev/null' EXIT +sleep 3 + +SERVED=$(curl -sf --max-time 10 http://localhost:4000/v1/models | jq -r '[.data[].id] | index("*") // "missing"') +if [ "$SERVED" = "missing" ]; then + echo "FAIL: the wildcard model from the example config is not served" + exit 1 +fi +RESOLVED=$(curl -sf --max-time 10 http://localhost:15000/config_dump | jq -r ' + [ .backends[].backend.ai + | select(. != null) + | .target.providers[].active[].endpoint + | "\(.provider | keys[0])|\(.provider.bedrock.region)" + ] | first') +if [ "$RESOLVED" != "bedrock|us-west-2" ]; then + echo "FAIL: expected bedrock|us-west-2 but agentgateway resolved $RESOLVED" + exit 1 +fi +echo "✓ Wildcard model is served and resolves to bedrock in us-west-2" +{{< /doc-test >}} diff --git a/content/docs/standalone/main/llm/providers/cerebras.md b/content/docs/standalone/main/llm/providers/cerebras.md index 6c5894944..abac97186 100644 --- a/content/docs/standalone/main/llm/providers/cerebras.md +++ b/content/docs/standalone/main/llm/providers/cerebras.md @@ -3,10 +3,40 @@ title: Cerebras weight: 20 icon: /integrations/providers/bw/cerebras.svg description: Route agentgateway LLM traffic to models hosted on Cerebras. +test: + cerebras: + - file: ${versionRoot}/llm/providers/cerebras.md + path: cerebras --- Configure Cerebras as an LLM provider in agentgateway. +{{< doc-test paths="cerebras" >}} +# ============================================================================ +# Doc test coverage for this guide (these comments are not rendered on the page) +# ============================================================================ +# WHAT THIS TEST VALIDATES: +# * "Configuration": the example config is accepted by agentgateway +# (--validate-only), so `provider: cerebras` is a recognized provider and the +# `name` / `params.apiKey` fields are correct. +# * The `params.baseUrl` row of the settings table: with the config loaded, +# agentgateway resolves this provider's upstream to the documented default +# (https://api.cerebras.ai/v1), so the table cannot drift from the product without +# this test failing. +# +# WHAT THIS TEST DOES NOT VALIDATE (and why): +# * "Example request" - external dependency; sending the curl request needs a +# real Cerebras API key and bills a live completion, so the test uses a +# placeholder key and asserts on resolved config instead. +# * `params.model` - display-only table row; the example config omits it, so +# there is nothing on this page to run. +{{< reuse "agw-docs/snippets/install-agentgateway-binary.md" >}} + +# The example config reads the API key from the environment. --validate-only and +# the config dump still resolve env vars, so a placeholder is enough here. +export CEREBRAS_API_KEY="${CEREBRAS_API_KEY:-test}" +{{< /doc-test >}} + ## Configuration {{< reuse "agw-docs/snippets/review-configuration.md" >}} @@ -22,6 +52,20 @@ llm: apiKey: "$CEREBRAS_API_KEY" ``` +{{< doc-test paths="cerebras" >}} +cat <<'EOF' > config.yaml +# yaml-language-server: $schema=https://agentgateway.dev/schema/config + +llm: + models: + - name: "*" + provider: cerebras + params: + apiKey: "$CEREBRAS_API_KEY" +EOF +agentgateway -f config.yaml --validate-only +{{< /doc-test >}} + {{< reuse "agw-docs/snippets/review-configuration.md" >}} | Setting | Description | @@ -44,3 +88,26 @@ curl -X POST http://localhost:4000/v1/chat/completions \ "messages": [{"role": "user", "content": "Hello from Cerebras!"}] }' ``` + +{{< doc-test paths="cerebras" >}} +# Confirm the default `params.baseUrl` documented in the settings table is what +# agentgateway actually resolves. The admin config dump reports the upstream host +# and path prefix separately, so they are recombined before comparing. +agentgateway -f config.yaml & +AGW_PID=$! +trap 'kill $AGW_PID 2>/dev/null' EXIT +sleep 3 + +EXPECTED="https://api.cerebras.ai/v1" +RESOLVED=$(curl -sf --max-time 10 http://localhost:15000/config_dump | jq -r ' + [ .backends[].backend.ai + | select(. != null) + | .target.providers[].active[].endpoint + | "https://" + (.hostOverride | sub(":443$"; "")) + (.pathPrefix // "") + ] | first') +if [ "$RESOLVED" != "$EXPECTED" ]; then + echo "FAIL: settings table documents a default baseUrl of $EXPECTED but agentgateway resolved $RESOLVED" + exit 1 +fi +echo "✓ Provider default baseUrl resolves to $EXPECTED" +{{< /doc-test >}} diff --git a/content/docs/standalone/main/llm/providers/cohere.md b/content/docs/standalone/main/llm/providers/cohere.md index 26cdf8d55..91686acf0 100644 --- a/content/docs/standalone/main/llm/providers/cohere.md +++ b/content/docs/standalone/main/llm/providers/cohere.md @@ -3,10 +3,40 @@ title: Cohere weight: 20 icon: /integrations/providers/bw/cohere.svg description: Route agentgateway LLM traffic to Cohere's models. +test: + cohere: + - file: ${versionRoot}/llm/providers/cohere.md + path: cohere --- Configure Cohere as an LLM provider in agentgateway. +{{< doc-test paths="cohere" >}} +# ============================================================================ +# Doc test coverage for this guide (these comments are not rendered on the page) +# ============================================================================ +# WHAT THIS TEST VALIDATES: +# * "Configuration": the example config is accepted by agentgateway +# (--validate-only), so `provider: cohere` is a recognized provider and the +# `name` / `params.apiKey` fields are correct. +# * The `params.baseUrl` row of the settings table: with the config loaded, +# agentgateway resolves this provider's upstream to the documented default +# (https://api.cohere.ai), so the table cannot drift from the product without +# this test failing. +# +# WHAT THIS TEST DOES NOT VALIDATE (and why): +# * "Example request" - external dependency; sending the curl request needs a +# real Cohere API key and bills a live completion, so the test uses a +# placeholder key and asserts on resolved config instead. +# * `params.model` - display-only table row; the example config omits it, so +# there is nothing on this page to run. +{{< reuse "agw-docs/snippets/install-agentgateway-binary.md" >}} + +# The example config reads the API key from the environment. --validate-only and +# the config dump still resolve env vars, so a placeholder is enough here. +export COHERE_API_KEY="${COHERE_API_KEY:-test}" +{{< /doc-test >}} + ## Configuration {{< reuse "agw-docs/snippets/review-configuration.md" >}} @@ -22,6 +52,20 @@ llm: apiKey: "$COHERE_API_KEY" ``` +{{< doc-test paths="cohere" >}} +cat <<'EOF' > config.yaml +# yaml-language-server: $schema=https://agentgateway.dev/schema/config + +llm: + models: + - name: "*" + provider: cohere + params: + apiKey: "$COHERE_API_KEY" +EOF +agentgateway -f config.yaml --validate-only +{{< /doc-test >}} + {{< reuse "agw-docs/snippets/review-configuration.md" >}} | Setting | Description | @@ -44,3 +88,26 @@ curl -X POST http://localhost:4000/v1/chat/completions \ "messages": [{"role": "user", "content": "Hello from Cohere!"}] }' ``` + +{{< doc-test paths="cohere" >}} +# Confirm the default `params.baseUrl` documented in the settings table is what +# agentgateway actually resolves. The admin config dump reports the upstream host +# and path prefix separately, so they are recombined before comparing. +agentgateway -f config.yaml & +AGW_PID=$! +trap 'kill $AGW_PID 2>/dev/null' EXIT +sleep 3 + +EXPECTED="https://api.cohere.ai" +RESOLVED=$(curl -sf --max-time 10 http://localhost:15000/config_dump | jq -r ' + [ .backends[].backend.ai + | select(. != null) + | .target.providers[].active[].endpoint + | "https://" + (.hostOverride | sub(":443$"; "")) + (.pathPrefix // "") + ] | first') +if [ "$RESOLVED" != "$EXPECTED" ]; then + echo "FAIL: settings table documents a default baseUrl of $EXPECTED but agentgateway resolved $RESOLVED" + exit 1 +fi +echo "✓ Provider default baseUrl resolves to $EXPECTED" +{{< /doc-test >}} diff --git a/content/docs/standalone/main/llm/providers/deepinfra.md b/content/docs/standalone/main/llm/providers/deepinfra.md index 593648b9f..af4ffa2a0 100644 --- a/content/docs/standalone/main/llm/providers/deepinfra.md +++ b/content/docs/standalone/main/llm/providers/deepinfra.md @@ -3,10 +3,40 @@ title: DeepInfra weight: 20 icon: /integrations/providers/bw/deepinfra.svg description: Route agentgateway LLM traffic to models hosted on DeepInfra. +test: + deepinfra: + - file: ${versionRoot}/llm/providers/deepinfra.md + path: deepinfra --- Configure DeepInfra as an LLM provider in agentgateway. +{{< doc-test paths="deepinfra" >}} +# ============================================================================ +# Doc test coverage for this guide (these comments are not rendered on the page) +# ============================================================================ +# WHAT THIS TEST VALIDATES: +# * "Configuration": the example config is accepted by agentgateway +# (--validate-only), so `provider: deepinfra` is a recognized provider and the +# `name` / `params.apiKey` fields are correct. +# * The `params.baseUrl` row of the settings table: with the config loaded, +# agentgateway resolves this provider's upstream to the documented default +# (https://api.deepinfra.com/v1/openai), so the table cannot drift from the product without +# this test failing. +# +# WHAT THIS TEST DOES NOT VALIDATE (and why): +# * "Example request" - external dependency; sending the curl request needs a +# real DeepInfra API key and bills a live completion, so the test uses a +# placeholder key and asserts on resolved config instead. +# * `params.model` - display-only table row; the example config omits it, so +# there is nothing on this page to run. +{{< reuse "agw-docs/snippets/install-agentgateway-binary.md" >}} + +# The example config reads the API key from the environment. --validate-only and +# the config dump still resolve env vars, so a placeholder is enough here. +export DEEPINFRA_API_KEY="${DEEPINFRA_API_KEY:-test}" +{{< /doc-test >}} + ## Configuration {{< reuse "agw-docs/snippets/review-configuration.md" >}} @@ -22,6 +52,20 @@ llm: apiKey: "$DEEPINFRA_API_KEY" ``` +{{< doc-test paths="deepinfra" >}} +cat <<'EOF' > config.yaml +# yaml-language-server: $schema=https://agentgateway.dev/schema/config + +llm: + models: + - name: "*" + provider: deepinfra + params: + apiKey: "$DEEPINFRA_API_KEY" +EOF +agentgateway -f config.yaml --validate-only +{{< /doc-test >}} + {{< reuse "agw-docs/snippets/review-configuration.md" >}} | Setting | Description | @@ -44,3 +88,26 @@ curl -X POST http://localhost:4000/v1/chat/completions \ "messages": [{"role": "user", "content": "Hello from DeepInfra!"}] }' ``` + +{{< doc-test paths="deepinfra" >}} +# Confirm the default `params.baseUrl` documented in the settings table is what +# agentgateway actually resolves. The admin config dump reports the upstream host +# and path prefix separately, so they are recombined before comparing. +agentgateway -f config.yaml & +AGW_PID=$! +trap 'kill $AGW_PID 2>/dev/null' EXIT +sleep 3 + +EXPECTED="https://api.deepinfra.com/v1/openai" +RESOLVED=$(curl -sf --max-time 10 http://localhost:15000/config_dump | jq -r ' + [ .backends[].backend.ai + | select(. != null) + | .target.providers[].active[].endpoint + | "https://" + (.hostOverride | sub(":443$"; "")) + (.pathPrefix // "") + ] | first') +if [ "$RESOLVED" != "$EXPECTED" ]; then + echo "FAIL: settings table documents a default baseUrl of $EXPECTED but agentgateway resolved $RESOLVED" + exit 1 +fi +echo "✓ Provider default baseUrl resolves to $EXPECTED" +{{< /doc-test >}} diff --git a/content/docs/standalone/main/llm/providers/deepseek.md b/content/docs/standalone/main/llm/providers/deepseek.md index bddf03113..ebd787f13 100644 --- a/content/docs/standalone/main/llm/providers/deepseek.md +++ b/content/docs/standalone/main/llm/providers/deepseek.md @@ -3,10 +3,40 @@ title: DeepSeek weight: 20 icon: /integrations/providers/bw/deepseek.svg description: Route agentgateway LLM traffic to DeepSeek's models. +test: + deepseek: + - file: ${versionRoot}/llm/providers/deepseek.md + path: deepseek --- Configure DeepSeek as an LLM provider in agentgateway. +{{< doc-test paths="deepseek" >}} +# ============================================================================ +# Doc test coverage for this guide (these comments are not rendered on the page) +# ============================================================================ +# WHAT THIS TEST VALIDATES: +# * "Configuration": the example config is accepted by agentgateway +# (--validate-only), so `provider: deepseek` is a recognized provider and the +# `name` / `params.apiKey` fields are correct. +# * The `params.baseUrl` row of the settings table: with the config loaded, +# agentgateway resolves this provider's upstream to the documented default +# (https://api.deepseek.com/v1), so the table cannot drift from the product without +# this test failing. +# +# WHAT THIS TEST DOES NOT VALIDATE (and why): +# * "Example request" - external dependency; sending the curl request needs a +# real DeepSeek API key and bills a live completion, so the test uses a +# placeholder key and asserts on resolved config instead. +# * `params.model` - display-only table row; the example config omits it, so +# there is nothing on this page to run. +{{< reuse "agw-docs/snippets/install-agentgateway-binary.md" >}} + +# The example config reads the API key from the environment. --validate-only and +# the config dump still resolve env vars, so a placeholder is enough here. +export DEEPSEEK_API_KEY="${DEEPSEEK_API_KEY:-test}" +{{< /doc-test >}} + ## Configuration {{< reuse "agw-docs/snippets/review-configuration.md" >}} @@ -22,6 +52,20 @@ llm: apiKey: "$DEEPSEEK_API_KEY" ``` +{{< doc-test paths="deepseek" >}} +cat <<'EOF' > config.yaml +# yaml-language-server: $schema=https://agentgateway.dev/schema/config + +llm: + models: + - name: "*" + provider: deepseek + params: + apiKey: "$DEEPSEEK_API_KEY" +EOF +agentgateway -f config.yaml --validate-only +{{< /doc-test >}} + {{< reuse "agw-docs/snippets/review-configuration.md" >}} | Setting | Description | @@ -44,3 +88,26 @@ curl -X POST http://localhost:4000/v1/chat/completions \ "messages": [{"role": "user", "content": "Hello from DeepSeek!"}] }' ``` + +{{< doc-test paths="deepseek" >}} +# Confirm the default `params.baseUrl` documented in the settings table is what +# agentgateway actually resolves. The admin config dump reports the upstream host +# and path prefix separately, so they are recombined before comparing. +agentgateway -f config.yaml & +AGW_PID=$! +trap 'kill $AGW_PID 2>/dev/null' EXIT +sleep 3 + +EXPECTED="https://api.deepseek.com/v1" +RESOLVED=$(curl -sf --max-time 10 http://localhost:15000/config_dump | jq -r ' + [ .backends[].backend.ai + | select(. != null) + | .target.providers[].active[].endpoint + | "https://" + (.hostOverride | sub(":443$"; "")) + (.pathPrefix // "") + ] | first') +if [ "$RESOLVED" != "$EXPECTED" ]; then + echo "FAIL: settings table documents a default baseUrl of $EXPECTED but agentgateway resolved $RESOLVED" + exit 1 +fi +echo "✓ Provider default baseUrl resolves to $EXPECTED" +{{< /doc-test >}} diff --git a/content/docs/standalone/main/llm/providers/fireworks.md b/content/docs/standalone/main/llm/providers/fireworks.md index 997a13ac4..c3b11dd19 100644 --- a/content/docs/standalone/main/llm/providers/fireworks.md +++ b/content/docs/standalone/main/llm/providers/fireworks.md @@ -3,10 +3,40 @@ title: Fireworks AI weight: 20 icon: /integrations/providers/bw/fireworks.svg description: Route agentgateway LLM traffic to models hosted on Fireworks AI. +test: + fireworks: + - file: ${versionRoot}/llm/providers/fireworks.md + path: fireworks --- Configure Fireworks AI as an LLM provider in agentgateway. +{{< doc-test paths="fireworks" >}} +# ============================================================================ +# Doc test coverage for this guide (these comments are not rendered on the page) +# ============================================================================ +# WHAT THIS TEST VALIDATES: +# * "Configuration": the example config is accepted by agentgateway +# (--validate-only), so `provider: fireworks` is a recognized provider and the +# `name` / `params.apiKey` fields are correct. +# * The `params.baseUrl` row of the settings table: with the config loaded, +# agentgateway resolves this provider's upstream to the documented default +# (https://api.fireworks.ai/inference/v1), so the table cannot drift from the product without +# this test failing. +# +# WHAT THIS TEST DOES NOT VALIDATE (and why): +# * "Example request" - external dependency; sending the curl request needs a +# real Fireworks AI API key and bills a live completion, so the test uses a +# placeholder key and asserts on resolved config instead. +# * `params.model` - display-only table row; the example config omits it, so +# there is nothing on this page to run. +{{< reuse "agw-docs/snippets/install-agentgateway-binary.md" >}} + +# The example config reads the API key from the environment. --validate-only and +# the config dump still resolve env vars, so a placeholder is enough here. +export FIREWORKS_API_KEY="${FIREWORKS_API_KEY:-test}" +{{< /doc-test >}} + ## Configuration {{< reuse "agw-docs/snippets/review-configuration.md" >}} @@ -22,6 +52,20 @@ llm: apiKey: "$FIREWORKS_API_KEY" ``` +{{< doc-test paths="fireworks" >}} +cat <<'EOF' > config.yaml +# yaml-language-server: $schema=https://agentgateway.dev/schema/config + +llm: + models: + - name: "*" + provider: fireworks + params: + apiKey: "$FIREWORKS_API_KEY" +EOF +agentgateway -f config.yaml --validate-only +{{< /doc-test >}} + {{< reuse "agw-docs/snippets/review-configuration.md" >}} | Setting | Description | @@ -45,3 +89,25 @@ curl -X POST http://localhost:4000/v1/chat/completions \ }' ``` +{{< doc-test paths="fireworks" >}} +# Confirm the default `params.baseUrl` documented in the settings table is what +# agentgateway actually resolves. The admin config dump reports the upstream host +# and path prefix separately, so they are recombined before comparing. +agentgateway -f config.yaml & +AGW_PID=$! +trap 'kill $AGW_PID 2>/dev/null' EXIT +sleep 3 + +EXPECTED="https://api.fireworks.ai/inference/v1" +RESOLVED=$(curl -sf --max-time 10 http://localhost:15000/config_dump | jq -r ' + [ .backends[].backend.ai + | select(. != null) + | .target.providers[].active[].endpoint + | "https://" + (.hostOverride | sub(":443$"; "")) + (.pathPrefix // "") + ] | first') +if [ "$RESOLVED" != "$EXPECTED" ]; then + echo "FAIL: settings table documents a default baseUrl of $EXPECTED but agentgateway resolved $RESOLVED" + exit 1 +fi +echo "✓ Provider default baseUrl resolves to $EXPECTED" +{{< /doc-test >}} diff --git a/content/docs/standalone/main/llm/providers/gemini.md b/content/docs/standalone/main/llm/providers/gemini.md index 43b19bf88..b49dd8a59 100644 --- a/content/docs/standalone/main/llm/providers/gemini.md +++ b/content/docs/standalone/main/llm/providers/gemini.md @@ -3,10 +3,39 @@ title: Gemini weight: 15 icon: /integrations/providers/bw/gemini.svg description: Route agentgateway LLM traffic to Google Gemini models. +test: + gemini: + - file: ${versionRoot}/llm/providers/gemini.md + path: gemini --- Configure Google Gemini as an LLM provider in agentgateway. +{{< doc-test paths="gemini" >}} +# ============================================================================ +# Doc test coverage for this guide (these comments are not rendered on the page) +# ============================================================================ +# WHAT THIS TEST VALIDATES: +# * "Configuration": the example config is accepted by agentgateway +# (--validate-only), so `provider: gemini` is a recognized provider and the +# `name` / `params.apiKey` fields are correct. +# * With the config loaded, agentgateway serves the wildcard model from the +# example and resolves it to the `gemini` provider, which is what the `name` +# and `provider` rows of the settings table describe. +# +# WHAT THIS TEST DOES NOT VALIDATE (and why): +# * "Example request" and any example responses - external dependency; the +# request needs a real Gemini API key and bills a live completion, so the test +# uses a placeholder key. +# * `params.model` - display-only table row; the example config omits it, so +# there is nothing on this page to run. +{{< reuse "agw-docs/snippets/install-agentgateway-binary.md" >}} + +# The example config reads the API key from the environment. --validate-only +# and the config dump still resolve env vars, so a placeholder is enough here. +export GEMINI_API_KEY="${GEMINI_API_KEY:-test}" +{{< /doc-test >}} + ## Configuration {{< reuse "agw-docs/snippets/review-configuration.md" >}} @@ -22,6 +51,20 @@ llm: apiKey: "$GEMINI_API_KEY" ``` +{{< doc-test paths="gemini" >}} +cat <<'EOF' > config.yaml +# yaml-language-server: $schema=https://agentgateway.dev/schema/config + +llm: + models: + - name: "*" + provider: gemini + params: + apiKey: "$GEMINI_API_KEY" +EOF +agentgateway -f config.yaml --validate-only +{{< /doc-test >}} + {{< reuse "agw-docs/snippets/review-configuration.md" >}} | Setting | Description | @@ -30,3 +73,30 @@ llm: | `provider` | The LLM provider, set to `gemini` for Google Gemini models. | | `params.model` | The specific Gemini model to use. If set, this model is used for all requests. If not set, the request must include the model to use. | | `params.apiKey` | The Gemini API key for authentication. You can reference environment variables using the `$VAR_NAME` syntax. | + +{{< doc-test paths="gemini" >}} +# Confirm the config serves the model named in the example and resolves it to +# this provider. First-class providers use built-in upstream defaults, so the +# config dump reports the provider discriminant rather than a host override. +agentgateway -f config.yaml & +AGW_PID=$! +trap 'kill $AGW_PID 2>/dev/null' EXIT +sleep 3 + +SERVED=$(curl -sf --max-time 10 http://localhost:4000/v1/models | jq -r '[.data[].id] | index("*") // "missing"') +if [ "$SERVED" = "missing" ]; then + echo "FAIL: the wildcard model from the example config is not served" + exit 1 +fi +PROVIDER=$(curl -sf --max-time 10 http://localhost:15000/config_dump | jq -r ' + [ .backends[].backend.ai + | select(. != null) + | .target.providers[].active[].endpoint + | .provider | keys[0] + ] | first') +if [ "$PROVIDER" != "gemini" ]; then + echo "FAIL: expected provider gemini but agentgateway resolved $PROVIDER" + exit 1 +fi +echo "✓ The wildcard model is served and resolves to the gemini provider" +{{< /doc-test >}} diff --git a/content/docs/standalone/main/llm/providers/groq.md b/content/docs/standalone/main/llm/providers/groq.md index 68031d246..b2a10fb0c 100644 --- a/content/docs/standalone/main/llm/providers/groq.md +++ b/content/docs/standalone/main/llm/providers/groq.md @@ -3,10 +3,40 @@ title: Groq weight: 20 icon: /integrations/providers/bw/groq.svg description: Route agentgateway LLM traffic to models served by Groq. +test: + groq: + - file: ${versionRoot}/llm/providers/groq.md + path: groq --- Configure Groq as an LLM provider in agentgateway. +{{< doc-test paths="groq" >}} +# ============================================================================ +# Doc test coverage for this guide (these comments are not rendered on the page) +# ============================================================================ +# WHAT THIS TEST VALIDATES: +# * "Configuration": the example config is accepted by agentgateway +# (--validate-only), so `provider: groq` is a recognized provider and the +# `name` / `params.apiKey` fields are correct. +# * The `params.baseUrl` row of the settings table: with the config loaded, +# agentgateway resolves this provider's upstream to the documented default +# (https://api.groq.com/openai/v1), so the table cannot drift from the product without +# this test failing. +# +# WHAT THIS TEST DOES NOT VALIDATE (and why): +# * "Example request" - external dependency; sending the curl request needs a +# real Groq API key and bills a live completion, so the test uses a +# placeholder key and asserts on resolved config instead. +# * `params.model` - display-only table row; the example config omits it, so +# there is nothing on this page to run. +{{< reuse "agw-docs/snippets/install-agentgateway-binary.md" >}} + +# The example config reads the API key from the environment. --validate-only and +# the config dump still resolve env vars, so a placeholder is enough here. +export GROQ_API_KEY="${GROQ_API_KEY:-test}" +{{< /doc-test >}} + ## Configuration {{< reuse "agw-docs/snippets/review-configuration.md" >}} @@ -22,6 +52,20 @@ llm: apiKey: "$GROQ_API_KEY" ``` +{{< doc-test paths="groq" >}} +cat <<'EOF' > config.yaml +# yaml-language-server: $schema=https://agentgateway.dev/schema/config + +llm: + models: + - name: "*" + provider: groq + params: + apiKey: "$GROQ_API_KEY" +EOF +agentgateway -f config.yaml --validate-only +{{< /doc-test >}} + {{< reuse "agw-docs/snippets/review-configuration.md" >}} | Setting | Description | @@ -44,3 +88,26 @@ curl -X POST http://localhost:4000/v1/chat/completions \ "messages": [{"role": "user", "content": "Hello from Groq!"}] }' ``` + +{{< doc-test paths="groq" >}} +# Confirm the default `params.baseUrl` documented in the settings table is what +# agentgateway actually resolves. The admin config dump reports the upstream host +# and path prefix separately, so they are recombined before comparing. +agentgateway -f config.yaml & +AGW_PID=$! +trap 'kill $AGW_PID 2>/dev/null' EXIT +sleep 3 + +EXPECTED="https://api.groq.com/openai/v1" +RESOLVED=$(curl -sf --max-time 10 http://localhost:15000/config_dump | jq -r ' + [ .backends[].backend.ai + | select(. != null) + | .target.providers[].active[].endpoint + | "https://" + (.hostOverride | sub(":443$"; "")) + (.pathPrefix // "") + ] | first') +if [ "$RESOLVED" != "$EXPECTED" ]; then + echo "FAIL: settings table documents a default baseUrl of $EXPECTED but agentgateway resolved $RESOLVED" + exit 1 +fi +echo "✓ Provider default baseUrl resolves to $EXPECTED" +{{< /doc-test >}} diff --git a/content/docs/standalone/main/llm/providers/huggingface.md b/content/docs/standalone/main/llm/providers/huggingface.md index 59053994f..e9406f38e 100644 --- a/content/docs/standalone/main/llm/providers/huggingface.md +++ b/content/docs/standalone/main/llm/providers/huggingface.md @@ -3,10 +3,40 @@ title: Hugging Face weight: 20 icon: /integrations/providers/bw/huggingface.svg description: Route agentgateway LLM traffic to models hosted on Hugging Face. +test: + huggingface: + - file: ${versionRoot}/llm/providers/huggingface.md + path: huggingface --- Configure Hugging Face as an LLM provider in agentgateway. +{{< doc-test paths="huggingface" >}} +# ============================================================================ +# Doc test coverage for this guide (these comments are not rendered on the page) +# ============================================================================ +# WHAT THIS TEST VALIDATES: +# * "Configuration": the example config is accepted by agentgateway +# (--validate-only), so `provider: huggingface` is a recognized provider and the +# `name` / `params.apiKey` fields are correct. +# * The `params.baseUrl` row of the settings table: with the config loaded, +# agentgateway resolves this provider's upstream to the documented default +# (https://router.huggingface.co/v1), so the table cannot drift from the product without +# this test failing. +# +# WHAT THIS TEST DOES NOT VALIDATE (and why): +# * "Example request" - external dependency; sending the curl request needs a +# real Hugging Face API key and bills a live completion, so the test uses a +# placeholder key and asserts on resolved config instead. +# * `params.model` - display-only table row; the example config omits it, so +# there is nothing on this page to run. +{{< reuse "agw-docs/snippets/install-agentgateway-binary.md" >}} + +# The example config reads the API key from the environment. --validate-only and +# the config dump still resolve env vars, so a placeholder is enough here. +export HUGGINGFACE_API_KEY="${HUGGINGFACE_API_KEY:-test}" +{{< /doc-test >}} + ## Configuration {{< reuse "agw-docs/snippets/review-configuration.md" >}} @@ -22,6 +52,20 @@ llm: apiKey: "$HUGGINGFACE_API_KEY" ``` +{{< doc-test paths="huggingface" >}} +cat <<'EOF' > config.yaml +# yaml-language-server: $schema=https://agentgateway.dev/schema/config + +llm: + models: + - name: "*" + provider: huggingface + params: + apiKey: "$HUGGINGFACE_API_KEY" +EOF +agentgateway -f config.yaml --validate-only +{{< /doc-test >}} + {{< reuse "agw-docs/snippets/review-configuration.md" >}} | Setting | Description | @@ -44,3 +88,26 @@ curl -X POST http://localhost:4000/v1/chat/completions \ "messages": [{"role": "user", "content": "Hello from Hugging Face!"}] }' ``` + +{{< doc-test paths="huggingface" >}} +# Confirm the default `params.baseUrl` documented in the settings table is what +# agentgateway actually resolves. The admin config dump reports the upstream host +# and path prefix separately, so they are recombined before comparing. +agentgateway -f config.yaml & +AGW_PID=$! +trap 'kill $AGW_PID 2>/dev/null' EXIT +sleep 3 + +EXPECTED="https://router.huggingface.co/v1" +RESOLVED=$(curl -sf --max-time 10 http://localhost:15000/config_dump | jq -r ' + [ .backends[].backend.ai + | select(. != null) + | .target.providers[].active[].endpoint + | "https://" + (.hostOverride | sub(":443$"; "")) + (.pathPrefix // "") + ] | first') +if [ "$RESOLVED" != "$EXPECTED" ]; then + echo "FAIL: settings table documents a default baseUrl of $EXPECTED but agentgateway resolved $RESOLVED" + exit 1 +fi +echo "✓ Provider default baseUrl resolves to $EXPECTED" +{{< /doc-test >}} diff --git a/content/docs/standalone/main/llm/providers/mistral.md b/content/docs/standalone/main/llm/providers/mistral.md index b22043beb..580348ad8 100644 --- a/content/docs/standalone/main/llm/providers/mistral.md +++ b/content/docs/standalone/main/llm/providers/mistral.md @@ -3,10 +3,40 @@ title: Mistral weight: 20 icon: /integrations/providers/bw/mistral.svg description: Route agentgateway LLM traffic to Mistral's models. +test: + mistral: + - file: ${versionRoot}/llm/providers/mistral.md + path: mistral --- Configure Mistral as an LLM provider in agentgateway. +{{< doc-test paths="mistral" >}} +# ============================================================================ +# Doc test coverage for this guide (these comments are not rendered on the page) +# ============================================================================ +# WHAT THIS TEST VALIDATES: +# * "Configuration": the example config is accepted by agentgateway +# (--validate-only), so `provider: mistral` is a recognized provider and the +# `name` / `params.apiKey` fields are correct. +# * The `params.baseUrl` row of the settings table: with the config loaded, +# agentgateway resolves this provider's upstream to the documented default +# (https://api.mistral.ai/v1), so the table cannot drift from the product without +# this test failing. +# +# WHAT THIS TEST DOES NOT VALIDATE (and why): +# * "Example request" - external dependency; sending the curl request needs a +# real Mistral API key and bills a live completion, so the test uses a +# placeholder key and asserts on resolved config instead. +# * `params.model` - display-only table row; the example config omits it, so +# there is nothing on this page to run. +{{< reuse "agw-docs/snippets/install-agentgateway-binary.md" >}} + +# The example config reads the API key from the environment. --validate-only and +# the config dump still resolve env vars, so a placeholder is enough here. +export MISTRAL_API_KEY="${MISTRAL_API_KEY:-test}" +{{< /doc-test >}} + ## Configuration {{< reuse "agw-docs/snippets/review-configuration.md" >}} @@ -22,6 +52,20 @@ llm: apiKey: "$MISTRAL_API_KEY" ``` +{{< doc-test paths="mistral" >}} +cat <<'EOF' > config.yaml +# yaml-language-server: $schema=https://agentgateway.dev/schema/config + +llm: + models: + - name: "*" + provider: mistral + params: + apiKey: "$MISTRAL_API_KEY" +EOF +agentgateway -f config.yaml --validate-only +{{< /doc-test >}} + {{< reuse "agw-docs/snippets/review-configuration.md" >}} | Setting | Description | @@ -45,3 +89,25 @@ curl -X POST http://localhost:4000/v1/chat/completions \ }' ``` +{{< doc-test paths="mistral" >}} +# Confirm the default `params.baseUrl` documented in the settings table is what +# agentgateway actually resolves. The admin config dump reports the upstream host +# and path prefix separately, so they are recombined before comparing. +agentgateway -f config.yaml & +AGW_PID=$! +trap 'kill $AGW_PID 2>/dev/null' EXIT +sleep 3 + +EXPECTED="https://api.mistral.ai/v1" +RESOLVED=$(curl -sf --max-time 10 http://localhost:15000/config_dump | jq -r ' + [ .backends[].backend.ai + | select(. != null) + | .target.providers[].active[].endpoint + | "https://" + (.hostOverride | sub(":443$"; "")) + (.pathPrefix // "") + ] | first') +if [ "$RESOLVED" != "$EXPECTED" ]; then + echo "FAIL: settings table documents a default baseUrl of $EXPECTED but agentgateway resolved $RESOLVED" + exit 1 +fi +echo "✓ Provider default baseUrl resolves to $EXPECTED" +{{< /doc-test >}} diff --git a/content/docs/standalone/main/llm/providers/multiple-llms.md b/content/docs/standalone/main/llm/providers/multiple-llms.md index d2124c4b8..c86268c41 100644 --- a/content/docs/standalone/main/llm/providers/multiple-llms.md +++ b/content/docs/standalone/main/llm/providers/multiple-llms.md @@ -2,8 +2,39 @@ title: Multiple LLM providers weight: 30 description: Define reusable LLM provider configurations once and reference them across multiple model definitions to avoid duplicating connection and authentication parameters. +test: + multiple-llms: + - file: ${versionRoot}/llm/providers/multiple-llms.md + path: multiple-llms --- +{{< doc-test paths="multiple-llms" >}} +# ============================================================================ +# Doc test coverage for this guide (these comments are not rendered on the page) +# ============================================================================ +# WHAT THIS TEST VALIDATES: +# * "Reusable provider configuration": the example config is accepted by +# agentgateway (--validate-only), covering `llm.providers[].name`, +# `llm.providers[].provider`, and `llm.models[].provider.reference`. +# * A reference actually resolves at runtime: with the config loaded, both the +# `fast` and `smart` models are served, which is only possible if each model +# inherited its upstream provider (and API key) from the `openai-prod` entry +# in `llm.providers[]`. A dangling reference fails to load. +# +# WHAT THIS TEST DOES NOT VALIDATE (and why): +# * That a completion request routed through `fast` or `smart` reaches OpenAI - +# external dependency; the test uses a placeholder API key and does not call +# the provider. +# * The other shared upstream settings the page mentions (host overrides, path +# overrides, other model defaults) - display-only prose with no example +# config on this page that sets them. +{{< reuse "agw-docs/snippets/install-agentgateway-binary.md" >}} + +# The example config reads the API key from the environment. --validate-only and +# model listing still resolve env vars, so a placeholder is enough here. +export OPENAI_API_KEY="${OPENAI_API_KEY:-test}" +{{< /doc-test >}} + ## Reusable provider configuration Reuse provider configuration to avoid duplicating connection and authentication parameters across multiple model definitions. Define named provider defaults once in `llm.providers[]` and reference them from multiple `llm.models[]` entries with `provider.reference`. @@ -12,7 +43,7 @@ Reuse provider configuration to avoid duplicating connection and authentication llm: providers: - name: openai-prod - provider: openai + provider: openAI params: apiKey: "$OPENAI_API_KEY" @@ -29,6 +60,64 @@ llm: model: gpt-4o ``` +{{< doc-test paths="multiple-llms" >}} +cat <<'EOF' > config.yaml +llm: + providers: + - name: openai-prod + provider: openAI + params: + apiKey: "$OPENAI_API_KEY" + + models: + - name: fast + provider: + reference: openai-prod + params: + model: gpt-4o-mini + - name: smart + provider: + reference: openai-prod + params: + model: gpt-4o +EOF +agentgateway -f config.yaml --validate-only +{{< /doc-test >}} + +{{< doc-test paths="multiple-llms" >}} +# Simplified LLM mode with no explicit port serves on 4000. +agentgateway -f config.yaml & +AGW_PID=$! +trap 'kill $AGW_PID 2>/dev/null' EXIT +sleep 3 +{{< /doc-test >}} + +{{< doc-test paths="multiple-llms" >}} +YAMLTest -f - <<'EOF' +- name: Both models that reference the shared provider are served + retries: 3 + http: + url: "http://localhost:4000" + path: /v1/models + method: GET + headers: + accept-encoding: identity + source: + type: local + expect: + statusCode: 200 + bodyJsonPath: + # Filter expressions rather than $.data[*].id, because a wildcard path + # resolves to a single match and the model order is not guaranteed. + - path: "$.data[?(@.id=='fast')].id" + comparator: equals + value: fast + - path: "$.data[?(@.id=='smart')].id" + comparator: equals + value: smart +EOF +{{< /doc-test >}} + In this example, `smart` inherits the upstream API key from `llm.providers[]` and only changes the model name. Named providers can hold shared upstream settings you want to reuse, such as authentication, host overrides, path overrides, or other model defaults. diff --git a/content/docs/standalone/main/llm/providers/openai.md b/content/docs/standalone/main/llm/providers/openai.md index 79b7c2cae..fda9f4104 100644 --- a/content/docs/standalone/main/llm/providers/openai.md +++ b/content/docs/standalone/main/llm/providers/openai.md @@ -3,10 +3,39 @@ title: OpenAI weight: 10 icon: /integrations/providers/bw/openai.svg description: Route agentgateway LLM traffic to OpenAI's GPT models. +test: + openai: + - file: ${versionRoot}/llm/providers/openai.md + path: openai --- Configure OpenAI as an LLM provider in agentgateway. +{{< doc-test paths="openai" >}} +# ============================================================================ +# Doc test coverage for this guide (these comments are not rendered on the page) +# ============================================================================ +# WHAT THIS TEST VALIDATES: +# * "Configuration": the example config is accepted by agentgateway +# (--validate-only), so `provider: openAI` is a recognized provider and the +# `name` / `params.apiKey` fields are correct. +# * With the config loaded, agentgateway serves the wildcard model from the +# example and resolves it to the `openAI` provider, which is what the `name` +# and `provider` rows of the settings table describe. +# +# WHAT THIS TEST DOES NOT VALIDATE (and why): +# * "Example request" and any example responses - external dependency; the +# request needs a real OpenAI API key and bills a live completion, so the test +# uses a placeholder key. +# * `params.model` - display-only table row; the example config omits it, so +# there is nothing on this page to run. +{{< reuse "agw-docs/snippets/install-agentgateway-binary.md" >}} + +# The example config reads the API key from the environment. --validate-only +# and the config dump still resolve env vars, so a placeholder is enough here. +export OPENAI_API_KEY="${OPENAI_API_KEY:-test}" +{{< /doc-test >}} + ## Configuration {{< reuse "agw-docs/snippets/review-configuration.md" >}} @@ -22,6 +51,20 @@ llm: apiKey: "$OPENAI_API_KEY" ``` +{{< doc-test paths="openai" >}} +cat <<'EOF' > config.yaml +# yaml-language-server: $schema=https://agentgateway.dev/schema/config + +llm: + models: + - name: "*" + provider: openAI + params: + apiKey: "$OPENAI_API_KEY" +EOF +agentgateway -f config.yaml --validate-only +{{< /doc-test >}} + {{< reuse "agw-docs/snippets/review-configuration.md" >}} | Setting | Description | @@ -36,3 +79,30 @@ llm: > [!NOTE] > To connect Codex to agentgateway, see the [Codex integration page]({{< link-hextra path="/integrations/llm-clients/codex" >}}). + +{{< doc-test paths="openai" >}} +# Confirm the config serves the model named in the example and resolves it to +# this provider. First-class providers use built-in upstream defaults, so the +# config dump reports the provider discriminant rather than a host override. +agentgateway -f config.yaml & +AGW_PID=$! +trap 'kill $AGW_PID 2>/dev/null' EXIT +sleep 3 + +SERVED=$(curl -sf --max-time 10 http://localhost:4000/v1/models | jq -r '[.data[].id] | index("*") // "missing"') +if [ "$SERVED" = "missing" ]; then + echo "FAIL: the wildcard model from the example config is not served" + exit 1 +fi +PROVIDER=$(curl -sf --max-time 10 http://localhost:15000/config_dump | jq -r ' + [ .backends[].backend.ai + | select(. != null) + | .target.providers[].active[].endpoint + | .provider | keys[0] + ] | first') +if [ "$PROVIDER" != "openAI" ]; then + echo "FAIL: expected provider openAI but agentgateway resolved $PROVIDER" + exit 1 +fi +echo "✓ The wildcard model is served and resolves to the openAI provider" +{{< /doc-test >}} diff --git a/content/docs/standalone/main/llm/providers/openrouter.md b/content/docs/standalone/main/llm/providers/openrouter.md index 978e74555..afe4c611e 100644 --- a/content/docs/standalone/main/llm/providers/openrouter.md +++ b/content/docs/standalone/main/llm/providers/openrouter.md @@ -3,10 +3,40 @@ title: OpenRouter weight: 20 icon: /integrations/providers/bw/openrouter.svg description: Route agentgateway LLM traffic to models available through OpenRouter. +test: + openrouter: + - file: ${versionRoot}/llm/providers/openrouter.md + path: openrouter --- Configure OpenRouter as an LLM provider in agentgateway. +{{< doc-test paths="openrouter" >}} +# ============================================================================ +# Doc test coverage for this guide (these comments are not rendered on the page) +# ============================================================================ +# WHAT THIS TEST VALIDATES: +# * "Configuration": the example config is accepted by agentgateway +# (--validate-only), so `provider: openrouter` is a recognized provider and the +# `name` / `params.apiKey` fields are correct. +# * The `params.baseUrl` row of the settings table: with the config loaded, +# agentgateway resolves this provider's upstream to the documented default +# (https://openrouter.ai/api/v1), so the table cannot drift from the product without +# this test failing. +# +# WHAT THIS TEST DOES NOT VALIDATE (and why): +# * "Example request" - external dependency; sending the curl request needs a +# real OpenRouter API key and bills a live completion, so the test uses a +# placeholder key and asserts on resolved config instead. +# * `params.model` - display-only table row; the example config omits it, so +# there is nothing on this page to run. +{{< reuse "agw-docs/snippets/install-agentgateway-binary.md" >}} + +# The example config reads the API key from the environment. --validate-only and +# the config dump still resolve env vars, so a placeholder is enough here. +export OPENROUTER_API_KEY="${OPENROUTER_API_KEY:-test}" +{{< /doc-test >}} + ## Configuration {{< reuse "agw-docs/snippets/review-configuration.md" >}} @@ -22,6 +52,20 @@ llm: apiKey: "$OPENROUTER_API_KEY" ``` +{{< doc-test paths="openrouter" >}} +cat <<'EOF' > config.yaml +# yaml-language-server: $schema=https://agentgateway.dev/schema/config + +llm: + models: + - name: "*" + provider: openrouter + params: + apiKey: "$OPENROUTER_API_KEY" +EOF +agentgateway -f config.yaml --validate-only +{{< /doc-test >}} + {{< reuse "agw-docs/snippets/review-configuration.md" >}} | Setting | Description | @@ -45,3 +89,25 @@ curl -X POST http://localhost:4000/v1/chat/completions \ }' ``` +{{< doc-test paths="openrouter" >}} +# Confirm the default `params.baseUrl` documented in the settings table is what +# agentgateway actually resolves. The admin config dump reports the upstream host +# and path prefix separately, so they are recombined before comparing. +agentgateway -f config.yaml & +AGW_PID=$! +trap 'kill $AGW_PID 2>/dev/null' EXIT +sleep 3 + +EXPECTED="https://openrouter.ai/api/v1" +RESOLVED=$(curl -sf --max-time 10 http://localhost:15000/config_dump | jq -r ' + [ .backends[].backend.ai + | select(. != null) + | .target.providers[].active[].endpoint + | "https://" + (.hostOverride | sub(":443$"; "")) + (.pathPrefix // "") + ] | first') +if [ "$RESOLVED" != "$EXPECTED" ]; then + echo "FAIL: settings table documents a default baseUrl of $EXPECTED but agentgateway resolved $RESOLVED" + exit 1 +fi +echo "✓ Provider default baseUrl resolves to $EXPECTED" +{{< /doc-test >}} diff --git a/content/docs/standalone/main/llm/providers/togetherai.md b/content/docs/standalone/main/llm/providers/togetherai.md index e4f44d1fd..cbd959969 100644 --- a/content/docs/standalone/main/llm/providers/togetherai.md +++ b/content/docs/standalone/main/llm/providers/togetherai.md @@ -3,10 +3,40 @@ title: Together AI weight: 20 icon: /integrations/providers/bw/togetherai.svg description: Route agentgateway LLM traffic to models hosted on Together AI. +test: + togetherai: + - file: ${versionRoot}/llm/providers/togetherai.md + path: togetherai --- Configure Together AI as an LLM provider in agentgateway. +{{< doc-test paths="togetherai" >}} +# ============================================================================ +# Doc test coverage for this guide (these comments are not rendered on the page) +# ============================================================================ +# WHAT THIS TEST VALIDATES: +# * "Configuration": the example config is accepted by agentgateway +# (--validate-only), so `provider: togetherai` is a recognized provider and the +# `name` / `params.apiKey` fields are correct. +# * The `params.baseUrl` row of the settings table: with the config loaded, +# agentgateway resolves this provider's upstream to the documented default +# (https://api.together.xyz/v1), so the table cannot drift from the product without +# this test failing. +# +# WHAT THIS TEST DOES NOT VALIDATE (and why): +# * "Example request" - external dependency; sending the curl request needs a +# real Together AI API key and bills a live completion, so the test uses a +# placeholder key and asserts on resolved config instead. +# * `params.model` - display-only table row; the example config omits it, so +# there is nothing on this page to run. +{{< reuse "agw-docs/snippets/install-agentgateway-binary.md" >}} + +# The example config reads the API key from the environment. --validate-only and +# the config dump still resolve env vars, so a placeholder is enough here. +export TOGETHER_API_KEY="${TOGETHER_API_KEY:-test}" +{{< /doc-test >}} + ## Configuration {{< reuse "agw-docs/snippets/review-configuration.md" >}} @@ -22,6 +52,20 @@ llm: apiKey: "$TOGETHER_API_KEY" ``` +{{< doc-test paths="togetherai" >}} +cat <<'EOF' > config.yaml +# yaml-language-server: $schema=https://agentgateway.dev/schema/config + +llm: + models: + - name: "*" + provider: togetherai + params: + apiKey: "$TOGETHER_API_KEY" +EOF +agentgateway -f config.yaml --validate-only +{{< /doc-test >}} + {{< reuse "agw-docs/snippets/review-configuration.md" >}} | Setting | Description | @@ -45,3 +89,25 @@ curl -X POST http://localhost:4000/v1/chat/completions \ }' ``` +{{< doc-test paths="togetherai" >}} +# Confirm the default `params.baseUrl` documented in the settings table is what +# agentgateway actually resolves. The admin config dump reports the upstream host +# and path prefix separately, so they are recombined before comparing. +agentgateway -f config.yaml & +AGW_PID=$! +trap 'kill $AGW_PID 2>/dev/null' EXIT +sleep 3 + +EXPECTED="https://api.together.xyz/v1" +RESOLVED=$(curl -sf --max-time 10 http://localhost:15000/config_dump | jq -r ' + [ .backends[].backend.ai + | select(. != null) + | .target.providers[].active[].endpoint + | "https://" + (.hostOverride | sub(":443$"; "")) + (.pathPrefix // "") + ] | first') +if [ "$RESOLVED" != "$EXPECTED" ]; then + echo "FAIL: settings table documents a default baseUrl of $EXPECTED but agentgateway resolved $RESOLVED" + exit 1 +fi +echo "✓ Provider default baseUrl resolves to $EXPECTED" +{{< /doc-test >}} diff --git a/content/docs/standalone/main/llm/providers/vertex.md b/content/docs/standalone/main/llm/providers/vertex.md index 85912c88b..05a68aa76 100644 --- a/content/docs/standalone/main/llm/providers/vertex.md +++ b/content/docs/standalone/main/llm/providers/vertex.md @@ -3,10 +3,39 @@ title: Vertex AI weight: 15 icon: /integrations/providers/bw/vertex.svg description: Route agentgateway LLM traffic to models on Google Cloud Vertex AI. +test: + vertex: + - file: ${versionRoot}/llm/providers/vertex.md + path: vertex --- Configure Google Cloud Vertex AI as an LLM provider in agentgateway. +{{< doc-test paths="vertex" >}} +# ============================================================================ +# Doc test coverage for this guide (these comments are not rendered on the page) +# ============================================================================ +# WHAT THIS TEST VALIDATES: +# * "Configuration": the example config is accepted by agentgateway +# (--validate-only), so `provider: vertex` is recognized and +# `params.model` / `params.vertexProject` / `params.vertexRegion` are correct. +# * The settings table rows for `name`, `params.model`, `params.vertexProject`, +# and `params.vertexRegion`: with the config loaded, agentgateway serves the +# client-facing model name `gemini-2.5-flash` and resolves the upstream to the +# configured model, project ID, and region. This makes the distinction between +# `name` (matched in requests) and `params.model` (sent upstream) observable +# rather than only asserted in prose. +# +# WHAT THIS TEST DOES NOT VALIDATE (and why): +# * "Authentication" - external dependency; Application Default Credentials +# require a real Google Cloud identity, which the test cannot stand up. The +# config loads without credentials because ADC is resolved per request. +# * The `auth.gcp` table row - display-only; the example config omits it and +# relies on the ADC default. +# * That a completion reaches Vertex AI - external dependency, as above. +{{< reuse "agw-docs/snippets/install-agentgateway-binary.md" >}} +{{< /doc-test >}} + ## Authentication Before you can use Vertex AI as an LLM provider, you must authenticate by using Google Cloud's [Application Default Credentials](https://docs.cloud.google.com/docs/authentication/application-default-credentials). Choose from one of the three methods: @@ -32,6 +61,22 @@ llm: vertexRegion: us-west2 ``` +{{< doc-test paths="vertex" >}} +cat <<'EOF' > config.yaml +# yaml-language-server: $schema=https://agentgateway.dev/schema/config + +llm: + models: + - name: gemini-2.5-flash + provider: vertex + params: + model: google/gemini-2.5-flash-lite-preview-06-17 + vertexProject: my-project-id + vertexRegion: us-west2 +EOF +agentgateway -f config.yaml --validate-only +{{< /doc-test >}} + {{< reuse "agw-docs/snippets/review-configuration.md" >}} | Setting | Description | @@ -42,3 +87,32 @@ llm: | `params.vertexProject` | The Google Cloud project ID. | | `params.vertexRegion` | The Google Cloud region. Defaults to `global` if not specified. | | `auth.gcp` | Google Cloud authentication configuration. Uses Application Default Credentials (ADC) by default. | + +{{< doc-test paths="vertex" >}} +# Confirm the client-facing `name` is served and that `params.model`, +# `params.vertexProject`, and `params.vertexRegion` reach the resolved provider +# config as documented in the settings table. +agentgateway -f config.yaml & +AGW_PID=$! +trap 'kill $AGW_PID 2>/dev/null' EXIT +sleep 3 + +SERVED=$(curl -sf --max-time 10 http://localhost:4000/v1/models | jq -r '[.data[].id] | index("gemini-2.5-flash") // "missing"') +if [ "$SERVED" = "missing" ]; then + echo "FAIL: the model name gemini-2.5-flash from the example config is not served" + exit 1 +fi +RESOLVED=$(curl -sf --max-time 10 http://localhost:15000/config_dump | jq -r ' + [ .backends[].backend.ai + | select(. != null) + | .target.providers[].active[].endpoint + | .provider.vertex + | "\(.model)|\(.projectId)|\(.region)" + ] | first') +EXPECTED="google/gemini-2.5-flash-lite-preview-06-17|my-project-id|us-west2" +if [ "$RESOLVED" != "$EXPECTED" ]; then + echo "FAIL: expected vertex params $EXPECTED but agentgateway resolved $RESOLVED" + exit 1 +fi +echo "✓ Vertex model, project, and region resolve to the documented values" +{{< /doc-test >}} diff --git a/content/docs/standalone/main/llm/providers/xai.md b/content/docs/standalone/main/llm/providers/xai.md index d61e1bd57..23025e91b 100644 --- a/content/docs/standalone/main/llm/providers/xai.md +++ b/content/docs/standalone/main/llm/providers/xai.md @@ -3,10 +3,40 @@ title: xAI weight: 20 icon: /integrations/providers/bw/xai.svg description: Route agentgateway LLM traffic to xAI's Grok models. +test: + xai: + - file: ${versionRoot}/llm/providers/xai.md + path: xai --- Configure xAI (Grok) as an LLM provider in agentgateway. +{{< doc-test paths="xai" >}} +# ============================================================================ +# Doc test coverage for this guide (these comments are not rendered on the page) +# ============================================================================ +# WHAT THIS TEST VALIDATES: +# * "Configuration": the example config is accepted by agentgateway +# (--validate-only), so `provider: xai` is a recognized provider and the +# `name` / `params.apiKey` fields are correct. +# * The `params.baseUrl` row of the settings table: with the config loaded, +# agentgateway resolves this provider's upstream to the documented default +# (https://api.x.ai/v1), so the table cannot drift from the product without +# this test failing. +# +# WHAT THIS TEST DOES NOT VALIDATE (and why): +# * "Example request" - external dependency; sending the curl request needs a +# real xAI API key and bills a live completion, so the test uses a +# placeholder key and asserts on resolved config instead. +# * `params.model` - display-only table row; the example config omits it, so +# there is nothing on this page to run. +{{< reuse "agw-docs/snippets/install-agentgateway-binary.md" >}} + +# The example config reads the API key from the environment. --validate-only and +# the config dump still resolve env vars, so a placeholder is enough here. +export XAI_API_KEY="${XAI_API_KEY:-test}" +{{< /doc-test >}} + ## Configuration {{< reuse "agw-docs/snippets/review-configuration.md" >}} @@ -22,6 +52,20 @@ llm: apiKey: "$XAI_API_KEY" ``` +{{< doc-test paths="xai" >}} +cat <<'EOF' > config.yaml +# yaml-language-server: $schema=https://agentgateway.dev/schema/config + +llm: + models: + - name: "*" + provider: xai + params: + apiKey: "$XAI_API_KEY" +EOF +agentgateway -f config.yaml --validate-only +{{< /doc-test >}} + {{< reuse "agw-docs/snippets/review-configuration.md" >}} | Setting | Description | @@ -45,3 +89,25 @@ curl -X POST http://localhost:4000/v1/chat/completions \ }' ``` +{{< doc-test paths="xai" >}} +# Confirm the default `params.baseUrl` documented in the settings table is what +# agentgateway actually resolves. The admin config dump reports the upstream host +# and path prefix separately, so they are recombined before comparing. +agentgateway -f config.yaml & +AGW_PID=$! +trap 'kill $AGW_PID 2>/dev/null' EXIT +sleep 3 + +EXPECTED="https://api.x.ai/v1" +RESOLVED=$(curl -sf --max-time 10 http://localhost:15000/config_dump | jq -r ' + [ .backends[].backend.ai + | select(. != null) + | .target.providers[].active[].endpoint + | "https://" + (.hostOverride | sub(":443$"; "")) + (.pathPrefix // "") + ] | first') +if [ "$RESOLVED" != "$EXPECTED" ]; then + echo "FAIL: settings table documents a default baseUrl of $EXPECTED but agentgateway resolved $RESOLVED" + exit 1 +fi +echo "✓ Provider default baseUrl resolves to $EXPECTED" +{{< /doc-test >}} diff --git a/content/docs/standalone/main/llm/virtual-models.md b/content/docs/standalone/main/llm/virtual-models.md index c9c0e698a..eeebacd40 100644 --- a/content/docs/standalone/main/llm/virtual-models.md +++ b/content/docs/standalone/main/llm/virtual-models.md @@ -2,8 +2,66 @@ title: Virtual models weight: 47 description: Configure virtual models with weighted, failover, and conditional routing in simplified LLM mode. +test: + virtual-models: + - file: ${versionRoot}/llm/virtual-models.md + path: virtual-models --- +{{< doc-test paths="virtual-models" >}} +# ============================================================================ +# Doc test coverage for this guide (these comments are not rendered on the page) +# ============================================================================ +# WHAT THIS TEST VALIDATES: +# * All three example configs are accepted by agentgateway (--validate-only), +# covering `llm.virtualModels[].routing.weighted.targets[].weight`, +# `routing.failover.targets[].priority`, and `routing.conditional.targets[].when`. +# * "Public and internal models": with each config loaded, the served model list +# contains the virtual model and any `visibility: public` model, and omits every +# `visibility: internal` model. This turns the prose description of `public` and +# `internal` into an observable assertion: +# - weighted -> gpt-4o-public, smart (2 internal targets hidden) +# - failover -> resilient (all 3 targets are internal) +# - conditional -> openai-public, adaptive (2 internal targets hidden) +# The failover case is the clearest: every target is internal, so only the +# virtual entrypoint is exposed. +# +# WHAT THIS TEST DOES NOT VALIDATE (and why): +# * That traffic is actually split 90/10 by `weight` - external dependency; +# observing the split needs many live completions against OpenAI. +# * That failover moves to a lower `priority` target on failure, and that +# same-priority targets are load balanced "based on health and latency" - +# external dependency; triggering a real upstream failure needs live providers. +# * That `when` expressions select a target by request header - requires +# config/traffic the page omits; the page shows no request example, and +# confirming which internal target served a response needs a live provider +# call. +{{< reuse "agw-docs/snippets/install-agentgateway-binary.md" >}} + +# The example configs read API keys from the environment. --validate-only and the +# model listing still resolve env vars, so placeholders are enough here. +export OPENAI_API_KEY="${OPENAI_API_KEY:-test}" +export ANTHROPIC_API_KEY="${ANTHROPIC_API_KEY:-test}" + +# Assert that a config serves exactly the expected client-facing models, which is +# what `visibility: public` / `internal` controls. +assert_models() { + local cfg="$1" expected="$2" + agentgateway -f "$cfg" & + local pid=$! + sleep 3 + local served + served=$(curl -sf --max-time 10 http://localhost:4000/v1/models | jq -cr '[.data[].id] | sort') + kill $pid 2>/dev/null + wait $pid 2>/dev/null + if [ "$served" != "$expected" ]; then + echo "FAIL: $cfg should serve $expected but served $served" + exit 1 + fi + echo "✓ $cfg serves $expected (internal targets are not exposed)" +} +{{< /doc-test >}} + Virtual models let you publish one client-facing model name and route requests across one or more internal target models. Use `llm.virtualModels[]` to define the virtual entrypoint and `llm.models[]` as the concrete upstream targets. @@ -57,6 +115,43 @@ llm: weight: 10 ``` +{{< doc-test paths="virtual-models" >}} +cat <<'EOF' > config-weighted.yaml +llm: + models: + - name: gpt-4o-public + visibility: public + provider: openAI + params: + model: gpt-4o + apiKey: "$OPENAI_API_KEY" + - name: gpt-4o-primary + visibility: internal + provider: openAI + params: + model: gpt-4o + apiKey: "$OPENAI_API_KEY" + - name: gpt-4o-fallback + visibility: internal + provider: openAI + params: + model: gpt-4o-mini + apiKey: "$OPENAI_API_KEY" + + virtualModels: + - name: smart + routing: + weighted: + targets: + - model: gpt-4o-primary + weight: 90 + - model: gpt-4o-fallback + weight: 10 +EOF +agentgateway -f config-weighted.yaml --validate-only +assert_models config-weighted.yaml '["gpt-4o-public","smart"]' +{{< /doc-test >}} + ### Failover routing Use `routing.failover.targets` and `priority` to define ordered failover targets. @@ -97,6 +192,45 @@ llm: priority: 2 ``` +{{< doc-test paths="virtual-models" >}} +cat <<'EOF' > config-failover.yaml +llm: + models: + - name: claude-primary + visibility: internal + provider: anthropic + params: + model: claude-sonnet-4-0 + apiKey: "$ANTHROPIC_API_KEY" + - name: claude-backup-a + visibility: internal + provider: anthropic + params: + model: claude-3-5-haiku-20241022 + apiKey: "$ANTHROPIC_API_KEY" + - name: claude-backup-b + visibility: internal + provider: anthropic + params: + model: claude-3-5-haiku-20241022 + apiKey: "$ANTHROPIC_API_KEY" + + virtualModels: + - name: resilient + routing: + failover: + targets: + - model: claude-primary + priority: 1 + - model: claude-backup-a + priority: 2 + - model: claude-backup-b + priority: 2 +EOF +agentgateway -f config-failover.yaml --validate-only +assert_models config-failover.yaml '["resilient"]' +{{< /doc-test >}} + ### Conditional routing Use `routing.conditional.targets` and `when` expressions to select targets by request context. @@ -134,5 +268,42 @@ llm: when: request.headers["x-tier"] == "pro" ``` +{{< doc-test paths="virtual-models" >}} +cat <<'EOF' > config-conditional.yaml +llm: + models: + - name: openai-public + visibility: public + provider: openAI + params: + model: gpt-4o-mini + apiKey: "$OPENAI_API_KEY" + - name: openai-fast + visibility: internal + provider: openAI + params: + model: gpt-4o-mini + apiKey: "$OPENAI_API_KEY" + - name: openai-smart + visibility: internal + provider: openAI + params: + model: gpt-4o + apiKey: "$OPENAI_API_KEY" + + virtualModels: + - name: adaptive + routing: + conditional: + targets: + - model: openai-fast + when: request.headers["x-tier"] == "free" + - model: openai-smart + when: request.headers["x-tier"] == "pro" +EOF +agentgateway -f config-conditional.yaml --validate-only +assert_models config-conditional.yaml '["adaptive","openai-public"]' +{{< /doc-test >}} + > [!NOTE] > For reusable provider defaults in simplified mode, see [Multiple LLM providers]({{< link-hextra path="/llm/providers/multiple-llms/" >}}). diff --git a/content/docs/standalone/main/mcp/connect/stdio.md b/content/docs/standalone/main/mcp/connect/stdio.md index 2894071c3..d0f092d91 100644 --- a/content/docs/standalone/main/mcp/connect/stdio.md +++ b/content/docs/standalone/main/mcp/connect/stdio.md @@ -2,10 +2,56 @@ title: Stdio weight: 10 description: Run a local MCP server as a subprocess and expose it through agentgateway over stdio. +test: + mcp-stdio: + - file: ${versionRoot}/mcp/connect/stdio.md + path: mcp-stdio --- An MCP backend allows exposing MCP servers through the agentgateway using {{< gloss "STDIO (Standard Input/Output)" >}}STDIO{{< /gloss >}}. +{{< doc-test paths="mcp-stdio" >}} +# ============================================================================ +# Doc test coverage for this guide (these comments are not rendered on the page) +# ============================================================================ +# WHAT THIS TEST VALIDATES: +# * "Configure the agentgateway" step 1: the documented download URL resolves and +# returns a config that agentgateway accepts (--validate-only). +# * "Verify access to tools" steps 2-3, through the MCP API rather than the UI +# playground: an MCP session initializes, tools/list includes the `echo` tool +# that the page tells you to select, and calling `echo` with the page's example +# message returns that message echoed back. These are the UI steps' scriptable +# equivalents, so the walkthrough's end state is verified even though the +# clicks are not. +# +# WHAT THIS TEST DOES NOT VALIDATE (and why): +# * The agentgateway UI itself (opening the UI, the Tool Playground, the +# "Apply CORS" button, the Result card screenshots) - UI-only steps with no +# command-line equivalent. The test drives the same MCP endpoint the playground +# drives. +# * The github-yaml rendering of the config in step 2 - display-only; it +# embeds the same file the test downloads and validates. +{{< reuse "agw-docs/snippets/install-agentgateway-binary.md" >}} + +# Open an MCP session and return its session ID. MCP responses are server-sent +# events, so `data:` lines are unwrapped before parsing. +mcp_session() { + curl -sS -D - -o /dev/null --max-time 30 -X POST http://localhost:3000/mcp \ + -H 'Content-Type: application/json' \ + -H 'Accept: application/json, text/event-stream' \ + -d '{"jsonrpc":"2.0","method":"initialize","params":{"protocolVersion":"2024-11-05","capabilities":{},"clientInfo":{"name":"doctest","version":"1.0"}},"id":1}' \ + | grep -i '^mcp-session-id:' | tr -d '\r' | awk '{print $2}' +} + +mcp_call() { + curl -sS --max-time 30 -X POST http://localhost:3000/mcp \ + -H 'Content-Type: application/json' \ + -H 'Accept: application/json, text/event-stream' \ + -H "Mcp-Session-Id: $1" \ + -d "$2" | sed -n 's/^data: //p' +} +{{< /doc-test >}} + ## Before you begin {{< reuse "agw-docs/snippets/prereq-agentgateway.md" >}} @@ -14,10 +60,14 @@ An MCP backend allows exposing MCP servers through the agentgateway using {{< gl 1. Download an MCP configuration for your agentgateway. - ```yaml + ```yaml {paths="mcp-stdio"} curl -L https://agentgateway.dev/examples/mcp-basic/config.yaml -o config.yaml ``` + {{< doc-test paths="mcp-stdio" >}} + agentgateway -f config.yaml --validate-only + {{< /doc-test >}} + 2. Review the configuration file. ``` @@ -55,3 +105,55 @@ An MCP backend allows exposing MCP servers through the agentgateway using {{< gl {{< reuse-image-light src="img/ui-playground-tool-echo.png" >}} {{< reuse-image-dark srcDark="img/ui-playground-tool-echo-dark.png" >}} + +{{< doc-test paths="mcp-stdio" >}} +# Run the gateway in the background so the MCP assertions below can drive it. The +# visible "Run the agentgateway" block is untagged because it runs in the foreground. +agentgateway -f config.yaml & +AGW_PID=$! +trap 'kill $AGW_PID 2>/dev/null' EXIT +# The stdio target launches the MCP server through npx, which downloads the package +# on first use, so allow time for the target to become ready. +for i in $(seq 1 30); do + curl -sf -o /dev/null --max-time 5 http://localhost:15021/healthz/ready && break + sleep 2 +done +{{< /doc-test >}} + +{{< doc-test paths="mcp-stdio" >}} +# The API equivalent of the "Verify access to tools" playground steps: open a +# session, confirm the `echo` tool the page tells you to select is listed, then call +# it with the page's example message and check the echoed result. +SESSION="" +for i in $(seq 1 20); do + SESSION=$(mcp_session) + [ -n "$SESSION" ] && break + sleep 3 +done +if [ -z "$SESSION" ]; then + echo "FAIL: could not open an MCP session against the configured target" + exit 1 +fi +echo "✓ MCP session initialized" + +mcp_call "$SESSION" '{"jsonrpc":"2.0","method":"notifications/initialized"}' >/dev/null + +TOOLS=$(mcp_call "$SESSION" '{"jsonrpc":"2.0","method":"tools/list","id":2}') +if [ "$(jq -r '[.result.tools[].name] | index("echo") // "missing"' <<<"$TOOLS")" = "missing" ]; then + echo "FAIL: tools/list did not include the echo tool" + jq -c '[.result.tools[].name]' <<<"$TOOLS" + exit 1 +fi +echo "✓ tools/list includes the echo tool" + +RESULT=$(mcp_call "$SESSION" '{"jsonrpc":"2.0","method":"tools/call","params":{"name":"echo","arguments":{"message":"This is my first agentgateway setup."}},"id":3}') +TEXT=$(jq -r '.result.content[0].text // ""' <<<"$RESULT") +case "$TEXT" in + *"This is my first agentgateway setup."*) + echo "✓ Calling the echo tool returned the message: $TEXT" ;; + *) + echo "FAIL: the echo tool did not echo the message back" + echo "$RESULT" + exit 1 ;; +esac +{{< /doc-test >}} diff --git a/content/docs/standalone/main/mcp/connect/virtual.md b/content/docs/standalone/main/mcp/connect/virtual.md index 36234a5c1..cb8389e22 100644 --- a/content/docs/standalone/main/mcp/connect/virtual.md +++ b/content/docs/standalone/main/mcp/connect/virtual.md @@ -2,10 +2,118 @@ title: Virtual MCP weight: 20 description: Federate multiple MCP servers into a unified virtual MCP backend +test: + mcp-virtual: + - file: ${versionRoot}/mcp/connect/virtual.md + path: mcp-virtual --- Federate tools of multiple MCP servers on the agentgateway by using MCP {{< gloss "Multiplex" >}}multiplexing{{< /gloss >}}. +{{< doc-test paths="mcp-virtual" >}} +# ============================================================================ +# Doc test coverage for this guide (these comments are not rendered on the page) +# ============================================================================ +# WHAT THIS TEST VALIDATES: +# * "Configure the agentgateway" step 1: the documented download URL resolves and +# returns a config that agentgateway accepts (--validate-only). +# * "Verify access to tools" steps 5-7, through the MCP API rather than the UI +# playground: the downloaded multiplex config runs, tools/list returns tools +# federated from both targets with target-name prefixes +# (`time_get_current_time`, `everything_echo`), calling `everything_echo` +# echoes the page's example message, and calling `time_get_current_time` with +# `America/New_York` returns a time result. +# * "Tool name prefixing": the `prefixMode: never` example config is accepted, and +# all three rows of the prefixMode table are asserted at runtime against a live +# MCP session: +# - conditional (default), two targets -> names are prefixed +# - always, one target -> names are prefixed even with one target +# - never -> names are plain (echo) +# +# WHAT THIS TEST DOES NOT VALIDATE (and why): +# * The published multiplex config's `time` target needs `uvx mcp-server-time` to +# resolve an MCP Python SDK older than 2.x, because that release renamed +# McpError and the server fails to import against it. agentgateway#2873 adds the +# `--with mcp<2` constraint to the example; until that lands and redeploys, the +# test applies the same constraint to its local copy. Once the published config +# carries it, the test runs the downloaded file verbatim. +# * The agentgateway UI steps (Tool Playground, Apply CORS, Initialize, the +# screenshots) - UI-only, no command-line equivalent. The test drives the same +# MCP endpoint the playground drives. +# * The two collapsed "details" example configs ("Example multiplexing configuration" +# and "Example load balancing configuration") - display-only structural excerpts; +# neither is a complete config (no gateways/routes, and the backends entry omits +# its required `name`), so neither can be validated as written. +# * The step 3 optional CORS config - display-only; it is an abbreviated snippet +# ending in `...`, not a complete file. +# * That load balancing distributes across backends by weight - requires +# config/traffic the page omits; the load balancing example is contrast material, +# not a walkthrough. +{{< reuse "agw-docs/snippets/install-agentgateway-binary.md" >}} + +# The stdio targets launch their MCP servers through npx and uvx. Fetch both up +# front: otherwise the first start pays a cold registry download inside every +# readiness retry loop below, which is slow enough to time the test out. +npm install -g @modelcontextprotocol/server-everything >/dev/null 2>&1 || true + +# "Before you begin" step 2 installs uv. The agentgateway install snippet already put +# $HOME/.local/bin on PATH, which is where the uv installer places its binaries. +if ! command -v uvx >/dev/null 2>&1; then + curl -LsSf https://astral.sh/uv/install.sh | sh >/dev/null 2>&1 || true +fi +uvx --with 'mcp<2' mcp-server-time --help >/dev/null 2>&1 || true + +# Open an MCP session and list the tool names it exposes. MCP responses are +# server-sent events, so `data:` lines are unwrapped before parsing. +mcp_tool_names() { + local sid + sid=$(curl -sS -D - -o /dev/null --max-time 10 -X POST http://localhost:3000/mcp \ + -H 'Content-Type: application/json' -H 'Accept: application/json, text/event-stream' \ + -d '{"jsonrpc":"2.0","method":"initialize","params":{"protocolVersion":"2024-11-05","capabilities":{},"clientInfo":{"name":"doctest","version":"1.0"}},"id":1}' \ + | grep -i '^mcp-session-id:' | tr -d '\r' | awk '{print $2}') + [ -n "$sid" ] || return 1 + curl -sS --max-time 10 -o /dev/null -X POST http://localhost:3000/mcp \ + -H 'Content-Type: application/json' -H 'Accept: application/json, text/event-stream' \ + -H "Mcp-Session-Id: $sid" -d '{"jsonrpc":"2.0","method":"notifications/initialized"}' + echo "$sid" > .mcp-session + curl -sS --max-time 15 -X POST http://localhost:3000/mcp \ + -H 'Content-Type: application/json' -H 'Accept: application/json, text/event-stream' \ + -H "Mcp-Session-Id: $sid" -d '{"jsonrpc":"2.0","method":"tools/list","id":2}' \ + | sed -n 's/^data: //p' | jq -r '[.result.tools[].name] | join(" ")' +} + +# Start a config in the background. This must NOT be called inside a command +# substitution: AGW_PID would be set in the subshell and the parent would never stop +# the gateway, leaving port 3000 held for the next config. The gateway's output goes +# to a file for the same reason -- inside `$( )` it would be captured into the value. +start_gateway() { + agentgateway -f "$1" > "agw-$1.log" 2>&1 & + AGW_PID=$! +} + +# Wait for the stdio targets to come up and echo the multiplexed tool names. Pure +# curl, no background jobs, so it is safe to call inside a command substitution. +# ~15 attempts x (10s max curl + 2s sleep) bounds this at about 3 minutes. +wait_for_tools() { + local names="" + for i in $(seq 1 15); do + names=$(mcp_tool_names 2>/dev/null || true) + [ -n "$names" ] && break + sleep 2 + done + echo "$names" +} + +stop_gateway() { + [ -n "${AGW_PID:-}" ] || return 0 + kill "$AGW_PID" 2>/dev/null || true + wait "$AGW_PID" 2>/dev/null || true + AGW_PID="" +} + +trap 'stop_gateway' EXIT +{{< /doc-test >}} + ## About multiplexing {#about} Multiplexing combines multiple MCP servers (targets) within a single backend into one unified MCP server. All targets are exposed together so that clients can access tools from all targets simultaneously. By default, when a backend has more than one target, tool names are prefixed with the target name (e.g., `time_get_current_time`, `everything_echo`) to avoid collisions. You can change this behavior with the `prefixMode` field, described in [Tool name prefixing](#tool-name-prefixing). @@ -20,7 +128,7 @@ backends: - name: time stdio: cmd: uvx - args: ["mcp-server-time"] + args: ["--with", "mcp<2", "mcp-server-time"] - name: everything stdio: cmd: npx @@ -59,10 +167,15 @@ routes: 1. Download a multiplex configuration for your agentgateway. - ```yaml + ```yaml {paths="mcp-virtual"} curl -L https://agentgateway.dev/examples/mcp-multiplex/config.yaml -o config.yaml ``` + {{< doc-test paths="mcp-virtual" >}} + # Step 1: the documented multiplex config downloads and is accepted + agentgateway -f config.yaml --validate-only + {{< /doc-test >}} + 2. Review the configuration file. ``` @@ -150,13 +263,122 @@ mcp: - name: time stdio: cmd: uvx - args: ["mcp-server-time"] + args: ["--with", "mcp<2", "mcp-server-time"] - name: everything stdio: cmd: npx args: ["@modelcontextprotocol/server-everything"] ``` +> [!NOTE] +> The `time` target pins the MCP Python SDK with `--with mcp<2` because `mcp-server-time` does not yet support version 2.x of the SDK. Without the constraint, the target fails to start. Drop the constraint after `mcp-server-time` adds support. + ## Next steps - Apply different policies to different MCP targets with [MCP target policies]({{< link-hextra path="/mcp/mcp-target-policies/" >}}). + +{{< doc-test paths="mcp-virtual" >}} +# Run the downloaded multiplex config and assert the federated tool list and both +# tool calls from "Verify access to tools" steps 5-7. +# +# The published example's `time` target needs an MCP Python SDK older than 2.x +# (agentgateway#2873). Use the downloaded file as-is once it carries that +# constraint; until then, apply the same constraint to a local copy so the target +# can start. +if grep -q 'mcp<2' config.yaml; then + cp config.yaml config-multiplex.yaml +else + sed 's/args: \["mcp-server-time"\]/args: ["--with", "mcp<2", "mcp-server-time"]/' \ + config.yaml > config-multiplex.yaml +fi +agentgateway -f config-multiplex.yaml --validate-only + +start_gateway config-multiplex.yaml +NAMES=$(wait_for_tools) +case "$NAMES" in + *time_get_current_time*) ;; + *) echo "FAIL: tools/list did not include time_get_current_time from the time target" + echo "$NAMES"; exit 1 ;; +esac +case "$NAMES" in + *everything_echo*) ;; + *) echo "FAIL: tools/list did not include everything_echo from the everything target" + echo "$NAMES"; exit 1 ;; +esac +echo "✓ Step 5: tools/list federates both targets with target-name prefixes" + +SESSION=$(cat .mcp-session) +mcp_tool_call() { + curl -sS --max-time 15 -X POST http://localhost:3000/mcp \ + -H 'Content-Type: application/json' -H 'Accept: application/json, text/event-stream' \ + -H "Mcp-Session-Id: $SESSION" -d "$1" | sed -n 's/^data: //p' +} + +RESULT=$(mcp_tool_call '{"jsonrpc":"2.0","method":"tools/call","params":{"name":"everything_echo","arguments":{"message":"hello world"}},"id":3}') +case "$(jq -r '.result.content[0].text // ""' <<<"$RESULT")" in + *"hello world"*) echo "✓ Step 6: everything_echo routes to the everything target and echoes the message" ;; + *) echo "FAIL: everything_echo did not return the message"; echo "$RESULT"; exit 1 ;; +esac + +RESULT=$(mcp_tool_call '{"jsonrpc":"2.0","method":"tools/call","params":{"name":"time_get_current_time","arguments":{"timezone":"America/New_York"}},"id":4}') +case "$(jq -r '.result.content[0].text // ""' <<<"$RESULT")" in + *America/New_York*) echo "✓ Step 7: time_get_current_time routes to the time target and returns a time" ;; + *) echo "FAIL: time_get_current_time did not return a result for America/New_York"; echo "$RESULT"; exit 1 ;; +esac +stop_gateway +echo "✓ prefixMode conditional (default): two targets produce prefixed names" +{{< /doc-test >}} + +{{< doc-test paths="mcp-virtual" >}} +# "Tool name prefixing": validate the documented prefixMode: never config, then assert +# the always and never rows of the table. Both use a single npx target, because +# `always` is only distinguishable from the default with one target and `never` +# requires names that do not collide. +cat <<'EOF' > config-prefix-never.yaml +# yaml-language-server: $schema=https://agentgateway.dev/schema/config +mcp: + port: 3000 + prefixMode: never + targets: + - name: time + stdio: + cmd: uvx + args: ["--with", "mcp<2", "mcp-server-time"] + - name: everything + stdio: + cmd: npx + args: ["@modelcontextprotocol/server-everything"] +EOF +agentgateway -f config-prefix-never.yaml --validate-only + +cat <<'EOF' > config-always.yaml +mcp: + port: 3000 + prefixMode: always + targets: + - name: alpha + stdio: + cmd: npx + args: ["@modelcontextprotocol/server-everything"] +EOF +sed 's/prefixMode: always/prefixMode: never/' config-always.yaml > config-never.yaml +agentgateway -f config-always.yaml --validate-only >/dev/null +agentgateway -f config-never.yaml --validate-only >/dev/null + +start_gateway config-always.yaml +NAMES=$(wait_for_tools) +stop_gateway +case "$NAMES" in + *alpha_echo*) echo "✓ prefixMode always: a single target still produces prefixed names" ;; + *) echo "FAIL: prefixMode always did not prefix names for a single target"; echo "$NAMES"; exit 1 ;; +esac + +start_gateway config-never.yaml +NAMES=$(wait_for_tools) +stop_gateway +case "$NAMES" in + *alpha_echo*) echo "FAIL: prefixMode never still prefixed names"; echo "$NAMES"; exit 1 ;; + *echo*) echo "✓ prefixMode never: names are unprefixed (echo)" ;; + *) echo "FAIL: prefixMode never did not expose the echo tool"; echo "$NAMES"; exit 1 ;; +esac +{{< /doc-test >}} diff --git a/content/docs/standalone/main/mcp/mcp-target-policies.md b/content/docs/standalone/main/mcp/mcp-target-policies.md index 982d616ce..42b1a7fa3 100644 --- a/content/docs/standalone/main/mcp/mcp-target-policies.md +++ b/content/docs/standalone/main/mcp/mcp-target-policies.md @@ -2,10 +2,50 @@ title: MCP target policies weight: 50 description: Scope policies to a single MCP server inside a multiplexed (virtual) MCP backend. +test: + mcp-target-policies: + - file: ${versionRoot}/mcp/mcp-target-policies.md + path: mcp-target-policies --- Apply policies at the MCP target level to control behavior for individual MCP servers within a multiplexed backend. +{{< doc-test paths="mcp-target-policies" >}} +# ============================================================================ +# Doc test coverage for this guide (these comments are not rendered on the page) +# ============================================================================ +# WHAT THIS TEST VALIDATES: +# * "Authentication per target": the example config is accepted by agentgateway +# (--validate-only), covering `mcp.targets[].policies` with `backendAuth.key` +# and `backendTLS.hostname` set per target. This example documented a +# `backendTLS.sni` field until this test was added; the schema calls it +# `hostname` ("Server name to use for TLS verification and SNI"), and `sni` was +# rejected as an unknown field. +# * "Supported policy types": each of the three listed policies +# (`backendAuth`, `backendTLS`, `requestHeaderModifier`) is accepted at the MCP +# target level. The table also listed `responseHeaderModifier` until this test +# was added; agentgateway rejects it there as an unknown field, so it moved to +# the unsupported note. +# * "Policy inheritance": a config that sets a policy at both the backend group +# level and the target level is accepted, so the documented two-level shape is +# valid. +# * All three unsupported policies from the note (`mcpAuthorization`, `ai`, `a2a`) +# are rejected as unknown fields at the target level, same as +# `responseHeaderModifier`. +# +# WHAT THIS TEST DOES NOT VALIDATE (and why): +# * That target-level policies actually override backend-level ones at request +# time - requires config/traffic the page omits; both example targets point at +# placeholder MCP servers (service-a.example.com) that the test cannot stand up, +# and the page shows no request to inspect. +# * The "Best practices" bullets - prose guidance, not runnable. +{{< reuse "agw-docs/snippets/install-agentgateway-binary.md" >}} + +export SERVICE_A_API_KEY="${SERVICE_A_API_KEY:-test}" +export SERVICE_B_API_KEY="${SERVICE_B_API_KEY:-test}" +{{< /doc-test >}} + + ## Overview MCP target policies allow you to configure policies for specific MCP backend targets, rather than applying them globally to all targets in a backend. This is useful when you have multiple MCP servers with different authentication or routing requirements. @@ -28,12 +68,12 @@ The following policies can be configured at the MCP target level. | `backendAuth` | Backend authentication (API key, passthrough, AWS, GCP, Azure) | | `backendTLS` | TLS configuration for backend connections | | `requestHeaderModifier` | Modify request headers | -| `responseHeaderModifier` | Modify response headers | > **Note:** The following policies are **not supported** at the MCP target level. They must be configured at the backend level instead: > - `mcpAuthorization`: Fine-grained authorization rules for tools, prompts, and resources. > - `ai`: LLM processing policies such as prompt guards, overrides, defaults, and model aliases. > - `a2a`: Mark traffic as agent-to-agent. +> - `responseHeaderModifier`: Modify response headers. Target-level policies apply to the connection that agentgateway opens to the target, so configure response header changes on the route or the backend instead. ### Policy inheritance @@ -68,7 +108,7 @@ mcp: backendAuth: key: "$SERVICE_A_API_KEY" backendTLS: - sni: service-a.example.com + hostname: service-a.example.com - name: service-b mcp: @@ -77,11 +117,114 @@ mcp: backendAuth: key: "$SERVICE_B_API_KEY" backendTLS: - sni: service-b.example.com + hostname: service-b.example.com ``` +{{< doc-test paths="mcp-target-policies" >}} +cat <<'EOF' > config.yaml +# yaml-language-server: $schema=https://agentgateway.dev/schema/config +mcp: + port: 3000 + targets: + - name: service-a + mcp: + host: https://service-a.example.com/mcp + policies: + backendAuth: + key: "$SERVICE_A_API_KEY" + backendTLS: + hostname: service-a.example.com + + - name: service-b + mcp: + host: https://service-b.example.com/mcp + policies: + backendAuth: + key: "$SERVICE_B_API_KEY" + backendTLS: + hostname: service-b.example.com +EOF +agentgateway -f config.yaml --validate-only +{{< /doc-test >}} + ## Learn more - [MCP Authorization]({{< link-hextra path="/mcp/mcp-authz" >}}) - [Backend Authentication]({{< link-hextra path="/configuration/security/backend-authn" >}}) - [Configuration Reference]({{< link-hextra path="/reference/configuration/schema/" >}}) + +{{< doc-test paths="mcp-target-policies" >}} +# "Supported policy types": the two header-modifier policies the table lists are also +# accepted at the target level, and "Policy inheritance": a policy set at the backend +# group level alongside a target-level override is a valid shape. +cat <<'EOF' > config-all-policies.yaml +# yaml-language-server: $schema=https://agentgateway.dev/schema/config +gateways: + default: + port: 3000 +routes: +- backends: + - mcp: + targets: + - name: service-a + mcp: + host: https://service-a.example.com/mcp + policies: + requestHeaderModifier: + add: + x-target: service-a + backendAuth: + key: "$SERVICE_A_API_KEY" + backendTLS: + hostname: service-a.example.com + policies: + backendAuth: + key: "$SERVICE_B_API_KEY" + responseHeaderModifier: + add: + x-from-backend: service-a +EOF +agentgateway -f config-all-policies.yaml --validate-only +echo "✓ All three documented target-level policies plus the backend/target inheritance shape are accepted" + +# The unsupported note says responseHeaderModifier belongs on the route or backend, +# not the target. Confirm agentgateway actually rejects it at the target level, so the +# note cannot drift back to claiming it is supported. +cat <<'EOF' > config-bad-target-policy.yaml +mcp: + port: 3000 + targets: + - name: service-a + mcp: + host: https://service-a.example.com/mcp + policies: + responseHeaderModifier: + add: + x-from-target: service-a +EOF +if agentgateway -f config-bad-target-policy.yaml --validate-only >/dev/null 2>&1; then + echo "FAIL: responseHeaderModifier was accepted at the MCP target level, so the unsupported note is now wrong" + exit 1 +fi +echo "✓ responseHeaderModifier is rejected at the MCP target level, as the note states" + +# The note also lists mcpAuthorization, ai, and a2a as unsupported at the target +# level. Confirm all three are rejected the same way responseHeaderModifier is. +for policy in mcpAuthorization ai a2a; do + cat < "config-bad-$policy.yaml" +mcp: + port: 3000 + targets: + - name: service-a + mcp: + host: https://service-a.example.com/mcp + policies: + $policy: {} +EOF + if agentgateway -f "config-bad-$policy.yaml" --validate-only >/dev/null 2>&1; then + echo "FAIL: $policy was accepted at the MCP target level, so the unsupported note is now wrong" + exit 1 + fi + echo "✓ $policy is rejected at the MCP target level, as the note states" +done +{{< /doc-test >}}