Skip to content
Open
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
77 changes: 77 additions & 0 deletions openhands/usage/llms/google-llms.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
description: OpenHands uses LiteLLM to make calls to Google's chat models. You can find their documentation on using Google as a provider -> [Gemini - Google AI Studio](https://docs.litellm.ai/docs/providers/gemini), [VertexAI - Google Cloud Platform](https://docs.litellm.ai/docs/providers/vertex)
---

## Gemini - Google AI Studio Configs

Check warning on line 6 in openhands/usage/llms/google-llms.mdx

View check run for this annotation

Mintlify / Mintlify Validation (allhandsai) - vale-spellcheck

openhands/usage/llms/google-llms.mdx#L6

Did you really mean 'Configs'?

When running OpenHands, you'll need to set the following in the OpenHands UI through the Settings under the `LLM` tab:
- `LLM Provider` to `Gemini`
Expand All @@ -12,7 +12,7 @@
(e.g. gemini/<model-name> like `gemini/gemini-2.0-flash`).
- `API Key` to your Gemini API key

## VertexAI - Google Cloud Platform Configs

Check warning on line 15 in openhands/usage/llms/google-llms.mdx

View check run for this annotation

Mintlify / Mintlify Validation (allhandsai) - vale-spellcheck

openhands/usage/llms/google-llms.mdx#L15

Did you really mean 'Configs'?

To use Vertex AI through Google Cloud Platform when running OpenHands, you'll need to set the following environment
variables using `-e` in the docker run command:
Expand All @@ -28,3 +28,80 @@
- `LLM Model` to the model you will be using.
If the model is not in the list, enable `Advanced` options, and enter it in `Custom Model`
(e.g. vertex_ai/<model-name>).

### Vertex AI Dependencies

The `vertex_ai/*` models (including Gemini and Claude via Vertex AI) require the
`google-cloud-aiplatform` package, which is **not included by default** in the published
agent-server image. How you enable it depends on your deployment:
Comment thread
rajshah4 marked this conversation as resolved.

<Note>
Unlike AWS Bedrock (whose `boto3` dependency is bundled by default), Vertex AI support is
opt-in. If you skip this step, you will see a <code>ModuleNotFoundError: No module named
'vertexai'</code> error when the agent tries to call a Vertex AI model.
</Note>

#### Local / Non-Docker Install

Install the `vertex` extra in your Python environment:

```bash
pip install "openhands-sdk[vertex]"
Comment thread
rajshah4 marked this conversation as resolved.
# or, with uv:
uv sync --extra vertex

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Suggestion: uv sync only works from inside a project directory whose pyproject.toml declares the vertex extra. End users running OpenHands (rather than hacking on it) will not be in such a directory, so this command will fail with Project directory not found or similar. Either drop the uv variant, or replace it with uv pip install "openhands-sdk[vertex]" which works in any Python environment the same way the pip install line above does.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed — replaced uv sync --extra vertex with uv pip install "openhands-sdk[vertex]" (commit 93966ca), which works in any Python environment.

```

#### Custom Agent-Server Image

Build the image with the `ENABLE_VERTEX` build flag:

```bash
docker build \
--build-arg ENABLE_VERTEX=1 \
Comment thread
rajshah4 marked this conversation as resolved.
-t my-agent-server:vertex \
-f openhands-agent-server/openhands/agent_server/docker/Dockerfile \
.
```

Then point OpenHands at your custom image via the `AGENT_SERVER_IMAGE_REPOSITORY` and
`AGENT_SERVER_IMAGE_TAG` environment variables (see the
[Custom Sandbox Guide](/openhands/usage/advanced/custom-sandbox-guide) for details).

#### OpenHands Enterprise (Replicated / Kubernetes)

The default OHE installer Vertex path routes LLM calls through a LiteLLM proxy — the
agent-server uses a `litellm_proxy/...` model, and the proxy makes the actual Vertex call.
So the agent-server image does **not** need Vertex enabled for the default path; `ENABLE_VERTEX=1`
is only relevant if you customize OHE to bypass the proxy and call `vertex_ai/*` directly from
the agent-server.
Comment thread
rajshah4 marked this conversation as resolved.

### Claude via Vertex AI

If you route Anthropic Claude through Google Vertex AI / Model Garden (rather than direct
Anthropic endpoints), use the `vertex_ai/` prefix with the Vertex-published model name,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Critical: The Anthropic Python SDK does not expose a vertex extras spec — there is no anthropic[vertex] distribution on PyPI. LiteLLM routes Claude-via-Vertex calls through google-cloud-aiplatform (already covered by the vertex extra), not through any anthropic[vertex] package. A user following this note will hit ERROR: Could not find a version that satisfies the requirement anthropic[vertex] and conclude the Vertex setup is broken. Either remove this paragraph entirely, or replace it with the actual LiteLLM-recommended install command (typically just google-cloud-aiplatform, possibly with the anthropic SDK itself for some LiteLLM code paths).

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pushing back on this one — anthropic[vertex] is a real PyPI extra. Verified directly against the published wheel METADATA:

Provides-Extra: vertex
Requires-Dist: google-auth[requests]<3,>=2; extra == 'vertex'

pip install "anthropic[vertex]" resolves successfully (tested against anthropic 0.112.0). The note stands, since Claude-via-Vertex historically surfaced ModuleNotFoundError: No module named 'anthropic' (OpenHands/OpenHands#3505) when only google-cloud-aiplatform was present.

which is date-stamped:

- `Custom Model`: `vertex_ai/claude-3-5-sonnet@20240620`

Use the exact model name shown in your Vertex AI Model Garden console.

<Note>
Claude via Vertex AI may also require the <code>anthropic[vertex]</code> package in
addition to <code>google-cloud-aiplatform</code>. If you encounter
<code>ModuleNotFoundError: No module named 'anthropic'</code>, ensure the image includes
both dependencies.
</Note>

### Troubleshooting

#### "No module named 'vertexai'" Error

Check warning on line 97 in openhands/usage/llms/google-llms.mdx

View check run for this annotation

Mintlify / Mintlify Validation (allhandsai) - vale-spellcheck

openhands/usage/llms/google-llms.mdx#L97

Did you really mean 'vertexai'?

If you encounter this error:
```
litellm.BadRequestError: Vertex_aiException BadRequestError - vertexai import failed
please run `pip install -U "google-cloud-aiplatform>=1.38"`.
Got error: No module named 'vertexai'
```

This means the agent-server image does not include the Vertex AI SDK. Enable the `vertex`
extra as described in [Vertex AI Dependencies](#vertex-ai-dependencies) above.
Loading