-
Notifications
You must be signed in to change notification settings - Fork 28
docs: document Vertex AI dependencies for the agent-server image #595
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from 3 commits
9181693
793b108
cf39b7c
93966ca
55286b0
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -3,7 +3,7 @@ | |
| description: OpenHands uses LiteLLM to make calls to Google's chat models. You can find their documentation on using Google as a provider -> [Gemini - Google AI Studio](https://docs.litellm.ai/docs/providers/gemini), [VertexAI - Google Cloud Platform](https://docs.litellm.ai/docs/providers/vertex) | ||
| --- | ||
|
|
||
| ## Gemini - Google AI Studio Configs | ||
|
|
||
| When running OpenHands, you'll need to set the following in the OpenHands UI through the Settings under the `LLM` tab: | ||
| - `LLM Provider` to `Gemini` | ||
|
|
@@ -12,7 +12,7 @@ | |
| (e.g. gemini/<model-name> like `gemini/gemini-2.0-flash`). | ||
| - `API Key` to your Gemini API key | ||
|
|
||
| ## VertexAI - Google Cloud Platform Configs | ||
|
|
||
| To use Vertex AI through Google Cloud Platform when running OpenHands, you'll need to set the following environment | ||
| variables using `-e` in the docker run command: | ||
|
|
@@ -28,3 +28,80 @@ | |
| - `LLM Model` to the model you will be using. | ||
| If the model is not in the list, enable `Advanced` options, and enter it in `Custom Model` | ||
| (e.g. vertex_ai/<model-name>). | ||
|
|
||
| ### Vertex AI Dependencies | ||
|
|
||
| The `vertex_ai/*` models (including Gemini and Claude via Vertex AI) require the | ||
| `google-cloud-aiplatform` package, which is **not included by default** in the published | ||
| agent-server image. How you enable it depends on your deployment: | ||
|
|
||
| <Note> | ||
| Unlike AWS Bedrock (whose `boto3` dependency is bundled by default), Vertex AI support is | ||
| opt-in. If you skip this step, you will see a <code>ModuleNotFoundError: No module named | ||
| 'vertexai'</code> error when the agent tries to call a Vertex AI model. | ||
| </Note> | ||
|
|
||
| #### Local / Non-Docker Install | ||
|
|
||
| Install the `vertex` extra in your Python environment: | ||
|
|
||
| ```bash | ||
| pip install "openhands-sdk[vertex]" | ||
|
rajshah4 marked this conversation as resolved.
|
||
| # or, with uv: | ||
| uv sync --extra vertex | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🟡 Suggestion:
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Fixed — replaced |
||
| ``` | ||
|
|
||
| #### Custom Agent-Server Image | ||
|
|
||
| Build the image with the `ENABLE_VERTEX` build flag: | ||
|
|
||
| ```bash | ||
| docker build \ | ||
| --build-arg ENABLE_VERTEX=1 \ | ||
|
rajshah4 marked this conversation as resolved.
|
||
| -t my-agent-server:vertex \ | ||
| -f openhands-agent-server/openhands/agent_server/docker/Dockerfile \ | ||
| . | ||
| ``` | ||
|
|
||
| Then point OpenHands at your custom image via the `AGENT_SERVER_IMAGE_REPOSITORY` and | ||
| `AGENT_SERVER_IMAGE_TAG` environment variables (see the | ||
| [Custom Sandbox Guide](/openhands/usage/advanced/custom-sandbox-guide) for details). | ||
|
|
||
| #### OpenHands Enterprise (Replicated / Kubernetes) | ||
|
|
||
| The default OHE installer Vertex path routes LLM calls through a LiteLLM proxy — the | ||
| agent-server uses a `litellm_proxy/...` model, and the proxy makes the actual Vertex call. | ||
| So the agent-server image does **not** need Vertex enabled for the default path; `ENABLE_VERTEX=1` | ||
| is only relevant if you customize OHE to bypass the proxy and call `vertex_ai/*` directly from | ||
| the agent-server. | ||
|
rajshah4 marked this conversation as resolved.
|
||
|
|
||
| ### Claude via Vertex AI | ||
|
|
||
| If you route Anthropic Claude through Google Vertex AI / Model Garden (rather than direct | ||
| Anthropic endpoints), use the `vertex_ai/` prefix with the Vertex-published model name, | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🔴 Critical: The Anthropic Python SDK does not expose a
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Pushing back on this one —
|
||
| which is date-stamped: | ||
|
|
||
| - `Custom Model`: `vertex_ai/claude-3-5-sonnet@20240620` | ||
|
|
||
| Use the exact model name shown in your Vertex AI Model Garden console. | ||
|
|
||
| <Note> | ||
| Claude via Vertex AI may also require the <code>anthropic[vertex]</code> package in | ||
| addition to <code>google-cloud-aiplatform</code>. If you encounter | ||
| <code>ModuleNotFoundError: No module named 'anthropic'</code>, ensure the image includes | ||
| both dependencies. | ||
| </Note> | ||
|
|
||
| ### Troubleshooting | ||
|
|
||
| #### "No module named 'vertexai'" Error | ||
|
|
||
| If you encounter this error: | ||
| ``` | ||
| litellm.BadRequestError: Vertex_aiException BadRequestError - vertexai import failed | ||
| please run `pip install -U "google-cloud-aiplatform>=1.38"`. | ||
| Got error: No module named 'vertexai' | ||
| ``` | ||
|
|
||
| This means the agent-server image does not include the Vertex AI SDK. Enable the `vertex` | ||
| extra as described in [Vertex AI Dependencies](#vertex-ai-dependencies) above. | ||
Uh oh!
There was an error while loading. Please reload this page.