-
Notifications
You must be signed in to change notification settings - Fork 2.5k
Document explicit prompt caching support in ChatOpenAI #4829
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
|
|
@@ -1887,6 +1887,79 @@ response1 = llm.invoke(messages) | |||||
| response2 = llm.invoke(messages, prompt_cache_key="override-cache-v1") | ||||||
| ``` | ||||||
|
|
||||||
| ### Explicit caching with breakpoints (Responses API) | ||||||
|
|
||||||
| <Note> | ||||||
| Requires `langchain-openai>=1.3.5` and `openai>=2.45.0`. Only supported when using the [Responses API](/oss/python/integrations/chat/openai#responses-api). | ||||||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
need to verify this claim |
||||||
| </Note> | ||||||
|
|
||||||
| OpenAI's Responses API supports [explicit prompt-cache breakpoints](https://developers.openai.com/api/docs/guides/prompt-caching?prompt-cache-api=responses#prompt-cache-breakpoints), which let you designate specific content blocks as cache boundaries. This gives you fine-grained control over which parts of a prompt are cached, rather than relying solely on automatic prefix caching. | ||||||
|
|
||||||
| To mark a content block as a cache breakpoint, add `"prompt_cache_breakpoint": True` to the block: | ||||||
|
|
||||||
| ```python | ||||||
| from langchain_openai import ChatOpenAI | ||||||
|
|
||||||
| llm = ChatOpenAI(model="gpt-4.1", use_responses_api=True) | ||||||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
need to verify model string don't think 4.1 supports |
||||||
|
|
||||||
| messages = [ | ||||||
| { | ||||||
| "role": "system", | ||||||
| "content": [ | ||||||
| { | ||||||
| "type": "text", | ||||||
| "text": "You are a helpful assistant with access to a large knowledge base.", | ||||||
| "prompt_cache_breakpoint": True, # [!code highlight] | ||||||
| } | ||||||
| ], | ||||||
| }, | ||||||
| {"role": "user", "content": "Summarize the key points."}, | ||||||
| ] | ||||||
|
|
||||||
| response = llm.invoke(messages) | ||||||
| ``` | ||||||
|
|
||||||
| Breakpoints are supported on text, image, and file content blocks. You can also nest `prompt_cache_breakpoint` inside an `extras` dict if you prefer to keep the LangChain content block structure clean: | ||||||
|
|
||||||
| ```python | ||||||
| content_block = { | ||||||
| "type": "text", | ||||||
| "text": "Long system prompt...", | ||||||
| "extras": {"prompt_cache_breakpoint": True}, | ||||||
| } | ||||||
| ``` | ||||||
|
|
||||||
| ### Request-level cache options | ||||||
|
|
||||||
| <Note> | ||||||
| Requires `langchain-openai>=1.3.5`. | ||||||
| </Note> | ||||||
|
|
||||||
| You can pass request-level prompt cache options to the Responses API using the `prompt_cache_options` parameter on the model: | ||||||
|
|
||||||
| ```python | ||||||
| llm = ChatOpenAI( | ||||||
| model="gpt-4.1", | ||||||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. change |
||||||
| use_responses_api=True, | ||||||
| prompt_cache_options={"cache_type": "ephemeral"}, | ||||||
| ) | ||||||
| ``` | ||||||
|
|
||||||
| ### Cache write tokens | ||||||
|
|
||||||
| When OpenAI writes new content to the prompt cache, it reports `cache_write_tokens` in the response. `ChatOpenAI` surfaces this as `cache_creation` in `input_token_details`: | ||||||
|
|
||||||
| ```python | ||||||
| response = llm.invoke(messages) | ||||||
|
|
||||||
| cache_read = response.usage_metadata["input_token_details"].get("cache_read") | ||||||
| cache_creation = response.usage_metadata["input_token_details"].get("cache_creation") | ||||||
| print(f"Cache read tokens: {cache_read}") | ||||||
| print(f"Cache creation tokens: {cache_creation}") | ||||||
| ``` | ||||||
|
|
||||||
| On the `"priority"` and `"flex"` service tiers, these keys are prefixed with the tier name — for example, `"priority_cache_read"` and `"priority_cache_creation"`. | ||||||
|
|
||||||
| --- | ||||||
|
|
||||||
| ## Flex processing | ||||||
|
|
||||||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this is done transitively