You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
docs(api): document MIME and PDF processor options (#93)
* docs(mcp): add OAuth login steps
* docs(mcp): rebuild MCP section from the hosted server as source of truth
Verified every claim against the live server at mcp.scrapegraphai.com and
the apps/mcp implementation.
- remove the "Star us on GitHub" card
- drop the local stdio path: `pip install scrapegraph-mcp` resolves to PyPI
1.0.1, which exposes 8 pre-v3 tools (markdownify, smartscraper, ...) and
does not match the v2 API this section documents
- drop the Smithery page for the same reason, with a redirect to the
introduction and inbound links repointed
- document the transport honestly: Streamable HTTP only, SSE unsupported,
60s per tool call
- document all three accepted API key headers, plus OAuth 2.1 with PKCE
S256 and the mcp:use scope
- correct the tool table to /api/* paths with real defaults and limits
- replace the migration note's dead v3.0.0 release link with a legacy
name mapping table
- collapse the four client pages onto one shape: configure, authenticate,
API key alternative, verify with `credits`
- bump the Claude Desktop mcp-remote pin from 0.1.25 to 0.1.38
* docs(search): document contentTypes filter
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(search): document PDF options
* docs(search): explain default MIME allowlist
* docs(api): align MIME and PDF options
* docs(api): clarify default PDF page cap
* docs(pdf): clarify per-page credit cost
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Copy file name to clipboardExpand all lines: api-reference/endpoint/extract.mdx
+11Lines changed: 11 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -37,6 +37,17 @@ Exactly one of `url`, `html`, or `markdown` must be supplied as the source.
37
37
HTML pre-processing mode: `"normal"`, `"reader"`, or `"prune"`.
38
38
</ParamField>
39
39
40
+
<ParamFieldbody="allowedTypes"type="array">
41
+
Optional non-empty MIME allowlist for URL input. Omit it to allow every supported type; `"all"` and `"*"` are not accepted.
42
+
</ParamField>
43
+
44
+
<ParamFieldbody="processors"type="array">
45
+
Optional processing configuration for URL input. Omit this field to use the 25-page PDF cap. You
46
+
may also send `{"type":"pdf"}` and omit `maxPages`; it defaults to `25`. Set `maxPages` only to
47
+
override the default, using `1`–`500`, or `-1` for no page limit. PDF processing costs 1 credit
48
+
per page actually processed.
49
+
</ParamField>
50
+
40
51
<ParamFieldbody="fetchConfig"type="object">
41
52
Fetch-time options. See the [Scrape endpoint](/api-reference/endpoint/scrape#request-body) for the full field list (`mode`, `stealth`, `headers`, `cookies`, `scrolls`, `wait`, `timeout`, `country`). Ignored when `html` or `markdown` is supplied.
Copy file name to clipboardExpand all lines: api-reference/endpoint/search.mdx
+14-1Lines changed: 14 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -39,6 +39,17 @@ Replaces the v1 `searchscraper` endpoint. Returns the top pages with their conte
39
39
ISO 3166-1 alpha-2 country code for localized results (e.g. `"us"`, `"it"`).
40
40
</ParamField>
41
41
42
+
<ParamFieldbody="allowedTypes"type="array">
43
+
Optional MIME allowlist for fetched results. Omit this field to allow every supported type, including `application/pdf`. There is no `"all"` or `"*"` keyword. When provided, it must be a non-empty array of exact supported MIME types, such as `["text/html", "application/pdf"]`. Rejected types are reported as failed pages and do not appear in `results`.
44
+
</ParamField>
45
+
46
+
<ParamFieldbody="processors"type="array">
47
+
Optional processing configuration. Omit this field to use the 25-page PDF cap. You may also send
48
+
`{"type":"pdf"}` and omit `maxPages`; it defaults to `25`. Set `maxPages` only to override the
49
+
default, using `1`–`500`, or `-1` for no page limit. PDF processing costs 1 credit per page
50
+
actually processed. This does not change `allowedTypes`.
51
+
</ParamField>
52
+
42
53
<ParamFieldbody="fetchConfig"type="object">
43
54
Fetch-time options applied when crawling each result. See the [Scrape endpoint](/api-reference/endpoint/scrape#request-body) for the full field list.
44
55
</ParamField>
@@ -51,7 +62,9 @@ curl -X POST https://v2-api.scrapegraphai.com/api/search \
| `locationGeoCode` | `string` | No | Two-letter country code (e.g. `"us"`) |
243
+
| `allowedTypes` | `string[]` | No | Non-empty MIME allowlist. Omit it for all supported types; `"all"` and `"*"` are not accepted |
244
+
| `processors` | `object[]` | No | Omit for the 25-page PDF cap. `{ type:"pdf" }` also defaults to 25; set `maxPages` only to override it, or use `-1` for unlimited. PDF processing costs 1 credit per page actually processed |
243
245
| `fetchConfig` | `FetchConfig` | No | Fetch configuration |
| `allowedTypes` | `string[]` | No | Non-empty MIME allowlist; omit it to allow every supported type |
307
+
| `processors` | `object[]` | No | Omit for the 25-page PDF cap. `{ type:"pdf" }` also defaults to 25; set `maxPages` only to override it, or use `-1` for unlimited. PDF processing costs 1 credit per page actually processed |
305
308
| `fetchConfig` | `FetchConfig` | No | Fetch configuration |
Copy file name to clipboardExpand all lines: sdks/python.mdx
+4-1Lines changed: 4 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -270,6 +270,8 @@ if res.status == "success":
270
270
|`schema`|`dict`| No | JSON Schema for structured output. Pass a Pydantic model's `model_json_schema()` to reuse a `BaseModel`. |
271
271
|`location_geo_code`|`str`| No | Two-letter country code (e.g. `"us"`, `"it"`) |
272
272
|`time_range`|`str`| No |`"past_hour"`, `"past_24_hours"`, `"past_week"`, `"past_month"`, `"past_year"`|
273
+
|`allowed_types`|`list[str]`| No | Non-empty MIME allowlist. Omit it for all supported types; `"all"` and `"*"` are not accepted |
274
+
|`processors`|`list[PdfProcessor]`| No | Omit for the 25-page PDF cap. `PdfProcessor()` also defaults to 25; set `max_pages` only to override it, or use `-1` for unlimited. PDF processing costs 1 credit per page actually processed |
273
275
|`fetch_config`|`FetchConfig`| No | Fetch configuration |
274
276
275
277
### Crawl
@@ -316,7 +318,8 @@ sgai.crawl.delete(crawl_id)
316
318
|`allow_external`|`bool`| No | Default `False`|
317
319
|`include_patterns`|`list[str]`| No | URL glob patterns to include |
318
320
|`exclude_patterns`|`list[str]`| No | URL glob patterns to exclude |
319
-
|`content_types`|`list[str]`| No | Allowed response content types |
321
+
|`allowed_types`|`list[str]`| No | Non-empty MIME allowlist; omit it to allow every supported type |
322
+
|`processors`|`list[PdfProcessor]`| No | Omit for the 25-page PDF cap. `PdfProcessor()` also defaults to 25; set `max_pages` only to override it, or use `-1` for unlimited. PDF processing costs 1 credit per page actually processed |
320
323
|`fetch_config`|`FetchConfig`| No | Fetch configuration |
Copy file name to clipboardExpand all lines: services/crawl.mdx
+2-1Lines changed: 2 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -133,7 +133,8 @@ curl -X GET "https://v2-api.scrapegraphai.com/api/crawl/:id/pages?limit=50&curso
133
133
|`allowExternal` / `allow_external`| bool | No | Whether to follow links to other domains. Default `false` (same-origin only). |
134
134
|`includePatterns` / `include_patterns`| array | No | URL patterns to include (e.g. `["/blog/*"]`). |
135
135
|`excludePatterns` / `exclude_patterns`| array | No | URL patterns to exclude (e.g. `["/admin/*"]`). |
136
-
|`contentTypes` / `content_types`| array | No | Limit crawled pages to these MIME types, e.g. `["text/html", "application/pdf"]`. |
136
+
|`allowedTypes` / `allowed_types`| array | No | Non-empty MIME allowlist. Omit it for all supported types; `"all"` and `"*"` are invalid. |
137
+
|`processors`| array | No | Omit for the 25-page PDF cap. Send it only to override the cap; `maxPages` also defaults to 25 when omitted, and `-1` means unlimited. PDF processing costs 1 credit per page actually processed. |
137
138
|`fetchConfig` / `fetch_config`| object | No | Fetch options (see [Scrape · FetchConfig](/services/scrape#fetchconfig)). |
Copy file name to clipboardExpand all lines: services/extract.mdx
+2Lines changed: 2 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -79,6 +79,8 @@ curl -X POST https://v2-api.scrapegraphai.com/api/extract \
79
79
| `prompt` | string | Yes | Natural-language description of what to extract. |
80
80
| `schema` | object | No | JSON schema describing the desired output shape. In Python you can pass a Pydantic model via `MyModel.model_json_schema()`. |
81
81
| `mode` | string | No | HTML processing mode: `"normal"`, `"reader"`, `"prune"`. |
82
+
| `allowedTypes` / `allowed_types` | array | No | Non-empty MIME allowlist for URL input. Omit it for all supported types; `"all"` and `"*"` are invalid. |
83
+
| `processors` | array | No | Omit for the 25-page PDF cap. Send it only to override the cap; `maxPages` also defaults to 25 when omitted, and `-1` means unlimited. PDF processing costs 1 credit per page actually processed. |
82
84
| `fetchConfig` / `fetch_config` | object | No | Fetch options (see [Scrape · FetchConfig](/services/scrape#fetchconfig)). |
Copy file name to clipboardExpand all lines: services/scrape.mdx
+2Lines changed: 2 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -88,6 +88,8 @@ curl -X POST https://v2-api.scrapegraphai.com/api/scrape \
88
88
| `url` | string | Yes | The URL of the webpage to scrape. |
89
89
| `formats` | array | Yes | One or more output formats (see [Formats](#output-formats)). |
90
90
| `contentType` | string | No | Override auto-detected content type (e.g. `"text/html"`, `"application/pdf"`). |
91
+
| `allowedTypes` / `allowed_types` | array | No | Non-empty MIME allowlist. Omit it for all supported types; `"all"` and `"*"` are invalid. |
92
+
| `processors` | array | No | Omit for the 25-page PDF cap. Send it only to override the cap; `maxPages` also defaults to 25 when omitted, and `-1` means unlimited. PDF processing costs 1 credit per page actually processed. |
Copy file name to clipboardExpand all lines: services/search.mdx
+58Lines changed: 58 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -86,12 +86,70 @@ curl -X POST https://v2-api.scrapegraphai.com/api/search \
86
86
| `mode` | string | No | HTML processing mode: `"normal"`, `"reader"`, or `"prune"`. Default: `"prune"` (different from Scrape/Extract, which default to `"normal"`). |
| `allowedTypes` / `allowed_types` | array | No | Optional non-empty MIME allowlist. Omit it to allow every supported type, including PDF. |
90
+
| `processors` | array | No | Omit for the 25-page PDF cap. Send it only to override the cap; `maxPages` also defaults to 25 when omitted, and `-1` means unlimited. PDF processing costs 1 credit per page actually processed. |
89
91
| `fetchConfig` / `fetch_config` | object | No | Fetch options (see [Scrape · FetchConfig](/services/scrape#fetchconfig)). |
90
92
91
93
<Note>
92
94
Get your API key from the [dashboard](https://scrapegraphai.com/dashboard).
93
95
</Note>
94
96
97
+
## Content Types and PDF Processing
98
+
99
+
By default Search accepts every supported content type, including PDFs. Use `allowedTypes` to
100
+
restrict which MIME types may be fetched, and configure the PDF page cap independently with
101
+
`processors`. Excluding `application/pdf` rejects PDF results even when a PDF processor is present.
102
+
103
+
<Note>
104
+
There is no special `"all"` or `"*"` value. To allow every supported type, omit `allowedTypes`
105
+
entirely. The playground follows this rule in generated snippets: when “All types” is selected, it
106
+
does not print the full MIME list. If `allowedTypes` is present, it must contain at least one exact
0 commit comments