Skip to content

Custom htmlLimitedBots UA not in built-in HTML_LIMITED_BOT_UA_RE breaks PPR resume and drops all page metadata #95406

Description

@gmen1057

Link to the code that reproduces this issue

https://github.com/gmen1057/next-html-limited-bots-ppr-repro

To Reproduce

  1. npm install && npm run build && PORT=3622 npm start
  2. Request a non-prerendered PPR page with a UA that matches the custom htmlLimitedBots regex from next.config.mjs but does not match the built-in HTML_LIMITED_BOT_UA_RE:
    curl -s -A 'MySpecialBot/1.0' http://localhost:3622/en/items/a | grep -c '<title>'   # → 0 (!)
    curl -s -A 'Mozilla/5.0 (compatible; GoogleOther)' http://localhost:3622/en/items/b | grep -c '<title>'   # → 0 (!)
    Server log gets, on every such request:
    ⨯ Error: Expected the resume to render <div> in this slot but instead it rendered <__next_metadata_boundary__>. The tree doesn't match so React will fallback to client rendering.
    
  3. Control cases — no error, metadata present:
    curl -s -A 'Twitterbot/1.0' http://localhost:3622/en/items/c | grep -c '<title>'   # → 1 (UA in BOTH custom regex and built-in list)
    curl -s http://localhost:3622/en/items/d | grep -c '<title>'                        # → 1 (regular browser, streamed metadata)

The repro page (app/[locale]/items/[slug]/page.js) is fully static with generateMetadata from params; generateStaticParams exists only on the [locale] level, so article slugs are served from the PPR fallback shell (build output ◐ Partial Prerender). Config:

const nextConfig = {
  cacheComponents: true,
  htmlLimitedBots: /GoogleOther|MySpecialBot|Twitterbot/i,
};

Current vs. Expected behavior

Current: for a UA that matches the configured htmlLimitedBots but not the built-in HTML_LIMITED_BOT_UA_RE (packages/next/src/shared/lib/router/utils/html-bots.ts):

  1. every request to a non-prerendered PPR route logs Expected the resume to render <div> in this slot but instead it rendered <__next_metadata_boundary__> and falls back to client rendering, and
  2. the response HTML contains no metadata at all — no <title>, no <meta name="description">, no canonical, neither in <head> nor streamed in <body>. Since html-limited bots are exactly the clients that don't execute JS, these crawlers never see any page metadata.

UAs that match both the custom regex and the built-in list (e.g. Twitterbot) work correctly, which suggests some code path in the PPR resume / metadata pipeline consults the built-in HTML_LIMITED_BOT_UA_RE constant directly instead of the resolved htmlLimitedBots config, so two layers disagree about the rendering mode.

Expected: a UA matching the configured htmlLimitedBots behaves exactly like a built-in html-limited bot: blocking metadata rendered into <head>, no resume mismatch, no server error.

Provide environment information

Operating System:
  Platform: linux
  Arch: x64
  Version: #27-Ubuntu SMP PREEMPT_DYNAMIC Thu Jun 18 19:13:49 UTC 2026
  Available memory (MB): 64291
  Available CPU cores: 16
Binaries:
  Node: 24.14.1
  npm: 11.11.0
  Yarn: N/A
  pnpm: N/A
Relevant Packages:
  next: 16.2.10 // Latest available version is detected (16.2.10).
  eslint-config-next: N/A
  react: 19.2.4
  react-dom: 19.2.4
  typescript: N/A
Next.js Config:
  output: N/A

Which area(s) are affected? (Select all that apply)

cacheComponents, Metadata, Partial Prerendering (PPR)

Which stage(s) are affected? (Select all that apply)

next start (local), Other (self-hosted production)

Additional context

Observed in production (Next 16.2.10, self-hosted) with htmlLimitedBots: /bot|crawler|spider|yandex|google|.../i: every GoogleOther, GPTBot, AhrefsBot, etc. hit on non-prerendered PPR pages logged the resume error (~50/day, Sentry) and received pages with no <title>/canonical/description, while Googlebot and YandexBot (covered by the built-in lists) were unaffected. Reproduced on 16.2.6 and 16.2.10.

Possibly related (same error message, different trigger): #86870

Metadata

Metadata

Assignees

No one assigned

    Labels

    MetadataRelated to Next.js' Metadata API.

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions