@directus/gatsby-source-directus throws on building gatsby application, too many requests.

[limbofee]

Describe the Bug

I am trying to build my gatsby application (running v4). My app uses @directus/gatsby-source-directus as a gatsby plugin to access CMS stored data for my app. When running both the development and the production build (gatsby develop and gatsby build), I encounter a rate limiting error message, thrown at the line the directus query is used. The app throws on trying to create a production build (gatsby build). The development build also shows this error at buildtime, but the app runs after the errors appear in the console.

My app also uses image processing and transforming plugins like:

• gatsby-plugin-image
• gatsby-plugin-sharp
• gatsby-transformer-rehype
• gatsby-transformer-sharp

To Reproduce

Run a gatsby application that queries a larger amount of documents (over 200), documents that each contain their own set of images.

You can run the application by using:

• gatsby develop
  or
• gatsby build

Errors Shown

Error: Too many requests, retry after 278ms.

• sdk.cjs.js:696 Transport.request
  [website]/[@directus]/sdk/dist/sdk.cjs.js:696:23

• runMicrotasks

• task_queues:96 processTicksAndRejections
  node:internal/process/task_queues:96:5

• sdk.cjs.js:710 Transport.get
  [website]/[@directus]/sdk/dist/sdk.cjs.js:710:16

• sdk.cjs.js:37 FilesHandler.readOne
  [website]/[@directus]/sdk/dist/sdk.cjs.js:37:26

• gatsby-node.js:88 resolve
  [website]/[@directus]/gatsby-source-directus/gatsby-node.js:88:36

• async Promise.all

• async Promise.all

• async Promise.all

• async Promise.all

• async Promise.all

• graphql-runner.ts:220 GraphQLRunner.query
  [website]/[gatsby]/src/query/graphql-runner.ts:220:14

• create-pages.ts:36 wrappedGraphQL
  [website]/[gatsby]/src/services/create-pages.ts:36:25

• gatsby-node.js:109 Object.exports.createPages
  /Users/limbo/Documents/BLOCKTALK/website/gatsby-node.js:109:20

• api-runner-node.js:487 runAPI
  [website]/[gatsby]/src/utils/api-runner-node.js:487:16

What version of Directus are you using?

latest

What version of Node.js are you using?

16.6.0

What database are you using?

What browser are you using?

Chrome

How are you deploying Directus?

[rijkvanzanten]

Not a gatsby expert myself, but you either have to increase the rate limit allowed points, or slow down how many requests gatsby is allowed to make in the given rate limiter configuration. If I'm not mistaken, Gatsby's data fetching automatically handles rate limit errors. I vaguely remember hearing some other people running into rate limit issues with the gatsby-plugin-image package

[limbofee]

Not a gatsby expert myself, but you either have to increase the rate limit allowed points, or slow down how many requests gatsby is allowed to make in the given rate limiter configuration. If I'm not mistaken, Gatsby's data fetching automatically handles rate limit errors. I vaguely remember hearing some other people running into rate limit issues with the gatsby-plugin-image package

I dug in deeper and found out that no external gatsby-plugin package was faulty. On the gatsby side of things everything seems in place. I'm not a gatsby expert either, but by looking at the stack of the error, each document is queried/processed in a Promise.all() sequence, which means that this is meant to be done in parallel. By doing that, you instantiate a fileResolver for each directus document you fetch (every document points to a file, an image), then each fileResolver calls its resolve() method, which will later use requests to fetch files. The custom request method used to create the request will later throw a TransportError (because the endpoint is called over and over again with little to no delay between requests) error that you can see at the top of the error stack in the issue.

This happens only if you have multiple files to be queried. I've tried to execute the query (and nothing else) in a test-repo, and everything is fine if you don't fetch files. If you add file queries, then it throws the error again.

[BasitEllahi]

Hey, I got the same issue, when im building the app with gatsby build. I get different errors from the graphql api

[rijkvanzanten]

Linear: ENG-61

[freekrai]

hi @limbofee @BasitEllahi can you try with the latest version of the gatsby-source-directus plugin? there were some adjustments done with https://github.com/directus/gatsby-source-directus/releases/tag/v9.14.6 to help improve image handling and we've done a few other fixes since then as well so this may be fixed or at least improved for you as of latest release which is currently https://github.com/directus/gatsby-source-directus/releases/tag/v9.15.0