From 099369bda7b0fcbfbcfc3b6c1acde80899509de3 Mon Sep 17 00:00:00 2001 From: Ben Lea <80100530+darkopia@users.noreply.github.com> Date: Fri, 28 Aug 2026 03:02:53 +0200 Subject: [PATCH 1/2] Add import rate and timestamp ordering guidance to migrations docs Adds two best practices to the historical migrations overview: keep DIY import throughput below a few thousand events per second, and batch events by time range (roughly one day) and import oldest first. Generated-By: PostHog Desktop Task-Id: 01a3e0db-9f18-40a1-b97c-cd050581624f --- contents/docs/migrate/index.mdx | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/contents/docs/migrate/index.mdx b/contents/docs/migrate/index.mdx index 42d1cfa6f37a..f101d49bc4c9 100644 --- a/contents/docs/migrate/index.mdx +++ b/contents/docs/migrate/index.mdx @@ -55,3 +55,15 @@ An example `cURL` implementation using the `batch` API endpoint looks like this: - Build resumability into your exports and imports, so you can just resume the process from the last successful point if any problems occur. For example, we use a cursor-based approach in our self-hosted migration tool. - To batch user updates, use the same request but with the `$identify` event. Same for groups and the `$group_identify` event. + +### Limit your import rate + +Add rate limiting to your import script and keep the total below a few thousand events per second across all workers and processes. A faster import does not land your data sooner. It fills the ingestion queue, which delays your own events and the events of other customers. If your import destabilizes ingestion, we can delay or drop your events to protect the pipeline. + +A [managed migration](/docs/migrate/managed-migrations) does this for you, so use one if your source is Amplitude, Mixpanel, or an S3 bucket. + +### Send events in timestamp order + +Group your events into batches that each cover a short time range, then import the batches from oldest to newest. One day per batch is a good default. + +A strict sort is not necessary. What matters is that a single batch does not mix events from across your full history. Scrambled timestamps make ClickHouse write many small data parts instead of a few large ones. This causes `too many parts` errors, which slow your import and can lose events. From 37211600881cc4fc8294f870fa1367eec8c320a7 Mon Sep 17 00:00:00 2001 From: Ben Lea <80100530+darkopia@users.noreply.github.com> Date: Fri, 28 Aug 2026 03:05:21 +0200 Subject: [PATCH 2/2] Drop too many parts error detail from ordering guidance Generated-By: PostHog Desktop Task-Id: 01a3e0db-9f18-40a1-b97c-cd050581624f --- contents/docs/migrate/index.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/contents/docs/migrate/index.mdx b/contents/docs/migrate/index.mdx index f101d49bc4c9..3a545a1923a6 100644 --- a/contents/docs/migrate/index.mdx +++ b/contents/docs/migrate/index.mdx @@ -66,4 +66,4 @@ A [managed migration](/docs/migrate/managed-migrations) does this for you, so us Group your events into batches that each cover a short time range, then import the batches from oldest to newest. One day per batch is a good default. -A strict sort is not necessary. What matters is that a single batch does not mix events from across your full history. Scrambled timestamps make ClickHouse write many small data parts instead of a few large ones. This causes `too many parts` errors, which slow your import and can lose events. +A strict sort is not necessary. What matters is that a single batch does not mix events from across your full history. Scrambled timestamps make ClickHouse write many small data parts instead of a few large ones.