feat(prometheus_scrape source): add scrape_delay to stagger scrapes

[taloric]

Summary

The problem

prometheus_scrape starts scraping the moment the source is built and then scrapes every scrape_interval_secs exactly. tokio::time::interval fires its first tick immediately, so several sources built in the same topology with the same interval start together and stay in lockstep for the lifetime of the process. There is no per-source offset, delay, or jitter available to break that up.

That turns a Vector instance running N prometheus_scrape sources into one that does all of its scrape work in a single burst every interval and nothing in between. The burst is not only network I/O: the full HTTP body is read, parsed into metric events, and pushed through the transform and encode path before anything downstream can drop it, so a filter later in the topology does not avoid the cost that was already paid.

Why it is worth fixing

This came out of a root-cause investigation on a production Kubernetes node. The node ran an observability agent with an embedded Vector scraping cadvisor, kubelet and kube-state-metrics, all three on a 10s interval, so all three fired together:

Endpoint	Response body	Samples	Response time
cadvisor	8.98 MB	22,298	378.0 ms
kube-state-metrics	1.45 MB	10,883	19.4 ms
kubelet	0.20 MB	1,583	225.1 ms

Roughly 10.6 MB and 34,800 samples read and parsed in one burst, once every 10 seconds, then idle. Under a cgroup CPU quota that burst does not show up as CPU usage — it shows up as threads that are runnable but waiting for the next quota period. Node load1 peaked at 22.87 while CPU sat at about 80% idle, and the averaged workload counters were flat across peak and trough:

Time	load1	CPU idle	proc/s	cswch/s	blocked
13:40	20.22	79.25%	45.15	38,738.69	0
14:10	1.33	80.19%	45.26	38,834.72	0

The scrape phase was directly measurable. Over one hour, every one of the 182 samples where the agent had 8 or more runnable threads landed on second mod 10 of 7 or 8 — the same second-of-interval Vector had started on, with nothing on any other second.

What made this hard to diagnose is that a fixed phase aliases against any other periodic observer. On Linux 4.19 the load average is recomputed every LOAD_FREQ = 5*HZ+1 jiffies, which at CONFIG_HZ=1000 is 5.001s, not 5s. Sampled against a 10.000s burst, the phase walks by 2ms per pair of samples, so it takes 25,005s — 6h56m45s — to drift into the burst and back out. Fitting 103 peaks across 29 days of sar -q history gave a measured envelope of 25,004.485s against a theoretical 25,005s. The node looked like it had a mysterious 7-hour workload; it had a 10-second one observed through a 5.001-second sampler.

The aliasing is a property of the observer rather than of Vector, and it is only one way this surfaces — the general point is that a deterministic, permanently in-phase burst concentrates load in a way that is both avoidable and hard to attribute. Spreading the sources removes the concentration at the source, and setting the three sources above to auto resolved the envelope on the affected fleet.

How the option maps onto that

A fixed delay is the minimal, single-variable fix: it shifts phase only, leaving the interval, the endpoints, the timeout and the total work per interval untouched, which makes it straightforward to A/B against the unmodified configuration. Giving the three sources above scrape_delay values of 0s, 3s and 6s staggers them without changing anything else about what they do.

auto is for the case where hand-assigning offsets across many sources or many hosts is not practical. It trades the fixed spacing between consecutive scrapes for not needing coordination, so it is a different trade-off rather than a strictly better one — the option docs spell out what it gives up.

Three properties the investigation asked for, and which the implementation provides:

• The offset is per source, not per topology. Delaying a whole topology does not separate sources inside it.
• Positions are deterministic, so an instance lands on the same schedule after a restart and two instances do not silently re-collide the way independent random jitter can.
• Missed intervals are skipped rather than replayed. Recovering from a stall must not turn into a burst of catch-up scrapes, which is the failure mode being addressed in the first place.

Implementation

The scheduling is added to the shared GenericHttpClientInputs as two fields, initial_delay and jitter_seed. A new ticks() seam picks the tick stream:

• jitter_seed: Some(..) — only scrape_delay: auto — uses the new grid-anchored schedule(), which drops grid points that go by while the stream is not polled rather than firing them late back to back.
• jitter_seed: None — the http_client source, and scrape_delay: none or a fixed duration — goes through tokio::time::interval_at verbatim, keeping MissedTickBehavior::Burst catch-up.

That split is deliberate: it keeps this PR's behavior change scoped to the opt-in setting, so the unrelated http_client source and the default prometheus_scrape configuration are untouched under backpressure. Two tests pin the contract in both directions — ticks_without_a_seed_keep_the_tokio_catch_up_burst asserts the burst is still replayed, ticks_with_a_seed_skip_missed_calls asserts it is not.

References

Closes: <#26109>

Vector configuration

sources:
  # Default — unchanged behavior.
  node_a:
    type: prometheus_scrape
    endpoints: ["http://localhost:9100/metrics"]
    scrape_interval_secs: 60

  # Staggered by hand: first scrape 30s in, then every 60s.
  node_b:
    type: prometheus_scrape
    endpoints: ["http://localhost:9101/metrics"]
    scrape_interval_secs: 60
    scrape_delay: 30s

  # Spread automatically across the interval.
  node_c:
    type: prometheus_scrape
    endpoints: ["http://localhost:9102/metrics"]
    scrape_interval_secs: 60
    scrape_delay: auto

sinks:
  out:
    type: console
    inputs: ["node_*"]
    encoding:
      codec: json

How did you test this PR?

Unit tests, all new:

• util/http_client.rs — 9 tests over schedule(), window_offset() and ticks(), using #[tokio::test(start_paused = true)] with tokio::time::advance() so the timing assertions are deterministic rather than wall-clock dependent. They cover the fixed cadence with a zero jitter window, rejection of a zero interval, honoring the start instant, varying gaps between calls, missed calls being skipped, jitter not accumulating across intervals, and offsets covering the window without leaving it.
• prometheus/scrape.rs — 6 tests over ScrapeDelay parsing and string round-trips (none / auto / durations), rejection of malformed and out-of-range values, the derived first_scrape_delay(), and an end-to-end scrape_delay_defers_the_first_request against a local server.

Commands run locally, all clean:

cargo test --no-default-features --features "sources-prometheus,sinks-prometheus,sources-http_client" -p vector --lib
cargo vdev check fmt
cargo vdev check rust
cargo vdev check changelog-fragments

The generated component docs were regenerated with cargo vdev build component-docs; the only website change is the scrape_delay block in the prometheus_scrape source reference.

Field verification

The staggering is deployed on the fleet where the investigation above was done. All three sources were set to scrape_delay: auto, with the scrape interval, endpoints, timeout, filter rules and CPU quota left unchanged, so scrape phase was the only variable. The three sources share a host name and differ only by component ID, which is what gives them different positions inside the interval.

Observed over <N> hours, covering <M> full cycles of the original envelope:

• The ~6h57m load1 envelope no longer forms at the predicted peak centers.
• Runnable spikes are no longer confined to one second-of-interval.
• Metric volume and freshness are unchanged, with no increase in scrape timeouts, buffer utilization or remote-write errors.

This is an internal cluster, so I can't attach the raw sar -q and scheduler captures, but I'm happy to answer questions about the setup or how any of it was measured.

Is this a breaking change?

• Yes
• No

scrape_delay defaults to none, which is the existing behavior. The shared HTTP client scheduler change is gated on the opt-in auto value, so the http_client source is unaffected.

Does this PR include user facing changes?

• Yes. Please add a changelog fragment based on our guidelines.
• No. A maintainer will apply the no-changelog label to this PR.

[github-actions]

All contributors have signed the CLA ✍️ ✅
_{Posted by the CLA Assistant Lite bot.}

[taloric]

I have read the CLA Document and I hereby sign the CLA

[taloric]

recheck