feat: Async HTTP outcalls refunds

[eichhorl]

Background

The new pay-as-you-go pricing for HTTP outcalls consists of four phases:

1. Base cost. This fee is charged for every request upfront.
2. Per-replica cost. The remaining cycles after charging the base cost are split evenly between the participating replicas. Each replica consumes some of their allowance as the HTTP request is processed. In the end, the amount of consumed cycles is gossiped as part of the response share.
3. Consensus cost. This fee is charged for including the aggregated HTTP response as part of a block.
4. Asynchronous refunds. Shares that did not contribute to the initial response (i.e. because they were received too late) are delivered asynchronously, such that their unused cycles can be reported and refunded, as well.

In #10897, we implemented the infrastructure required for phases 3 and 4, and partly implemented phase 3: After a response for a "pay-as-you-go" context is delivered, the initial refund is applied, and the context is moved into a new collection delivered_canister_http_contexts, where it waits for further asynchronous receipts to be delivered. After two minutes since the creation of the original context, the "delivered context" times out, refunding the allowances of all replicas that haven't reported a receipt yet, in full.

Proposed Changes

This PR implements the inclusion, validation and delivery of asynchronous refund receipts (phase 4):

Client (pool manager):

• Shares whose context was moved to delivered_canister_http_contexts are no longer purged
• As before, we will not make new requests to the HTTP adapter for contexts that were already moved to delivered_canister_http_contexts. No work was done, so the replica's allowance will be refunded in full on timeout.
• If an ongoing HTTP adapter request finishes, we continue to create and gossip a share, as long as the context is still part of delivered_canister_http_contexts. This is to report the work that was done.
• We continue to accept and validate shares whose context was moved to delivered_canister_http_contexts

Payload builder:

• We add a new field to the payload, holding the async receipts
• We will include async receipts for contexts that are part of delivered_canister_http_request_contexts and have not yet expired.
• We will only include an async receipt, if the issuing node has not refunded part of their allowance yet, i.e. as part of the initial response, or a previous async receipt. This is checked via the refund status of the context in the state, and all delivered async receipts in payloads above it.

Future Work

For flexible and non-replicated outcalls, the gossiped receipts still contain the full response bodies (just like normal shares for these replication types). However, these responses will never be included into a block, so we should avoid wasting bandwidth by only transmitting the share (like we do for fully replicated outcalls).

[Copilot]

Pull request overview

Implements asynchronous HTTP outcall refund receipts across consensus, validation, gossip, delivery, and metrics.

Changes:

• Adds async receipts to payload encoding and accounting.
• Retains, validates, and gossips late response shares.
• Adds receipt validation, tests, benchmarks, and metrics.

Reviewed changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
rs/types/types/src/batch/canister_http.rs	Adds async receipts to payload types.
rs/protobuf/src/gen/types/types.v1.rs	Adds generated receipt variant.
rs/protobuf/def/types/v1/canister_http.proto	Defines receipt wire encoding.
rs/interfaces/src/canister_http.rs	Adds validation errors.
rs/https_outcalls/consensus/src/test_utils.rs	Updates test payload construction.
rs/https_outcalls/consensus/src/pool_manager.rs	Retains and validates late shares.
rs/https_outcalls/consensus/src/payload_builder/utils.rs	Selects eligible receipts.
rs/https_outcalls/consensus/src/payload_builder/tests.rs	Tests receipt behavior.
rs/https_outcalls/consensus/src/payload_builder/parse.rs	Encodes and tracks receipts.
rs/https_outcalls/consensus/src/payload_builder.rs	Builds, validates, and delivers receipts.
rs/https_outcalls/consensus/src/gossip.rs	Requests shares for delivered contexts.
rs/https_outcalls/consensus/benches/payload_validation.rs	Updates validation benchmark.
rs/consensus/src/consensus/metrics.rs	Records delivered receipt metrics.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[zeropath-ai]

✅ No security or compliance issues detected. Reviewed everything up to eda9f49.

Security Overview

• 🔎 Scanned files: 13 changed file(s)
• 🔗 Scan Link: https://zeropath.com/app/repositories/f5e9d2bb-29a4-4b64-902a-391d65fd8b2d?scanId=ea6621dd-c496-455e-874d-13b5ee01a45f&codeScanTypes=PrScan&tab=issues

Detected Code Changes

Change Type	Relevant files
Enhancement	► rs/consensus/src/consensus/metrics.rs     Add canister_http_async_receipts_delivered
Enhancement	► rs/https_outcalls/consensus/benches/payload_validation.rs     Update imports and expectations to reflect PastPayloads usage
Enhancement	► rs/https_outcalls/consensus/src/gossip.rs     Add tests module for bouncer behavior and delivered context scenarios
Enhancement	► rs/https_outcalls/consensus/src/payload_builder.rs     Integrate PastPayloads, delivered/delivered contexts handling, and async receipts; extend payload construction logic; adjust imports and types for asynchronous receipts and membership validation; add new helper methods (request_committee, non_flexible_committee) and extended validation flow
Enhancement	► rs/https_outcalls/consensus/src/payload_builder.rs (parse.rs changes within same file path implied by diff)     Introduce PastPayloads struct and parse_past_payloads function; add callback_and_signer_of_share helper
Enhancement	► rs/https_outcalls/consensus/src/payload_builder/parse.rs     Extend parsing to support AsyncReceipt messages and PastPayloads parsing; adjust data structures to accommodate delivered_ids and refunded_nodes
Enhancement	► rs/https_outcalls/consensus/src/payload_builder/tests.rs     Update tests to account for async receipts and delivered context handling
Enhancement	► rs/https_outcalls/consensus/src/payload_builder/tests.rs (multiple test blocks)     Extend test setup to inject delivered contexts and async receipts; adapt tests to new PastPayloads semantics
Enhancement	► rs/https_outcalls/consensus/src/payload_builder/tests.rs (additional test cases for async receipts)