feat(secure): add NoiseHFS - post-quantum hybrid Noise (Noise_XXhfs_25519+ML-KEM-768)

[paschal533]

Draft: opened for early feedback rather than as something ready to merge - see "What's left" below.

What this adds

A post-quantum hybrid variant of the Noise handshake, NoiseHFS, implementing Noise_XXhfs_25519+ML-KEM-768_ChaChaPoly_SHA256 under protocol id /noise-mlkem768-hfs/0.1.0. It applies the Noise Hybrid Forward Secrecy extension (the e1/ekem1 tokens) to the classical XX pattern, so a hybrid-capable node keeps the same three-message handshake structure as plain /noise, just with an extra ML-KEM-768 key encapsulation mixed into the chaining key alongside the existing X25519 DH operations:

-> e, e1
<- e, ee, ekem1, s, es
-> s, se

The session stays confidential if X25519 is later broken by a quantum computer (the KEM covers that), and stays confidential if ML-KEM-768 turns out to have a classical weakness (the DH tokens cover that) - neither side's failure weakens the other's contribution to the final keys.

Full wire format, the reasoning behind picking raw ML-KEM-768 over a composite KEM like X-Wing, and the exact ekem1 token ordering (which matters - swapping two steps there silently produces divergent chaining keys) are written up in libp2p/protocols/secure/NOISE_HFS_SPEC.md.

Why raw ML-KEM-768 specifically

This wire format is meant to match Noise_XXhfs_25519+ML-KEM-768_ChaChaPoly_SHA256 as published in a research paper I've been working on ("Post-Quantum Cryptography Integration into the Noise Protocol"), which already covers TypeScript and Python implementations of the same profile. The design rationale is that the XXhfs pattern's three DH tokens already provide classical security on their own, so wrapping the KEM slot in a composite primitive like X-Wing would just be a redundant extra X25519 operation.

Where the ML-KEM-768 implementation comes from

Rather than pulling in a separate PQC library, libp2p/crypto/mlkem768.nim binds directly to the MLKEM768_* C API that BoringSSL already ships (crypto/mlkem/mlkem.cc), which nim-libp2p already compiles in through its boringssl dependency for TLS. It's the same ML-KEM-768 implementation Chrome uses. No new C dependency, no new nimble dependency.

How it's wired in

NoiseHFS reuses noise.nim's SymmetricState/CipherState/KeyPair and message framing as-is (I exported what was needed rather than duplicating it - see the "export primitives" commit, which is purely additive visibility changes plus one new optional parameter with a default that keeps the classical Noise class's behaviour unchanged). Only the e1/ekem1 token handling and the top-level connection encrypter are new.

It's registered as a second SecureProtocol variant alongside the existing one, with a withNoiseHFS() builder method, meant to be mounted next to plain Noise so a hybrid-capable node still falls back to classical /noise transparently against peers that don't support it - no coordination needed, multistream-select handles the negotiation.

What's verified

I didn't have a Nim toolchain locally, so I set one up from scratch (portable Nim + mingw-w64 + a from-source BoringSSL build) specifically so I could actually run this rather than just eyeball it.

Unit and integration tests (tests/libp2p/protocols/test_noisehfs.nim, 9/9 pass locally): MLKEM768 encapsulate/decapsulate round-trips, malformed-length rejection, wrong-key implicit rejection (FIPS 203 6.4 - this must not crash), key uniqueness, and a full two-node NoiseHFS handshake over a real TCP connection with peer identity verification, plus a peer-id-mismatch rejection case.

Live cross-language interop against py-libp2p, confirmed working. py-libp2p has its own implementation of this same profile (libp2p/py-libp2p, branch feat/pqc-noise-xxhfs, migrated to raw ML-KEM-768). I added interop/noise-pq/interop_dial.nim and ran it against py-libp2p's scripts/interop_listen_mlkem768.py:

# py-libp2p side (responder)
READY 9999
Connection from 127.0.0.1:56421
PEER 12D3KooWLitocTge1Lm2TmS3THHfrMWfV3d6UJZpPZNweL3c6CFD

# nim-libp2p side (initiator)
DIALING port 9999
HANDSHAKE_OK remotePeer=12D3KooWJGqi39m6ykyVhs8K1c1z8eeb4nFEqVxHLPopHdm8rV9h

Both sides completed the full three-message handshake - real X25519 DH, real ML-KEM-768 encapsulate/decapsulate, real ChaCha20-Poly1305 AEAD, real Ed25519 peer identity signature verification - with no changes needed to either implementation's wire format. The differing peer ids are expected: each side is printing the other side's freshly-generated identity. Full details, including a note on why I used Ed25519 identity keys rather than the crypto module's default ECDSA (py-libp2p's key-type deserializer doesn't cover ECDSA yet - an unrelated gap on their side, hit after the actual Noise/KEM handshake had already succeeded), are in interop/noise-pq/README.md.

What's left

• Rust and JavaScript interop haven't been run. Rust (test(noise): add interop tests, listener binary, and criterion benchmarks for ML-KEM-768 HFS royzah/rust-libp2p#1) would need a from-scratch rust-libp2p workspace build, which I didn't attempt this session; JavaScript (feat: add post-quantum hybrid Noise handshake ChainSafe/js-libp2p-noise#665) likewise. The same interop/noise-pq/ dial/listen scripts should work against both once those toolchains are set up - that's the natural next step.
• Not wired into libp2p.nim's public re-exports - kept the surface area small for this first pass; happy to add that plus any other builder ergonomics reviewers want if the general approach looks right.
• Peer identity is still classical (Ed25519/etc via the existing signature scheme) - this only addresses the key exchange, matching the scope of the underlying research. ML-DSA identity keys would be a separate, later piece of work.

Files

• libp2p/crypto/mlkem768.nim - raw ML-KEM-768 via BoringSSL
• libp2p/protocols/secure/noise.nim - exports the primitives NoiseHFS reuses (no behaviour change)
• libp2p/protocols/secure/noisehfs.nim - the NoiseHFS connection encrypter
• libp2p/protocols/secure/NOISE_HFS_SPEC.md - wire format, design rationale, interop status
• libp2p/builders.nim - SecureProtocol.NoiseHFS / withNoiseHFS()
• tests/libp2p/protocols/test_noisehfs.nim - unit + integration tests
• interop/noise-pq/ - standalone dial/listen scripts and the confirmed py-libp2p interop run

[AkshayaMani]

Cross-checked against the spec. Looks good. Left a couple of (non-blocking) suggestions inline.

[AkshayaMani]

Might be worth extending with a compile-time assert to guard against a future BoringSSL bump silently writing past the fixed-size buffers.

[AkshayaMani]

Suggestion:
sharedSecret is no longer needed. Binding as var and adding burnMem(sharedSecret) keeps defensive zeroization consistent (same pattern as burnMem(hs) below).

[AkshayaMani]

Same pattern as comment at line 112: encapRes.sharedSecret here would benefit from the same burnMem.

[paschal533]

Thanks for the review @AkshayaMani ! I addressed all three... struct size guard: added a _Static_assert against sizeof(struct MLKEM768_public_key)/private_key from the real BoringSSL header, so a future bump that changes the opaque layout fails the C build instead of silently writing past our buffers. Had to put it in its own file (mlkem768layout.nim) rather than inline... including <openssl/mlkem.h> in mlkem768.nim itself collided with the loosely-typed (byte*) importc declarations we already have there for the same BoringSSL functions, since gcc won't accept two different prototypes for the same symbol in one translation unit. burnMem on the decap shared secret: done, right after mixKey consumes it in the outbound handshake. burnMem on the encap shared secret: same, in the inbound handshake.
Verified all of this compiles and test_noisehfs.nim still passes (8/8).

[AkshayaMani]

LGTM on the fixes, thanks!