Bound POSIX SHM mapping by the shared object size

[yosuke-wolfssl]

Problem

posixTransportShm_Map() sized its mapping from the shared object's header,
which the peer writes:

size = sizeof(*header) + header->req_size + header->resp_size +
       header->dma_size;

Nothing checked those fields against the real size of the object. A header
declaring buffers or a DMA region larger than the object it describes steered
the transport past the end of the mapping, and the sum could wrap — a
dma_size of SIZE_MAX collapsed the total to a small value and was accepted
outright. The server applied no bound to its own req_size/resp_size either,
and a rejected configuration left the shared object linked for clients to trip
over.

Fix (port/posix/posix_transport_shm.c)

• fstat() now provides the mapping size; the header is never trusted for it.
• Header validated against the object before the full mapping: req_size
  and resp_size must each be a non-zero multiple of sizeof(whTransportMemCsr),
  the header-plus-buffers total must fit, and dma_size is checked against the
  remainder. The order short-circuits, so the subtraction cannot underflow.
• Only the declared footprint is mapped, so trailing padding in an oversized
  object is never reachable.
• Sizes snapshot once into ptshmMapping; callers no longer re-read the
  peer-writable header.
• posixTransportShm_CreateMap() applies the same buffer rule to the server
  config before the object exists, and unlinks on a failed map.

posix_transport_shm.h gains a threat-model note: this port is for testing,
debugging, and reference use, and is not a security boundary. The bound is
robustness against a corrupt object, not protection from a hostile peer.

Closes f-4223.

Tests

No test is included. The original 215-line test in test/wh_test_comm.c needed
a hand-built shared object that a well-behaved creator never produces, so it was
dropped per review.

Verification

• Legacy test/ and test-refactor/ suites build clean under
  -std=c90 -Werror -Wall -Wextra -Wpointer-arith.
• Functional: legacy suite exits 0; test-refactor 43 passed, 0 failed of 69.
• ASan + UBSan clean against a throwaway out-of-tree harness covering the 11
  accept/reject cases. Reverting the fix fails 9 of them, including
  dma_size = SIZE_MAX being accepted.

[Copilot]

Pull request overview

Note

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

This PR hardens the POSIX shared-memory transport against malicious or malformed SHM headers by bounding mappings to the actual SHM object size, snapshotting and validating peer-writable header fields once, and enforcing alignment/size rules for request/response buffers. It also adds regression tests that squat the SHM name and validate both client-side rejection and server-side fail-fast behavior.

Changes:

• Derive mmap length from fstat() object size and validate a single snapshot of header sizes before computing buffer offsets.
• Enforce “non-zero multiple of sizeof(whTransportMemCsr)” for request/response buffer sizes on both client and server creation paths.
• Add negative tests for malformed SHM headers and for server misconfiguration leaving no stale SHM objects behind.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 4 comments.

File	Description
test/wh_test_comm.c	Adds negative SHM layout/config tests by squatting the SHM name and writing malformed headers.
port/posix/posix_transport_shm.h	Documents the new req/resp buffer sizing/alignment rule and expected error behavior.
port/posix/posix_transport_shm.c	Maps SHM based on fstat() size, snapshots + validates header fields once, and uses validated sizes throughout.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[wolfSSL-Fenrir-bot]

Fenrir Automated Review — PR #479

Scan targets checked: wolfhsm-core-bugs, wolfhsm-src

No new issues found in the changed files. ✅

[rizlik]

The PR improves the POSIX SHM robustness.
What I see problematic is that it might hint that the POSIX SHM can provide a true isolation boundary between the client and the server.

Unfortunately there are plenty of other attack vectors for the client over POSIX SHM (both peers typically run with same UID, a malicious client can still ftruncate the file and SIGBUS the server, and an attacker can race to create a same-name object and interpose between client and server).

Moreover, I don't think POSIX SHM was ever meant to be a secure transport protocol but merely used for test, debug and as a reference.

I'm OK with the PR but this is the right moment to document the threat model explicitly in posix_transport_shm.h: same trust domain assumed; no peer authentication or isolation;

[rizlik]

NIT: superfluous

[yosuke-wolfssl]

Dropped, it's reverted to the original line.

[rizlik]

superfluous

[yosuke-wolfssl]

Same here, reverted

[yosuke-wolfssl]

Hi @rizlik ,
Thank you for reviewing.
I agreed with your comment on thread model — the framing was the real problem with the PR. POSIX SHM is a test/debug/reference transport and nothing here changes that; a peer at the same uid can still ftruncate() the object and fault the other side, or win the name race and interpose.

Added an explicit Threat model section to the top of posix_transport_shm.h stating: same trust domain assumed (same uid, peer contents are trusted input), no peer authentication (whoever opens the name is the peer, and the client unlinks on successful mapping so the name can be recreated by anyone afterwards), and no isolation or availability guarantee (ftruncate(), or simply never advancing the CSRs).

It closes by scoping what this PR actually buys: the header fields are validated against the real object size so a malformed or truncated object can't drive an out-of-bounds access in this process — robustness against a corrupt object, not protection against a hostile peer. Explicitly not a promotion to a trusted channel.

[rizlik]

@bigbrett over to you for merging