dd: align the I/O buffer to the page size for O_DIRECT

[relative23]

Fixes #12085.

dd iflag=direct currently fails immediately with dd: IO error: Invalid input when reading from any block device whose queue/dma_alignment is stricter than the allocator's default alignment — which is most of them (sd, loop, virtio_blk, nbd, zram, … report 511; NVMe reports 3, which is why the bug does not reproduce there). Downstream impact: LXD has started working around this (canonical/lxd#18429).

Root cause

The copy loop's I/O buffer is a plain Vec<u8>; the allocator only guarantees small alignment for it (16 bytes with glibc):

openat(AT_FDCWD, "/dev/loop0", O_RDONLY|O_DIRECT|O_CLOEXEC) = 3
read(3, 0x7cd224228010, 1048576)        = -1 EINVAL (Invalid argument)

The write path has the same misalignment, but it is masked by handle_o_direct_write(): every full-block write first fails with EINVAL, is then retried buffered with O_DIRECT stripped via fcntl, and the flag is restored afterwards — three extra syscalls per block, while oflag=direct silently does no direct I/O at all:

write(4, ..., 1048576) = -1 EINVAL (Invalid argument)
write(4, ..., 1048576) = 1048576
write(4, ..., 1048576) = -1 EINVAL (Invalid argument)
write(4, ..., 1048576) = 1048576

Fix

Allocate the copy-loop buffer page-aligned, matching GNU dd (ibuf = ptr_align (…, page_size)). AlignedBuffer allocates exactly bsize bytes at the runtime page size via std::alloc::alloc_zeroed(Layout) — no memory overhead over the previous Vec<u8> of the same size, and the alignment covers the strictest DMA alignment Linux block drivers advertise in practice (O_DIRECT can additionally constrain I/O length and file offset, which follow from the chosen block sizes rather than from the buffer). No new dependencies.

The unsafe surface consists of four small, documented expressions covering three responsibilities — allocation, deallocation, and shared/mutable slice construction; the pointer is owned by AlignedBuffer alone, freed with the same Layout it was allocated with, and never handed to Vec or any other allocator-aware container — which is precisely what made the earlier memalign + Vec::from_raw_parts attempt (#9104) unsound. The unit tests pass under Miri.

The copy loop reads into a prefix of that fixed-size buffer through a plain &mut [u8] taken once outside the loop, adopting the slice-based read-path structure of #12143: fill_consecutive/fill_blocks report the number of valid bytes instead of truncating the buffer, and conv=block/unblock output goes to a separate scratch Vec, since those conversions can change the byte count. This keeps the hot loop free of per-iteration buffer bookkeeping (see #11544).

With this change, full-block oflag=direct writes also no longer take the EINVAL fallback path; the fallback keeps handling the final partial block, as GNU does. As a drive-by, the fill_blocks padding now uses slice::fill instead of allocating a padding Vec and splicing it in.

Relation to existing PRs

I only found these after opening this PR — apologies for the overlap:

• dd: page-align read buffer #12143 (chrboe) addresses the same read-path bug with a #[repr(align(4096))] page approach. This PR adopts its slice-based read-path structure (no per-iteration resize/truncate, separate conv scratch); compared to it, the allocation here uses the runtime page size instead of a compile-time 4096 (16K/64K-page systems), allocates exactly bsize bytes (no rounding to page multiples), restores true direct I/O on the write path, and adds unit and integration tests.
• dd: optimize O_DIRECT buffer alignment to reduce syscall overhead #9104 also touches this area, but relies on libc::memalign + Vec::from_raw_parts, which hands memory not obtained through Vec back to Vec and violates the allocator contract (UB, e.g. when building with a non-libc global allocator). Here the raw allocation never crosses into allocator-aware containers.

Related follow-up

Writing an additional short-read regression test for this area surfaced a pre-existing data-corruption bug in the shared read path (#13458, fix in #13459). It reproduces identically on main and is orthogonal to this PR; the refactor here inherits that behavior unchanged (only the visible filler byte changes from 0xDD to 0x00, since the aligned buffer is zero-initialized). The textual overlap in fill_consecutive is trivial — whichever PR lands second gets rebased.

Testing

• Unit tests for AlignedBuffer (alignment across sizes and alignments for empty and non-empty buffers, zero-initialization, pinned error kinds for the rejection paths); they pass under Miri (run standalone — the uu_dd test binary itself cannot run under Miri because uucore's startup code calls sigaction).
• Direct-I/O integration tests gated by an independent O_DIRECT capability probe (open with O_DIRECT + one page-aligned posix_memalign transfer): read and write support are probed separately, and only a failing open reports unsupported, for the errnos that mean the filesystem has no direct I/O at all (EINVAL/EOPNOTSUPP). Once the descriptor exists, an EINVAL from the aligned transfer means the filesystem demands stricter alignment than a page — a finding worth surfacing rather than a reason to skip — so that, a short transfer, and every other unexpected outcome panic instead. Where the probe reports unsupported the tests skip and say so; once it succeeds the dd invocations are required to succeed, so a failure can no longer be excused as a filesystem limitation. iflag=direct (across 4K/64K/512K block sizes), oflag=direct, and the combined round-trip are covered separately, over position-dependent data whose every aligned 8-byte chunk is a distinct SplitMix64 value, so swapped, duplicated, or stale blocks cannot cancel out.
• A dedicated oflag=direct test proves the write stays direct rather than only successful: it traces the open and fcntl under strace and asserts the output file was opened with O_DIRECT and that no F_SETFL followed on that descriptor while it was open (the scan is bounded by the matching close(), so a later reuse of the descriptor number cannot produce a false positive), because the buffered-retry fallback (handle_o_direct_write) is the only path that toggles O_DIRECT after open() — a plain success check cannot tell the two apart, since the fallback also produces correct output. Confirmed to discriminate on a strict-alignment loop device (dma_alignment=511): the aligned buffer opens with O_DIRECT and issues zero F_SETFL, while the previous misaligned Vec triggers the fallback (four F_SETFL toggles on the output fd for two blocks). The test skips cleanly when strace is absent or cannot ptrace.
• Ran the whole direct-I/O group on three filesystems, all of which support O_DIRECT on this kernel (7.0; tmpfs has since 6.1), so the tests really ran everywhere rather than skipping: tmpfs, ext4 on NVMe (dma_alignment=3), and an ext4 loop device (dma_alignment=511, where the strace test is the discriminating one, run 10× to rule out timing flakiness). The skip path was exercised separately by hiding strace from PATH.
• Also built and ran the direct-I/O tests with --no-default-features --features dd, and ran clippy with -D warnings over that minimal build, dd,printf, uu_dd, and the default feature set.
• Verified against the reproducer from dd iflag=direct reads fail with "IO error: Invalid input" on devices with strict dma_alignment #12085 on Ubuntu 26.04 (kernel 7.0, loop device with dma_alignment=511): before dd: IO error: Invalid input, after the read succeeds; strace shows zero EINVAL and no fcntl toggling for full-block direct writes.
• cargo test -p uu_dd (95 tests) and the dd integration suite (130 tests) pass; outputs are byte-identical to main across an 18-configuration sweep (odd ibs/obs, swab, sync, block/unblock, count_bytes, pipes).
• Performance (cachegrind instruction counts, current head vs current main, same toolchain): all nine dd benchmarks are at or below the Vec<u8> baseline — dd_copy_1m_blocks −11.4%, dd_copy_partial −2.7%, dd_copy_4k_blocks −2.5%, dd_copy_8k_blocks −2.4%, dd_copy_with_skip/with_seek −1.9%, dd_copy_64k_blocks −1.6%, dd_copy_default −0.9%, dd_copy_separate_blocks ±0%. Peak heap (DHAT) is identical to main, since AlignedBuffer allocates exactly bsize bytes.

[relative23]

Updated the description to properly credit #12143, which I had missed when opening this (my issue search didn't match its title). See the "Relation to existing PRs" section.

[codspeed-hq]

Merging this PR will not alter performance

✅ 339 untouched benchmarks
⏩ 46 skipped benchmarks¹

---

_{Comparing relative23:dd-odirect-buffer-alignment (95d9fe9) with main (be9fb63)}

Footnotes

1. 46 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

[github-actions]

GNU testsuite comparison:

Skipping an intermittent issue tests/date/date-locale-hour (passes in this run but fails in the 'main' branch)
Note: The gnu test tests/tail/pipe-f is now being skipped but was previously passing.
Congrats! The gnu test tests/seq/seq-epipe is now passing!

[relative23]

Thanks for the report — breakdown of the two kinds of deltas:

Memory (all six: exactly +4.0 KB): this is the intentional over-allocation by one page that provides the alignment guarantee (capacity + page_size, the safe-code equivalent of GNU dd's ptr_align(ibuf, page_size)), so it's a constant +4 KiB regardless of block size, not a leak or scaling issue.

Simulation (−4…−5%): per-iteration bookkeeping of the new buffer wrapper (capacity check + slice re-borrow), which adds up at bs=512/bs=4K where the copy loop runs tens of thousands of iterations. Addressed in 9eecf33 with a steady-state fast path in resize plus inlining. Local divan wall-time medians (same machine, 100 samples), main vs. this branch after the fix:

bench	main	this PR
dd_copy_default	23.8 ms	23.0 ms
dd_copy_8k_blocks	6.68 ms	6.39 ms
dd_copy_64k_blocks	12.4 ms	12.0 ms
dd_copy_partial	749 µs	730 µs

Worth keeping in mind when weighing the remaining instruction-count delta: these benchmarks don't use direct I/O, and CodSpeed doesn't measure syscall cost — while the bug this PR fixes currently makes every full-block oflag=direct write go through a failed write + two fcntl calls per block (and makes iflag=direct reads fail outright).

(The Android job failure is unrelated — it died in "Create and cache emulator image".)

[relative23]

Verified the fast path against the gate's own metric (instruction counts, single-pass analysis-mode bench binaries under valgrind --tool=cachegrind, whole-process I refs, main vs. this branch):

bench	main	this PR	delta
dd_copy_default	37,668,379	38,195,426	+1.40%
dd_copy_4k_blocks	4,741,075	4,809,777	+1.45%
dd_copy_partial	3,229,459	3,244,042	+0.45%

Down from the reported 4–5% to well under 1.5% (≈8 instructions per copied block, from the remaining slice re-borrows). The constant +4 KiB in the memory-mode numbers is the page over-allocation that provides the alignment guarantee itself.

[relative23]

Round 3, following up on the flame graphs from the last run: Input::read and Output::write_blocks were instruction-identical to main, so the residual simulation delta sat in the copy loop itself — every Deref of AlignedBuffer re-materialized the data slice (bounds check + RawVec::ptr reconstruction, ~4% combined on dd_copy_4k_blocks), plus the leftover per-iteration resize/truncate bookkeeping.

Rather than keep shaving at wrapper overhead, b859333 removes it structurally by consolidating with the read path from #12143 (credit to @chrboe — this is essentially his slice-based design on top of the safe runtime-page-size allocation):

• the copy loop takes the byte slice once, outside the loop; AlignedBuffer shrinks to an allocation helper and its Deref never appears in the hot loop
• fill_consecutive/fill_blocks operate on &mut [u8] and return the number of valid bytes instead of truncating
• conv=block/unblock output moves to a separate scratch Vec (those conversions can change the byte count); everything else stays in place
• the per-iteration resize is gone entirely, which also retires the dd takes a really long time to allocate memory with large buffers (tested with 1G) #11544 concern for this loop

The diff is net −75 lines. This run's CodSpeed result reflects it: no simulation benchmark is flagged anymore (previously default/4k_blocks/with_skip/with_seek at −3.2 to −3.8%). Local cachegrind on the codspeed analysis builds, same machine and toolchain for both sides (I refs, main 3aa5f8a vs this branch):

benchmark	main	this PR	Δ
dd_copy_default	37,668,434	37,081,830	−1.56%
dd_copy_4k_blocks	4,741,093	4,691,701	−1.04%
dd_copy_with_skip	8,405,560	8,310,586	−1.13%
dd_copy_with_seek	8,505,965	8,408,019	−1.15%
dd_copy_partial	3,229,514	3,220,910	−0.27%
dd_copy_8k_blocks	4,409,293	4,375,979	−0.76%
dd_copy_64k_blocks	5,110,850	5,103,359	−0.15%
dd_copy_1m_blocks	9,415,242	8,422,033	−10.55%
dd_copy_separate_blocks	55,875,370	55,938,721	+0.11%

That leaves the headline number (−16.32%), which is now purely the seven Memory entries averaged: peak memory is up by exactly one page (+4,096 B, constant, not proportional to the buffer size) because the buffer over-allocates one alignment unit to place the data page-aligned. That is the fix itself — the same trade GNU dd makes via ptr_align (…, page_size) — so I'd suggest acknowledging those seven entries in CodSpeed rather than gating on them; happy to adjust if there's a preferred way to handle intentional memory deltas.

Re-verified after the refactor: cargo test -p uu_dd (90) and the dd integration suite (123) pass; outputs are byte-identical to main across an 18-configuration sweep (odd ibs/obs, swab, sync, block/unblock, count_bytes, pipes); on a dma_alignment=511 loop device, iflag=direct reads succeed and a 16-block oflag=direct copy performs 3 fcntl calls in total instead of 2 per block.

The other two failing checks look unrelated to this diff: the macOS job failed on a test_timeout signal-timing flake (all 112 dd tests pass there), and cargo-deny hit a broken action invocation this morning (unrecognized subcommand 'warn') that other PRs stopped seeing within the hour — both should clear on a re-run.

[relative23]

Round 4: the seven Memory flags are gone as well — 5868242 removes the one-page over-allocation instead of asking for it to be acknowledged.

AlignedBuffer now allocates exactly bsize bytes at the runtime page size via std::alloc::alloc_zeroed(Layout) rather than over-allocating a Vec by one alignment unit and offsetting into it. That was the source of the constant +4,096 B peak-memory delta on every dd benchmark; with the exact allocation the peak matches the plain Vec<u8> baseline byte for byte. DHAT (allocator-level peak heap, dd bs=4K over a 3 MB file):

revision	peak heap	vs main
main	118,055 B	—
previous (over-allocating)	122,117 B	+4,062 B
this revision	118,021 B	−34 B

Being explicit about the trade-off, since the PR previously advertised "no unsafe": this adds three small unsafe blocks (alloc_zeroed, dealloc, Deref via slice::from_raw_parts), each with a SAFETY comment. The pointer is owned by AlignedBuffer alone and freed with the same Layout it was allocated with — unlike #9104's memalign + Vec::from_raw_parts, it never crosses into an allocator-aware container, so the allocator contract holds. The unit tests pass under Miri; note that they need to be run against the module standalone (e.g. copied into a scratch crate), because the uu_dd test binary as a whole cannot run under Miri — uucore's startup code calls sigaction, which Miri does not support. I think exact allocation plus a small, auditable unsafe surface is the better end state than a permanent by-design CodSpeed exception; happy to hear if you see it differently.

Side effect on the hot loop: Deref via from_raw_parts is cheaper than the previous bounds-checked storage[offset..] reconstruction, so the simulation numbers improved again (cachegrind I refs on the codspeed analysis builds, same machine/toolchain both sides, main 058a2a6):

benchmark	main	this PR	Δ
dd_copy_default	37,670,482	36,753,717	−2.43%
dd_copy_4k_blocks	4,743,133	4,662,232	−1.71%
dd_copy_with_skip	8,407,642	8,252,223	−1.85%
dd_copy_with_seek	8,508,047	8,348,418	−1.88%
dd_copy_8k_blocks	4,411,333	4,356,792	−1.24%
dd_copy_partial	3,231,604	3,219,789	−0.37%
dd_copy_64k_blocks	5,112,862	5,099,334	−0.26%
dd_copy_1m_blocks	9,417,324	8,424,089	−10.55%
dd_copy_separate_blocks	55,877,451	55,928,424	+0.09%

Re-verified after the change: 91 unit + 123 integration tests pass, outputs byte-identical to main across the 18-configuration sweep, and on a dma_alignment=511 loop device iflag=direct reads succeed while a 16-block oflag=direct copy performs 3 fcntl calls in total instead of 2 per block. The branch is also rebased onto current main (058a2a6).

[relative23]

Hardened the buffer internals a bit further ahead of review: AlignedBuffer now stores the Layout it validated at construction, so the deallocation invariant is simply "freed with exactly the stored layout" (and the unwrap is gone from the Drop path). The alignment contract is now enforced for zero-length buffers as well — previously try_new(0, 3) succeeded in release builds — and empty buffers get an aligned dangling pointer, so the alignment promise holds for them too. New edge-case tests cover rejected alignments and an unrepresentable size; the module's six unit tests still pass under Miri (standalone, as noted above).

One correction to my earlier phrasing for precision: the unsafe surface is four unsafe expressions in three categories — allocation, deallocation, and the two Deref slice constructions. No behavioral change to the copy loop; the DHAT peak is unchanged.

[sylvestre]

I guess this long comment is AI generated
could you please make it shorter ?

stuff like

/// The buffer is allocated directly with the requested alignment via
/// [std::alloc], so it is exactly len bytes — no memory overhead compared
/// to a Vec<u8> of the same size.

isn't super interesting

[sylvestre]

needs to use the translate!() macro

[relative23]

Agreed. The safe replacement should remove this error path; otherwise I will add a locale key.

[sylvestre]

we rarely use debug_assert_eq in this project
is it really interesting ?

[sylvestre]

do we really need unsafe ?
can't we use something from rustix or another crate to avoid this ?

[relative23]

rustix::mm::mmap is itself unsafe in rustix 1.1.4. I can use safe memmap2::MmapMut::map_anon, already a workspace dependency, or restore the safe over-allocated Vec. Which would you prefer? I also proposed splitting the probe/strace tests in the PR discussion.

[sylvestre]

please use the rustdoc format
thanks

[sylvestre]

same

[sylvestre]

can we avoid the unsafe here?

[sylvestre]

please read the contribution doc (or pass it to your LLM)

we have a don't panic policy
https://github.com/uutils/coreutils/blob/main/CONTRIBUTING.md#dont-panic

[relative23]

I have read the policy. I will remove the new panic and unwrap paths from this PR.

[sylvestre]

please try to use rustix

[relative23]

Agreed. If the probe remains, I will replace the libc calls with rustix; otherwise it will move to a follow-up.

[sylvestre]

i stopped reviewing it because i became tired ;)
I do appreciate the improvements that you are making but you need to use your AI better

A few comments:

• it is clearly written by an AI (i don't mind) but I mind that you aren't using it to make my life easier and these PR are impossible to review
• all your comments in this PR are too long
• This PR is WAY too big
• Too many unsafe
• please leverage other crates that we already use
• Don't follow our practices/recommendations
• If you improved the performances, write divan's benchmark to demonstrate this, i don't care about comments as they don't scale

I will stop there too but maybe i should start providing an AGENTS.md and skills to help contributors

[relative23]

Thanks, understood. Before reworking this: should I keep #13373 to the minimal alignment fix and move the probe/strace tests to a follow-up? rustix::mm::mmap is also unsafe; I can use safe memmap2::MmapMut::map_anon (already in the workspace) or the safe over-allocated Vec. Which do you prefer? I will drop the performance prose unless it is backed by divan.