Summary
The submission checker emitted a spurious [ERROR] [2.1.24 checkpointingTimestampGap]
for a checkpointing workload:
[ERROR] [2.1.24 checkpointingTimestampGap]
<results-path>/checkpointing/<workload>:
Gap between checkpoints is <gap>, which is >= the slower invocation's
duration <duration>. Benchmark activity between checkpoints can't be discarded.
This is the same class of issue as the earlier flush-between-checkpoint bug
(storage #714 / #782): the check was measuring the inter-invocation gap from the
wrong timestamps.
Root cause
A CLOSED checkpointing submission (per Rules.md 4.7.1) can consist of two
invocations — a write phase and a read phase — separated by a short quiet
window (cache flush). Two different timestamp sources exist for each invocation:
- DLIO summary
start / end — the DLIO benchmark loop bookends. These
include read-side framework startup (Python imports, MPI spawn, CAP
env-validation) and write-side post-benchmark cluster collection.
metadata.invocation_start_time / metadata.invocation_end_time — the
mlpstorage invocation bookends. These exclude that startup/collection
overhead and represent the true quiet window between phases.
2.1.24 was computing the gap from the DLIO summary fields, while the
authoritative 4.7.1 cache_flush_validation correctly uses the invocation
bookends. As a result the two checks disagreed for the same submission.
Behavior
| Measurement |
Source |
Result |
4.7.1 (correct) |
read.invocation_start_time − write.invocation_end_time |
small gap, within the 30 s threshold → PASS |
2.1.24 (buggy) |
read.summary.start − write.summary.end |
inflated gap → FAIL vs slower invocation duration |
The difference between the two measurements is exactly the overhead the
invocation bookends were introduced to exclude:
- read-side framework startup charged as "gap"
- write-side post-benchmark cluster collection charged as "gap"
The real quiet window between the phases is small, well within the 30 s
bound that 4.7.1 enforces.
This becomes especially evident for smaller checkpoints: when the
checkpoint operation itself is fast, its execution time can be shorter than
the fixed startup and cluster-collection overhead — which grows with the number
of nodes. On a large topology, that overhead (charged incorrectly against the
gap by the summary-based measurement) can dwarf the actual checkpoint duration,
so the buggy gap easily exceeds the slower invocation's duration and trips the
false violation. The larger the node count and the smaller the checkpoint, the
more pronounced the discrepancy.
Summary
The submission checker emitted a spurious
[ERROR] [2.1.24 checkpointingTimestampGap]for a checkpointing workload:
This is the same class of issue as the earlier flush-between-checkpoint bug
(storage #714 / #782): the check was measuring the inter-invocation gap from the
wrong timestamps.
Root cause
A CLOSED checkpointing submission (per Rules.md 4.7.1) can consist of two
invocations — a write phase and a read phase — separated by a short quiet
window (cache flush). Two different timestamp sources exist for each invocation:
start/end— the DLIO benchmark loop bookends. Theseinclude read-side framework startup (Python imports, MPI spawn, CAP
env-validation) and write-side post-benchmark cluster collection.
metadata.invocation_start_time/metadata.invocation_end_time— themlpstorage invocation bookends. These exclude that startup/collection
overhead and represent the true quiet window between phases.
2.1.24was computing the gap from the DLIO summary fields, while theauthoritative
4.7.1 cache_flush_validationcorrectly uses the invocationbookends. As a result the two checks disagreed for the same submission.
Behavior
4.7.1(correct)read.invocation_start_time − write.invocation_end_time2.1.24(buggy)read.summary.start − write.summary.endThe difference between the two measurements is exactly the overhead the
invocation bookends were introduced to exclude:
The real quiet window between the phases is small, well within the 30 s
bound that
4.7.1enforces.This becomes especially evident for smaller checkpoints: when the
checkpoint operation itself is fast, its execution time can be shorter than
the fixed startup and cluster-collection overhead — which grows with the number
of nodes. On a large topology, that overhead (charged incorrectly against the
gap by the summary-based measurement) can dwarf the actual checkpoint duration,
so the buggy gap easily exceeds the slower invocation's duration and trips the
false violation. The larger the node count and the smaller the checkpoint, the
more pronounced the discrepancy.