Tests comparing against cardano-cli, on top of cardano-testnet and tx-generator - #12
Open
palas wants to merge 27 commits into
Open
Tests comparing against cardano-cli, on top of cardano-testnet and tx-generator#12palas wants to merge 27 commits into
cardano-cli, on top of cardano-testnet and tx-generator#12palas wants to merge 27 commits into
Conversation
Compares queryLedgerState/protocolParameters (ogmios) against cardano-cli conway query protocol-parameters, checking key fields: minFeeCoefficient, minFeeConstant, maxBlockBodySize, maxTxSize, stakeCredentialDeposit, and stakePoolDeposit.
Compares queryLedgerState/tip slot against cardano-cli query tip.
Compares queryNetwork/tip slot against cardano-cli query tip.
Compares queryLedgerState/epoch (ogmios) against the epoch field from cardano-cli query tip.
Compares pool IDs from queryLedgerState/stakePools (ogmios) against cardano-cli conway query stake-pools.
Compares reward balances from queryLedgerState/rewardAccountSummaries (ogmios) against cardano-cli conway query stake-address-info.
Compares pool IDs in queryLedgerState/liveStakeDistribution (ogmios) against cardano-cli conway query stake-distribution.
Compares constitution anchor hash from queryLedgerState/constitution (ogmios) against cardano-cli conway query constitution.
Compares DRep IDs from queryLedgerState/delegateRepresentatives (ogmios) against cardano-cli conway query drep-state --all-dreps.
Redirect tx-generator stdout/stderr to logs/tx-generator.{stdout,stderr}
in the work directory, consistent with cardano-testnet and ogmios.
Jimbo4350
reviewed
Jul 13, 2026
Jimbo4350
reviewed
Jul 13, 2026
Jimbo4350
reviewed
Jul 13, 2026
Moved from cardano-node PR #6609 (bench/tx-generator/test-ogmios.sh), where it sat unreferenced by the cabal file, CI or nix; it exercises Ogmios's submitTransaction contract end to end and resolves its binaries from this repository's flake, so this is where it belongs. The script spins up a local cardano-testnet and an Ogmios instance, runs cardano-node's tx-generator with a submission endpoint config (every phase's transactions go through Ogmios as JSON-RPC 2.0 submitTransaction calls), and passes when new UTxOs appear at tx-generator's benchmark address. Adaptations from the original: ogmios now comes from this repository's own flake; tx-generator comes from a cardano-node flake ref given as $1, defaulting to github:IntersectMBO/cardano-node/master (until cardano-node#6609 is merged, pass its branch instead); and every binary can be supplied through the environment to skip its nix build. Kept as a plain script for now; integrating it with the server/test/integration suite is a follow-up.
Each failed readiness probe (connection refused while ogmios is still starting up) threw out of connect before close ran, leaking one socket fd per attempt. Wrap the socket in bracket so it is closed either way.
withResource only runs teardown when setup returns, so a throw after the processes had spawned (waitForFile/waitForTcpPort timeout, ogmios createProcess failure, non-zero tx-generator exit) orphaned cardano-testnet and ogmios and leaked the log handles. A leaked ogmios also keeps holding port 11337, which can make the next run's readiness probe pass against a stale daemon. Register each process and log handle in an IORef as it is created and wrap setup in bracketOnError, so a failing setup tears down whatever it had created (newest first) and re-raises. The success path is unchanged: teardown still owns cleanup once setup returns. runTxGenerator's resources, unlike setup's, do not outlive the call, so plain scoped cleanup fits there: the log handles move to withFile and the process itself to withCreateProcess, covering its failure paths (including cancellation mid-run) without a stack.
palas
force-pushed
the
testnet-tx-gen-tests
branch
2 times, most recently
from
July 29, 2026 17:35
7234aa9 to
89c3deb
Compare
Every query module carried its own copy of the WebSocket
send/receive/decode function (differing only in the method string and
optional params), the {result|error} unwrap case, a cardano-cli runner
and a parseEither-to-assertFailure wrapper. Move those into a shared
Test.Integration.Query:
queryOgmios - JSON-RPC request with the connection retry and the
result/error unwrap folded in
queryCli - cardano-cli query via --out-file; the socket, magic
and a fresh per-call out-file are appended, so
parallel tests cannot clobber each other's output
queryCliStdout - for commands without --out-file support
assertSameSet - set comparison printing each side's extra elements
parseIO - aeson parser failure to test failure
The per-module parsers and normalizers stay where they were; the query
modules keep only their method, cli arguments and comparisons.
palas
force-pushed
the
testnet-tx-gen-tests
branch
from
July 29, 2026 17:37
7234aa9 to
e730a2c
Compare
palas
force-pushed
the
testnet-tx-gen-tests
branch
from
July 29, 2026 17:38
e53b6b2 to
e730a2c
Compare
With `"aarch64-darwin"` enabled in `systems`, `packages.ogmios-musl`
resolves to `project.projectCross.aarch64-multiplatform-musl` on a macOS
host, i.e. a darwin -> aarch64-unknown-linux-musl cross build. No cache
has a substitute for the cross toolchain that needs, so Hydra compiles
`aarch64-unknown-linux-musl-ghc-9.8.4` from source on the darwin builders;
on evaluation 116643 it spent 1h32m on `oak-m2pro-2`, died with
`error: unexpected end-of-file`, was retried twice more (10m25s, 7m13s),
and took `aarch64-darwin.required` down with it.
A static Linux binary is only meaningful from a Linux host anyway, so gate
the job on the host platform. `x86_64-linux` and `aarch64-linux` keep
`ogmios-musl`; `aarch64-darwin` no longer offers it.
Verified with `nix eval .?submodules=1#hydraJobs.<system>.packages`:
aarch64-darwin [ default ogmios ogmios-integration-tests ogmios-lib ]
x86_64-linux [ default ogmios ogmios-integration-tests ogmios-lib
ogmios-musl ]
scripts/test-txgen-submission.sh checked that transactions submitted
through ogmios (cardano-node's tx-generator speaking its
`submissionEndpointProtocol: "Ogmios"` transport) are accepted and end up
in blocks. Everything it did by hand — start cardano-testnet, wait for the
node socket, start ogmios, wait for its port, run tx-generator, clean up —
the integration suite already does in Test.Integration.Env, and the five
binaries it resolved with nix are what the `integration` devshell provides.
So the script's own content boils down to one test case, which is what this
commit adds; the script goes away with it.
Test.Integration.TxSubmission reuses the suite's testnet and ogmios
instance, and asserts what the script asserted: tx-generator exits
cleanly, and new UTxOs appear at the address its last phase pays to (the
"BenchmarkingDone" key, whose address is derived here rather than
hardcoded, as in the script). Counting those UTxOs through both ogmios and
cardano-cli also covers an address-filtered ledger-state query, which the
whole-UTxO comparison in Test.Integration.Utxo does not.
Two details are worth pointing out, both marked in the code:
- The submission run does not pass --testnet-config-dir. Discovery always
picks utxo1 and overrides the config's own sigKey, and the funding run
in Env has already spent that key's genesis fund — spending it a second
time would be rejected for a missing input. The four connection
settings discovery would supply are given explicitly instead, with
utxo2 as the genesis key (cardano-testnet seeds three).
- targetNodes is spelled out as empty. An endpoint replaces the target
nodes as the submission target, so this is what the config compiler
wants, and it doubles as a guard: a tx-generator predating
IntersectMBO/cardano-node#6609 ignores the two submissionEndpoint* keys
and would otherwise quietly benchmark node-to-node, passing this test
without ogmios ever seeing a transaction. Such a version now fails on
the empty targetNodes, and the test checks tx-generator's own report of
its submission target as well.
Because the test mutates the UTxO set, it has to run after the comparison
tests, which assume nothing is moving underneath them: Main now sequences
the query groups ahead of it with AllFinish, so a failing comparison does
not skip submission coverage.
The transport landed in cardano-node after 11.0.1, so tx-generator comes
from a second flake input pinned past the merge, leaving the node, cli and
testnet binaries on the released tag the server is built against. That
input can be folded back into `cardano-node` once a release carries the
transport.
palas
force-pushed
the
testnet-tx-gen-tests
branch
from
July 30, 2026 20:27
cac3e57 to
a9b40e5
Compare
The unit test suite bakes the absolute path of the package root into
the test binary at compile time (Test.Path.Util.getProjectRoot, via
file-embed's `makeRelativeToProject ""`) and uses it to locate the
golden files, the network configurations and the test-vector output
directory. Nix builds and runs the suite in two separate derivations,
so the check recreated that path before running the tests:
mkdir -p /build/ogmios-src-test-unit-root/server/test
...
That only works on Linux, where every sandboxed build runs in /build.
On darwin build directories are unique per derivation (under
/private/tmp) and the root file system is sealed, so the check dies
immediately (hydra evaluation 117985 of this branch's jobset,
aarch64-darwin.checks.ogmios-unit):
Running phase: buildPhase
mkdir: cannot create directory '/build': Read-only file system
This is the last red job on aarch64-darwin: with the musl static build
gated to Linux hosts, everything else on the platform - the whole
Haskell dependency closure, packages.{ogmios,ogmios-lib,
ogmios-integration-tests,default} and both devShells - builds green on
that same evaluation, and `checks.ogmios-unit` is the one constituent
still failing `aarch64-darwin.required`.
Let the suite take the project root from an OGMIOS_TEST_PROJECT_ROOT
environment variable at run time, falling back to the baked path, and
have the check point that variable at a copy of the needed files under
its own (always writable) $TMPDIR. This removes the dependence on the
Linux sandbox layout altogether: the check now expresses the same thing
on every platform, and running the suite by hand keeps working exactly
as before since the fallback is untouched.
Two separate things kept `cabal build` from working inside
`nix develop .#integration` (and .#default alike):
1. cabal.project constrains `any.text source`. Its point is to keep the
solver away from GHC's bundled text so that the `-simdutf` flag
choice below actually applies to a from-source build, but `source`
also rejects the text that haskell.nix registers in the shell's
package database, while the shell's pre-built aeson only links
against exactly that instance:
rejecting: text; 2.1.4/installed-KVpHHZYgS6OIzTPD9vCwRV, ...
(constraint from project config cabal.project requires source instance)
rejecting: text; 2.1.4, 2.1.3, ...
(conflict: aeson => text==2.1.4/installed-KVpHHZYgS6OIzTPD9vCwRV)
Express the intent as a version bound instead: `>= 2.1.2` excludes
the bundled text (2.1.1 as of GHC 9.8) just as effectively, while
leaving installed instances of the versions it admits eligible. The
nix-side plan is unchanged - the solver still picks text-2.1.4,
built from Hackage with -simdutf (the very same unit id as before).
2. Even with that fixed, no target under cabal.project can be built
from the shell, because cabal resolves the project as a whole and
the ogmios package cannot be resolved there: it depends on public
sublibraries (ouroboros-network:api among others, and
ouroboros-network itself needs typed-protocols:cborg), and
cabal-install cannot satisfy a sublibrary dependency from an
installed package database, where each component is registered as
its own disjoint unit:
rejecting: ouroboros-network-1.1.0.0/installed-G4Sn...
(does not contain library 'api', which is required by ogmios)
That is a cabal-install limitation, not something project
configuration can express; building the server itself in the shell
keeps requiring nix until cabal learns to use installed
sublibraries.
The integration suite, however, does not depend on the ogmios
library at all (it drives the ogmios *binary*), and everything in
its build-depends is pre-built in the shell. So give it what it
really is: its own one-package project. Cabal's project discovery
stops at the first cabal.project on the way up, so from
server/test/integration a plain
cabal build
cabal run ogmios-integration-tests
now resolves instantly (only the suite itself gets compiled) and
works in both devshells. The index-state pin mirrors the main
project so the solver settles on the versions the shell provides
rather than on newer Hackage releases it would have to build. The
file changes nothing anywhere else: the outer project lists its
packages explicitly, and haskell.nix keeps planning from
server/cabal.project, so hydra and `nix build` are unaffected.
palas
force-pushed
the
testnet-tx-gen-tests
branch
from
August 1, 2026 02:53
2998b73 to
47066c0
Compare
cabal-install cannot take sublibraries out of an installed package
database: haskell.nix registers every component as its own disjoint
unit, and the solver wants a single ouroboros-network instance that
provides both the main library and `api` (likewise typed-protocols
with `cborg`, ouroboros-consensus with `cardano`, and so on). So it
re-plans that whole stack from source. Every such plan was in turn
dying on a single gate: ouroboros-consensus's `lmdb` sublibrary
declares `pkgconfig-depends: lmdb`, cabal checks the pkg-config
dependencies of every component of a source candidate at solve time -
whether or not that component would ever be built - and the shells'
pkg-config set (sodium, secp256k1, blst, ...) had no lmdb, since the
nix build of consensus never enables the lmdb backend either. Every
branch of the search failed on it, and the resulting "searching the
rest of the dependency tree exhaustively" error read as if the project
were structurally unresolvable from the shell. It is not.
Add lmdb to both shells. With it in the pkg-config set, plain
cabal build exe:ogmios # or lib:ogmios, or all
resolves with no extra flags and builds: the general Hackage layer
(aeson, websockets, wai, ...) is reused pre-built from the shell's
package db, while the Cardano layer - ledger with every era,
plutus-core, consensus, network, diffusion, io-classes, the crypto
packages; 52 components in all - compiles from source into cabal's
store. That happens once per checkout (measured ~8 minutes on an
8-core aarch64 machine) and is incremental afterwards. The lmdb
sublibrary itself is planned but never built, so liblmdb is only ever
consulted by pkg-config, not linked.
Verified: `cabal build exe:ogmios` from the integration shell links,
and the binary answers --version with the checkout's revision.
For iterating on the integration suite alone, the project root added
by the previous commit remains the cheap path - it needs none of this.
The Signatory schema in cardano.json fixes 'key' at 64 hex digits (a
plain Ed25519 verification key) and keeps the chain code in its own
optional 'chainCode' field, and upstream ogmios encodes Byron
transaction witnesses accordingly, splitting the 64-byte XPub in two:
encodeVerificationKey x =
"key" .= encodeByteStringBase16 (BS.take 32 xpub)
<>
"chainCode" .= encodeByteStringBase16 (BS.drop 32 xpub)
This branch dropped the split somewhere on the way to the dijkstra
integration and serialises the whole XPub (128 hex digits) under 'key',
so every Byron block with at least one transaction produces JSON that
violates ogmios' own schema. The unit suite's generative property over
NextBlockResponse catches it whenever QuickCheck draws such a block:
ogmios.json#/properties/NextBlockResponse
Assertion failed (after 42 tests):
...
invalid property 'key': '6e82da29b600684e...c474c890' should
have at most 64 character(s)
(Reproducible with --seed 1846810050. hspec randomises the seed on
every run, so each checks.ogmios-unit run on each platform is a dice
roll; the x86_64-linux job has simply been rolling well so far. It
first came up on an aarch64-linux run of the check.)
Restore the upstream encoding for witnesses. The block-level issuer and
delegate keys, and update-proposal voter keys, legitimately serialise
the full extended key - Block<BFT> points them at
ExtendedVerificationKey, 128 hex digits - so keep that behaviour under
upstream's name for it, encodeExtendedVerificationKey.
Test.Integration.Env resolves ogmios, cardano-node, cardano-cli, cardano-testnet and tx-generator from PATH (failing with e.g. "cardano-cli not found on PATH" otherwise). Inside `nix develop .#integration` they come from the shell, but the packaged `packages.ogmios-integration-tests` was the bare executable, so running it any other way - `nix run`, or the store path a `nix build` prints - fell over immediately. Wrap the packaged binary with those five tools as an execution dependency (prefixing PATH, so the pinned versions win no matter where it runs). `nix run .?submodules=1#ogmios-integration-tests` is now self-contained, which is also the shape a CI job that actually *executes* the suite would want. The devshell workflow is unchanged: `cabal run` builds stay unwrapped on purpose, so the shell's PATH (or a deliberately shadowed tool, as in the tx-generator 2.16 negative control) keeps deciding what the suite drives. The wrapper does mean hydra's `required` aggregate now carries the cardano-node tool closure on every platform; those store paths already substitute from cache.iog.io on both x86_64-linux and aarch64-darwin today (the `integration` devShell, green in the same jobset, pulls the very same ones). Verified: with no devshell at all, the wrapped binary runs the full suite - all 11 tests pass, tx-generator submitting through ogmios included.
So far CI only ever compiled the integration suite (packages.ogmios-integration-tests sits in the `required` aggregate); nothing executed it. Add `checks.ogmios-integration`, a derivation that runs the packaged suite from the previous commit - its wrapper already carries ogmios, cardano-node, cardano-cli, cardano-testnet and tx-generator - against a throwaway testnet spun up entirely inside the build, over loopback. On failure the build log ends with the tail of every daemon log, so hydra shows more than just the tasty summary. The check is defined for every system, so it can always be run by hand (`nix build .#checks.<system>.ogmios-integration`, or `nix flake check`; __darwinAllowLocalNetworking is set for sandboxed darwin machines - the suite only talks to 127.0.0.1). What hydra makes of it is decided separately, in hydraJobs.nix: * hydra only schedules it on Linux, where every build gets a private network namespace and the suite's fixed ogmios port (11337) cannot collide with anything. On darwin that port is shared with the whole builder, so concurrent evaluations could race for it; the suite runs fine on Apple silicon by hand, but CI leaves the darwin variant to manual runs. * on Linux it counts as advisory rather than merge-blocking: hydraJobs still lists every check as its own job, but the aggregates now split, `required` taking every check except those in `integrationChecks`, which land in `nonrequired` instead. A two-minute multi-daemon integration test has more ways to flake on a loaded builder than a pure unit suite; once it has a track record on the jobset, moving it over is a one-line change. Verified here: `nix build .#checks.aarch64-linux.ogmios-integration` runs the full suite inside the derivation - all 11 tests pass, the tx-generator submission included. Evaluating the jobsets shows the check present in flake checks on both platforms, scheduled by hydra only on Linux (in nonrequired's constituents, absent from required's, which are otherwise unchanged), and absent from the darwin jobset.
The repo defines no formatter, and flake-parts can only omit the
`formatter` output while its static heuristic manages to prove that no
perSystem module defines one. That proof is brittle in practice: with
the systems list as committed it holds, but merely enabling another
entry in `systems` made it give up, after which the flake exposes a
`formatter.<system>` attribute that throws when forced - and
`nix flake check` forces it:
error: flake-parts could not determine statically that no formatter
is defined for *all* systems.
Import flake-parts' touchup module and state the absence explicitly, as
that error message itself suggests. With this, `nix flake check
'.?submodules=1' --accept-flake-config` passes end to end here: it
evaluates every output (hydraJobs included), builds and runs the checks
for the current system - the unit suite and, since the previous commit,
the full integration suite - and omits other systems as usual unless
`--all-systems` is given.
palas
force-pushed
the
testnet-tx-gen-tests
branch
from
August 1, 2026 02:55
47066c0 to
9b89655
Compare
palas
force-pushed
the
testnet-tx-gen-tests
branch
from
August 1, 2026 03:08
5346710 to
85b6e0c
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
To run:
It includes a retry mechanism to avoid intermittent node connection issues.