Add Run.Shuffle to shuffle test execution order - #2948
Conversation
Adds Run.Random and Run.RandomSeed. When Run.Random is enabled the order of test files, and the blocks and tests inside them, is shuffled. Items are only reordered within their own level, so the Describes in a file, the Describes and Contexts in a Describe, and the Its in a block. The shuffle uses a seeded System.Random, so a run is repeatable. Set Run.RandomSeed to repeat a specific order. The default 0 picks a new seed each run and reports it at the start, so a failing order can be repeated. Fix pester#2425 🤖
Rename the option to Run.Shuffle / Run.ShuffleSeed. Shuffle reads better than Random and matches the wording in the run banner. Add a file level opt-out. A file or script block with a #pester:no-shuffle comment keeps its blocks and tests in declaration order even when Run.Shuffle is on, parsed the same way as #pester:no-parallel. Useful for tests that are order coupled on purpose, like mock history that accumulates across sibling Its. The resolved seed is written back to the run configuration, so it is returned on the result (Configuration.Run.ShuffleSeed) even when it was auto picked, and the run can be repeated on the same commit. 🤖
nohwnd
changed the title
|
@fflaten curious what you think about this one, mainly about the opt-out. Right now a whole file can opt out with 🤖 |
| t "reorders the Its inside a block" { | ||
| # Seeds are chosen so the C block's tests are not in declaration order. | ||
| $found = $false | ||
| foreach ($seed in 1..20) { |
There was a problem hiding this comment.
Use a known seed instead of brute force? 🙂
|
|
||
| t "reorders the Its inside a nested Context" { | ||
| $found = $false | ||
| foreach ($seed in 1..20) { |
| } | ||
| '@ | ||
| foreach ($n in 'Free1', 'Free2', 'Free3') { | ||
| Set-Content -Path (Join-Path $folder "$n.Tests.ps1") -Value "Describe '$n' { It 'a' { 1 | Should -Be 1 }; It 'b' { 1 | Should -Be 1 } }" |
There was a problem hiding this comment.
Assert that at least one of them was shuffled? As a implementation regression test so a single no-shuffle directive doesn't disable globally.
I think we should pick one, not both. Either we support:
Leaning towards block level for flexibility. The directive syntax is new (experimental) and might confuse as we still shuffle the container order - unlike no-parallel which runs it last IIRC. |
2 participants
Implements #2425.
Run.Shuffleshuffles the order things run in. It reorders the files, the Describes in a file, the Describes and Contexts in a Describe, and the Its in a block. Items only ever move within their own level.The order is repeatable. Each run picks a seed and prints it:
Set the seed to replay the same order:
You don't have to read it off the screen. The seed that was used is on the result too, so you can grab it after a run and replay:
Opting out
Some tests are ordered on purpose, for example mock history that adds up across sibling
Its. A file keeps its order with a comment, the same way it opts out of parallel:Its blocks and tests then run in declaration order even when
Run.Shuffleis on. A block level switch likeDescribe -Ordered { }could follow, keeping that out of this PR for now.How it works
The seed is resolved once in
Invoke-Pester, so it is reported once, shared by the parallel workers, and returned on the result. The files are shuffled inInvoke-Test, and each block's children inPostProcess-DiscoveredBlock, where.Blocksand.Testsare rebuilt from the shuffled order beforeFirst/Lastare marked, so the one time setup and teardown still wrap the real first and last item.Tried it on Pester itself
I ran Pester's own tests with
Run.Shuffle = $true. It found a few order coupled tests, all the expected kind, mock history that counts calls across siblingIts, and tests that check the exact setup and teardown sequence. Markingtst/functions/Mock.Tests.ps1with#pester:no-shuffletakes it from 6 failures back to 0 under a shuffled run, so the opt-out does its job.Tests are in
tst/Pester.RSpec.Shuffle.ts.ps1.Note: the opt-out keeps a file's own order, but the file still moves in the file order shuffle. And in parallel each worker shuffles its own slice of files, so cross file order is not globally reproducible there. The within file shuffle and the seed are.
🤖