Changed how fuzz tests are iterated to allow powerloss-fuzz testing

Instead of iterating over a number of seeds in the test itself, the
seeds are now permuted as a part of normal test defines.

This lets each seed take advantage of other test features, mainly the
ability to test powerlosses heuristically.

This is probably how it should have been done in the first place, but
the permutation tests can't do this since the number of permutations
changes as the size of the test input changes. The test define system
can't handle that very well.

The tradeoffs here are:

- We can't do cross-fuzz checks, such as the balance checks in the rbyd
  tests, though those really should be moved to benchmarks anyways.

- The large number of cheap fuzz permutations skews the total
  permutation count, though I'm not sure this matters.

  before: 3083 permutations (-Gnor)
  after: 409893 permutations (-Gnor)
This commit is contained in:
Christopher Haster
2023-07-18 21:13:30 -05:00
parent c928ed131f
commit c5e84e874f
6 changed files with 2795 additions and 3124 deletions

View File

@@ -758,20 +758,22 @@ def find_ids(runner, bench_ids=[], **args):
# find suite/case by id
bench_ids_ = []
for id in bench_ids:
# strip permutation
name, *_ = id.split(':', 1)
bench_ids__ = []
# resolve globs
if '*' in id:
if '*' in name:
bench_ids__.extend(suite
for suite in expected_suite_perms.keys()
if fnmatch.fnmatch(suite, id))
if fnmatch.fnmatch(suite, name))
bench_ids__.extend(case_
for case_ in expected_case_perms.keys()
if fnmatch.fnmatch(case_, id))
if fnmatch.fnmatch(case_, name))
# literal suite
elif id in expected_suite_perms:
elif name in expected_suite_perms:
bench_ids__.append(id)
# literal case
elif id in expected_case_perms:
elif name in expected_case_perms:
bench_ids__.append(id)
# no suite/case found? error