forked from Imagelibrary/littlefs
Changed how fuzz tests are iterated to allow powerloss-fuzz testing
Instead of iterating over a number of seeds in the test itself, the seeds are now permuted as a part of normal test defines. This lets each seed take advantage of other test features, mainly the ability to test powerlosses heuristically. This is probably how it should have been done in the first place, but the permutation tests can't do this since the number of permutations changes as the size of the test input changes. The test define system can't handle that very well. The tradeoffs here are: - We can't do cross-fuzz checks, such as the balance checks in the rbyd tests, though those really should be moved to benchmarks anyways. - The large number of cheap fuzz permutations skews the total permutation count, though I'm not sure this matters. before: 3083 permutations (-Gnor) after: 409893 permutations (-Gnor)
This commit is contained in:
@@ -758,20 +758,22 @@ def find_ids(runner, bench_ids=[], **args):
|
||||
# find suite/case by id
|
||||
bench_ids_ = []
|
||||
for id in bench_ids:
|
||||
# strip permutation
|
||||
name, *_ = id.split(':', 1)
|
||||
bench_ids__ = []
|
||||
# resolve globs
|
||||
if '*' in id:
|
||||
if '*' in name:
|
||||
bench_ids__.extend(suite
|
||||
for suite in expected_suite_perms.keys()
|
||||
if fnmatch.fnmatch(suite, id))
|
||||
if fnmatch.fnmatch(suite, name))
|
||||
bench_ids__.extend(case_
|
||||
for case_ in expected_case_perms.keys()
|
||||
if fnmatch.fnmatch(case_, id))
|
||||
if fnmatch.fnmatch(case_, name))
|
||||
# literal suite
|
||||
elif id in expected_suite_perms:
|
||||
elif name in expected_suite_perms:
|
||||
bench_ids__.append(id)
|
||||
# literal case
|
||||
elif id in expected_case_perms:
|
||||
elif name in expected_case_perms:
|
||||
bench_ids__.append(id)
|
||||
|
||||
# no suite/case found? error
|
||||
|
||||
Reference in New Issue
Block a user