Compare commits

...

31 Commits

Author SHA1 Message Date
Christopher Haster
ab59ab7f8d Bump default inline_max up to 1/4 block size
As noted by amgross, the current inline_max default (when littlefs
switches from inline files to CTZ skip-lists) does not match the
theoretical value in DESIGN.md.

The reason for this is 1/8 was chosen as a safer default during
development, due to concerns that 1/4 + mdirs splitting at 1/2 would
lead to poor distribution of large inline files in mdirs.

However, two things have happened since then:

1. Experiments have show "wasted" mdir space is less of a concern than
   initially thought. Extra mdir space contributes to logging, delays
   mdir compaction, and can overall lead to better performance.

2. inline_size was added as a configuration option, so if 1/4 is
   problematic users can always override it.

So bumping this back up to 1/4 may make sense.
2025-09-30 12:15:26 -05:00
Christopher Haster
8e251dd675 Merge pull request #1110 from Ryan-CW-Code/perf_gc
perf: gc might try to populate the lookahead buffer each time
2025-06-30 11:39:17 -05:00
Christopher Haster
25b9a4af85 Merge pull request #1109 from Ryan-CW-Code/never_read
refactor: value stored to 'diff' is never read
2025-06-30 11:39:05 -05:00
Christopher Haster
2acf939a00 Merge pull request #1106 from littlefs-project/fix-make-build-dep
make: Add missing BUILD_DEP include
2025-06-30 11:38:56 -05:00
ryancw
d5a86fd28d style: format code, limit to 80 columns. 2025-06-03 09:46:59 +08:00
ryancw
2349ac8c96 perf: gc might try to populate the lookahead buffer each time 2025-05-28 10:23:47 +08:00
ryancw
0755b00c21 refactor: value stored to 'diff' is never read 2025-05-27 20:00:29 +08:00
Christopher Haster
8365bbb7a2 make: Added missing BUILD_DEP include
This was preventing bench modifications from triggering relevant
bench-runner rebuilds.
2025-05-15 13:38:31 -05:00
Christopher Haster
16ceb67934 Merge pull request #1103 from littlefs-project/devel
Minor release: v2.11
2025-05-14 20:45:44 -05:00
Christopher Haster
8434536f0a Bumped minor version to v2.11 2025-05-13 13:18:31 -05:00
Christopher Haster
523319b685 Merge pull request #1104 from DvdGiessen/os-rename-between-filesystems
use shutil.move instead of os.rename to move file
2025-05-13 13:17:53 -05:00
Daniël van de Giessen
ba250a3075 use shutil.move instead of os.rename to move file
This prevents a "OSError: [Errno 18] Invalid cross-device link" if the temporary
file was created on different filesystem (such as a tmpfs mount).
2025-05-13 13:15:21 +02:00
Christopher Haster
8c458fa6bd Merge pull request #1094 from sosthene-nitrokey/shrink-fs
Add support for shrinking a filesystem
2025-05-13 00:45:32 -05:00
Christopher Haster
3149201ae5 Merge pull request #1091 from yamt/mach-o
adapt the linker sections usage to mach-o
2025-05-13 00:45:02 -05:00
Christopher Haster
6a43f3cdc3 Merge pull request #1090 from yamt/clang
drop a few unsupported CFLAGS for clang
2025-05-13 00:44:46 -05:00
Christopher Haster
d73fb8ef3c Merge pull request #1099 from littlefs-project/fix-remove-double-deorphan
Fix double deorphan caused by relocation mid dir remove
2025-05-13 00:44:26 -05:00
Christopher Haster
c1bf7cee84 Merge pull request #1100 from selimkeles/fix/bitshift_overflow
fix: added uint32_t cast to the bitshift places
2025-05-13 00:44:05 -05:00
Christopher Haster
b26bf3494c Merge pull request #1095 from DvdGiessen/lfs_crc
lfs_crc should be static if LFS_CRC is defined
2025-05-13 00:43:24 -05:00
Christopher Haster
0115cf6b74 gha: Dropped explicit CFLAGS from clang testing in CI
Thanks to yamt, GCC-specific flags should now be disabled if compiling
with clang. Dropping the explicit flags also doubles as a test that the
NO_GCC inference works.
2025-05-07 23:45:29 -05:00
Christopher Haster
bff4dfd1b1 Added NO_GCC to allow users to explicitly disable GCC-specific flags
This is the same as the implicit Clang => NO_GCC behavior introduced by
yamt, but with an explicit variable that can be assigned by users using
other, non-gcc, compilers:

  $ NO_GCC=1 make

Note, stack measurements are currently GCC specific:

  $ NO_GCC=1 make stack
  ... snip ...
  FileNotFoundError: [Errno 2] No such file or directory: 'lfs.ci'
  make: *** [Makefile:494: lfs.stack.csv] Error 1
2025-05-07 23:40:25 -05:00
Sosthène Guédon
edaaaf88ea Apply review comments 2025-05-07 10:38:43 +02:00
Sosthène Guédon
7d79423972 Rename SHRINKIFCHEAP to SHRINKNONRELOCATING 2025-05-07 10:34:24 +02:00
Sosthène Guédon
7782d3dfa3 Mention that shrinking is unlikely to work 2025-05-06 11:00:29 +02:00
selim.keles
f4a1bb328a fix: added uint32_t cast to the bitshift places
In 16 bit and 8 bit architectures, overflow and underflow issues were occuring while using functions lfs_frombe32 and lfs_fromle32
2025-05-05 13:38:05 +03:00
Sosthène Guédon
9b8f802b43 fixup! Add support for shrinking a filesystem 2025-05-05 11:37:39 +02:00
Christopher Haster
a3d6bec5f0 Fixed a double deorphan caused by relocation mid dir remove
Long story short: There is a specific case where removing a directory
can trigger a deorphan pass, but lfs_remove did not check for this,
would try to clean up the (already cleaned) directory orphan, and
trigger an assert:

  lfs.c:4890:assert: assert failed with false, expected eq true
      LFS_ASSERT(lfs_tag_size(lfs->gstate.tag) > 0x000 || orphans >= 0);

The specific case being a remove commit that triggers a relocation that
creates an orphan.

This is also possible in lfs_rename, but only if you're renaming a
directory that implies a remove, which is a pretty rare operation.

---

This was probably an oversight introduced in the non-recursive commit
logic rework.

Fortunately the fix is to just check if we even have an orphan before
trying to remove it. We can rely on this instead of the file type, so
this fix shouldn't even increase the code size.

Found and root-caused by Hugh-Baoa
2025-05-03 18:13:19 -05:00
Christopher Haster
0634d13e07 tests: Added non-reentrant variants of orphan/relocation tests
These are the same as the related reentrant variants, but by opting out
of powerloss testing, we can test a much larger number of states without
having to worry about the impact on powerloss testing runtime.

Bumped CYCLES from 20 -> 2000.

This reveals an orphan remove bug found by Hugh-Baoa.
2025-05-03 17:15:26 -05:00
Sosthène Guédon
2105e502c5 Add support for shrinking a filesystem
This PR adds a new `lfs_fs_shrink`, which functions similarly to
`lfs_fs_grow`, but supports reducing the block count.

This functions first checks that none of the removed block are in use.
If it is the case, it will fail.
2025-04-17 10:07:37 +02:00
Daniël van de Giessen
b823728420 lfs_crc should be static if LFS_CRC is defined 2025-04-16 18:17:21 +02:00
YAMAMOTO Takashi
0d861b7916 adapt the linker sections usage to mach-o
"make test" on macOS:

```
using runner: ./runners/test_runner
found 19 suites, 188 cases, 11242/11770 permutations

running test_alloc: 12/12 cases, 207/207 perms
running test_attrs: 4/4 cases, 20/20 perms
running test_badblocks: 4/4 cases, 300/300 perms
running test_bd: 5/5 cases, 85/85 perms
running test_compat: 17/17 cases, 205/205 perms
running test_dirs: 15/15 cases, 450/450 perms, 1756pls!
running test_entries: 8/8 cases, 32/32 perms
running test_evil: 8/8 cases, 105/105 perms
running test_exhaustion: 5/5 cases, 85/85 perms
running test_files: 10/10 cases, 7155/7155 perms, 9410pls!
running test_interspersed: 4/4 cases, 190/190 perms, 2835pls!
running test_move: 17/17 cases, 161/161 perms, 157pls!
running test_orphans: 6/6 cases, 50/50 perms, 846pls!
running test_paths: 33/33 cases, 325/325 perms
running test_powerloss: 2/2 cases, 21/21 perms
running test_relocations: 4/4 cases, 68/68 perms, 1612pls!
running test_seek: 10/10 cases, 195/195 perms, 1050pls!
running test_superblocks: 17/17 cases, 318/318 perms, 1437pls!
running test_truncate: 7/7 cases, 1270/1270 perms, 9691pls!

done: 11242/11242 passed, 0/11242 failed, 28794pls!, in 585.76s
```
2025-04-07 16:20:23 +09:00
YAMAMOTO Takashi
26bee8ad36 drop a few unsupported CFLAGS for clang 2025-04-07 16:06:01 +09:00
14 changed files with 581 additions and 61 deletions

View File

@@ -374,6 +374,22 @@ jobs:
run: |
CFLAGS="$CFLAGS -DLFS_NO_INTRINSICS" make test
test-shrink:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: install
run: |
# need a few things
sudo apt-get update -qq
sudo apt-get install -qq gcc python3 python3-pip
pip3 install toml
gcc --version
python3 --version
- name: test-no-intrinsics
run: |
CFLAGS="$CFLAGS -DLFS_SHRINKNONRELOCATING" make test
# run with all trace options enabled to at least make sure these
# all compile
test-yes-trace:
@@ -454,8 +470,7 @@ jobs:
TESTFLAGS="$TESTFLAGS --valgrind --context=1024 -Gdefault -Pnone" \
make test
# test that compilation is warning free under clang
# run with Clang, mostly to check for Clang-specific warnings
# compile/run with Clang, mostly to check for Clang-specific warnings
test-clang:
runs-on: ubuntu-latest
steps:
@@ -469,12 +484,8 @@ jobs:
python3 --version
- name: test-clang
run: |
# override CFLAGS since Clang does not support -fcallgraph-info
# and -ftrack-macro-expansions
make \
CC=clang \
CFLAGS="$CFLAGS -MMD -g3 -I. -std=c99 -Wall -Wextra -pedantic" \
test
CC=clang \
make test
# run benchmarks
#

View File

@@ -18,6 +18,12 @@ VALGRIND ?= valgrind
GDB ?= gdb
PERF ?= perf
# guess clang or gcc (clang sometimes masquerades as gcc because of
# course it does)
ifneq ($(shell $(CC) --version | grep clang),)
NO_GCC = 1
endif
SRC ?= $(filter-out $(wildcard *.t.* *.b.*),$(wildcard *.c))
OBJ := $(SRC:%.c=$(BUILDDIR)/%.o)
DEP := $(SRC:%.c=$(BUILDDIR)/%.d)
@@ -59,12 +65,15 @@ BENCH_PERF := $(BENCH_RUNNER:%=%.perf)
BENCH_TRACE := $(BENCH_RUNNER:%=%.trace)
BENCH_CSV := $(BENCH_RUNNER:%=%.csv)
CFLAGS += -fcallgraph-info=su
CFLAGS += -g3
CFLAGS += -I.
CFLAGS += -std=c99 -Wall -Wextra -pedantic
CFLAGS += -Wmissing-prototypes
ifndef NO_GCC
CFLAGS += -fcallgraph-info=su
CFLAGS += -ftrack-macro-expansion=0
endif
ifdef DEBUG
CFLAGS += -O0
else
@@ -466,6 +475,7 @@ benchmarks-diff: $(BENCH_CSV)
# rules
-include $(DEP)
-include $(TEST_DEP)
-include $(BENCH_DEP)
.SUFFIXES:
.SECONDARY:

97
lfs.c
View File

@@ -828,9 +828,6 @@ static int lfs_dir_getread(lfs_t *lfs, const lfs_mdir_t *dir,
size -= diff;
continue;
}
// rcache takes priority
diff = lfs_min(diff, rcache->off-off);
}
// load to cache, first condition can no longer fail
@@ -3932,7 +3929,9 @@ static int lfs_remove_(lfs_t *lfs, const char *path) {
}
lfs->mlist = dir.next;
if (lfs_tag_type3(tag) == LFS_TYPE_DIR) {
if (lfs_gstate_hasorphans(&lfs->gstate)) {
LFS_ASSERT(lfs_tag_type3(tag) == LFS_TYPE_DIR);
// fix orphan
err = lfs_fs_preporphans(lfs, -1);
if (err) {
@@ -4076,8 +4075,10 @@ static int lfs_rename_(lfs_t *lfs, const char *oldpath, const char *newpath) {
}
lfs->mlist = prevdir.next;
if (prevtag != LFS_ERR_NOENT
&& lfs_tag_type3(prevtag) == LFS_TYPE_DIR) {
if (lfs_gstate_hasorphans(&lfs->gstate)) {
LFS_ASSERT(prevtag != LFS_ERR_NOENT
&& lfs_tag_type3(prevtag) == LFS_TYPE_DIR);
// fix orphan
err = lfs_fs_preporphans(lfs, -1);
if (err) {
@@ -4330,7 +4331,7 @@ static int lfs_init(lfs_t *lfs, const struct lfs_config *cfg) {
LFS_ASSERT(lfs->cfg->inline_max == (lfs_size_t)-1
|| lfs->cfg->inline_max <= ((lfs->cfg->metadata_max)
? lfs->cfg->metadata_max
: lfs->cfg->block_size)/8);
: lfs->cfg->block_size)/4);
lfs->inline_max = lfs->cfg->inline_max;
if (lfs->inline_max == (lfs_size_t)-1) {
lfs->inline_max = 0;
@@ -4341,7 +4342,7 @@ static int lfs_init(lfs_t *lfs, const struct lfs_config *cfg) {
lfs->attr_max,
((lfs->cfg->metadata_max)
? lfs->cfg->metadata_max
: lfs->cfg->block_size)/8));
: lfs->cfg->block_size)/4));
}
// setup default state
@@ -5221,7 +5222,9 @@ static int lfs_fs_gc_(lfs_t *lfs) {
}
// try to populate the lookahead buffer, unless it's already full
if (lfs->lookahead.size < 8*lfs->cfg->lookahead_size) {
if (lfs->lookahead.size < lfs_min(
8 * lfs->cfg->lookahead_size,
lfs->block_count)) {
err = lfs_alloc_scan(lfs);
if (err) {
return err;
@@ -5233,40 +5236,64 @@ static int lfs_fs_gc_(lfs_t *lfs) {
#endif
#ifndef LFS_READONLY
#ifdef LFS_SHRINKNONRELOCATING
static int lfs_shrink_checkblock(void *data, lfs_block_t block) {
lfs_size_t threshold = *((lfs_size_t*)data);
if (block >= threshold) {
return LFS_ERR_NOTEMPTY;
}
return 0;
}
#endif
static int lfs_fs_grow_(lfs_t *lfs, lfs_size_t block_count) {
int err;
if (block_count == lfs->block_count) {
return 0;
}
#ifndef LFS_SHRINKNONRELOCATING
// shrinking is not supported
LFS_ASSERT(block_count >= lfs->block_count);
if (block_count > lfs->block_count) {
lfs->block_count = block_count;
// fetch the root
lfs_mdir_t root;
int err = lfs_dir_fetch(lfs, &root, lfs->root);
if (err) {
return err;
}
// update the superblock
lfs_superblock_t superblock;
lfs_stag_t tag = lfs_dir_get(lfs, &root, LFS_MKTAG(0x7ff, 0x3ff, 0),
LFS_MKTAG(LFS_TYPE_INLINESTRUCT, 0, sizeof(superblock)),
&superblock);
if (tag < 0) {
return tag;
}
lfs_superblock_fromle32(&superblock);
superblock.block_count = lfs->block_count;
lfs_superblock_tole32(&superblock);
err = lfs_dir_commit(lfs, &root, LFS_MKATTRS(
{tag, &superblock}));
#endif
#ifdef LFS_SHRINKNONRELOCATING
if (block_count < lfs->block_count) {
err = lfs_fs_traverse_(lfs, lfs_shrink_checkblock, &block_count, true);
if (err) {
return err;
}
}
#endif
lfs->block_count = block_count;
// fetch the root
lfs_mdir_t root;
err = lfs_dir_fetch(lfs, &root, lfs->root);
if (err) {
return err;
}
// update the superblock
lfs_superblock_t superblock;
lfs_stag_t tag = lfs_dir_get(lfs, &root, LFS_MKTAG(0x7ff, 0x3ff, 0),
LFS_MKTAG(LFS_TYPE_INLINESTRUCT, 0, sizeof(superblock)),
&superblock);
if (tag < 0) {
return tag;
}
lfs_superblock_fromle32(&superblock);
superblock.block_count = lfs->block_count;
lfs_superblock_tole32(&superblock);
err = lfs_dir_commit(lfs, &root, LFS_MKATTRS(
{tag, &superblock}));
if (err) {
return err;
}
return 0;
}
#endif

10
lfs.h
View File

@@ -21,7 +21,7 @@ extern "C"
// Software library version
// Major (top-nibble), incremented on backwards incompatible changes
// Minor (bottom-nibble), incremented on feature additions
#define LFS_VERSION 0x0002000a
#define LFS_VERSION 0x0002000b
#define LFS_VERSION_MAJOR (0xffff & (LFS_VERSION >> 16))
#define LFS_VERSION_MINOR (0xffff & (LFS_VERSION >> 0))
@@ -277,7 +277,7 @@ struct lfs_config {
// Optional upper limit on inlined files in bytes. Inlined files live in
// metadata and decrease storage requirements, but may be limited to
// improve metadata-related performance. Must be <= cache_size, <=
// attr_max, and <= block_size/8. Defaults to the largest possible
// attr_max, and <= block_size/4. Defaults to the largest possible
// inline_max when zero.
//
// Set to -1 to disable inlined files.
@@ -766,7 +766,11 @@ int lfs_fs_gc(lfs_t *lfs);
// Grows the filesystem to a new size, updating the superblock with the new
// block count.
//
// Note: This is irreversible.
// If LFS_SHRINKNONRELOCATING is defined, this function will also accept
// block_counts smaller than the current configuration, after checking
// that none of the blocks that are being removed are in use.
// Note that littlefs's pseudorandom block allocation means that
// this is very unlikely to work in the general case.
//
// Returns a negative error code on failure.
int lfs_fs_grow(lfs_t *lfs, lfs_size_t block_count);

View File

@@ -195,10 +195,10 @@ static inline uint32_t lfs_fromle32(uint32_t a) {
(defined(__BYTE_ORDER__) && defined(__ORDER_BIG_ENDIAN__) && __BYTE_ORDER__ == __ORDER_BIG_ENDIAN__))
return __builtin_bswap32(a);
#else
return (((uint8_t*)&a)[0] << 0) |
(((uint8_t*)&a)[1] << 8) |
(((uint8_t*)&a)[2] << 16) |
(((uint8_t*)&a)[3] << 24);
return ((uint32_t)((uint8_t*)&a)[0] << 0) |
((uint32_t)((uint8_t*)&a)[1] << 8) |
((uint32_t)((uint8_t*)&a)[2] << 16) |
((uint32_t)((uint8_t*)&a)[3] << 24);
#endif
}
@@ -218,10 +218,10 @@ static inline uint32_t lfs_frombe32(uint32_t a) {
(defined(__BYTE_ORDER__) && defined(__ORDER_BIG_ENDIAN__) && __BYTE_ORDER__ == __ORDER_BIG_ENDIAN__)
return a;
#else
return (((uint8_t*)&a)[0] << 24) |
(((uint8_t*)&a)[1] << 16) |
(((uint8_t*)&a)[2] << 8) |
(((uint8_t*)&a)[3] << 0);
return ((uint32_t)((uint8_t*)&a)[0] << 24) |
((uint32_t)((uint8_t*)&a)[1] << 16) |
((uint32_t)((uint8_t*)&a)[2] << 8) |
((uint32_t)((uint8_t*)&a)[3] << 0);
#endif
}
@@ -231,8 +231,8 @@ static inline uint32_t lfs_tobe32(uint32_t a) {
// Calculate CRC-32 with polynomial = 0x04c11db7
#ifdef LFS_CRC
uint32_t lfs_crc(uint32_t crc, const void *buffer, size_t size) {
return LFS_CRC(crc, buffer, size)
static inline uint32_t lfs_crc(uint32_t crc, const void *buffer, size_t size) {
return LFS_CRC(crc, buffer, size);
}
#else
uint32_t lfs_crc(uint32_t crc, const void *buffer, size_t size);

View File

@@ -123,8 +123,13 @@ typedef struct bench_id {
// bench suites are linked into a custom ld section
#if defined(__APPLE__)
extern struct bench_suite __start__bench_suites __asm("section$start$__DATA$_bench_suites");
extern struct bench_suite __stop__bench_suites __asm("section$end$__DATA$_bench_suites");
#else
extern struct bench_suite __start__bench_suites;
extern struct bench_suite __stop__bench_suites;
#endif
const struct bench_suite *bench_suites = &__start__bench_suites;
#define BENCH_SUITE_COUNT \

View File

@@ -136,8 +136,13 @@ typedef struct test_id {
// test suites are linked into a custom ld section
#if defined(__APPLE__)
extern struct test_suite __start__test_suites __asm("section$start$__DATA$_test_suites");
extern struct test_suite __stop__test_suites __asm("section$end$__DATA$_test_suites");
#else
extern struct test_suite __start__test_suites;
extern struct test_suite __stop__test_suites;
#endif
const struct test_suite *test_suites = &__start__test_suites;
#define TEST_SUITE_COUNT \

View File

@@ -404,12 +404,15 @@ def compile(bench_paths, **args):
f.writeln()
# create suite struct
#
f.writeln('#if defined(__APPLE__)')
f.writeln('__attribute__((section("__DATA,_bench_suites")))')
f.writeln('#else')
# note we place this in the custom bench_suites section with
# minimum alignment, otherwise GCC ups the alignment to
# 32-bytes for some reason
f.writeln('__attribute__((section("_bench_suites"), '
'aligned(1)))')
f.writeln('#endif')
f.writeln('const struct bench_suite __bench__%s__suite = {'
% suite.name)
f.writeln(4*' '+'.name = "%s",' % suite.name)

View File

@@ -73,7 +73,7 @@ def changefile(from_prefix, to_prefix, from_path, to_path, *,
shutil.copystat(from_path, to_path)
if to_path_temp:
os.rename(to_path, from_path)
shutil.move(to_path, from_path)
elif from_path != '-':
os.remove(from_path)

View File

@@ -412,12 +412,15 @@ def compile(test_paths, **args):
f.writeln()
# create suite struct
#
f.writeln('#if defined(__APPLE__)')
f.writeln('__attribute__((section("__DATA,_test_suites")))')
f.writeln('#else')
# note we place this in the custom test_suites section with
# minimum alignment, otherwise GCC ups the alignment to
# 32-bytes for some reason
f.writeln('__attribute__((section("_test_suites"), '
'aligned(1)))')
f.writeln('#endif')
f.writeln('const struct test_suite __test__%s__suite = {'
% suite.name)
f.writeln(4*' '+'.name = "%s",' % suite.name)

View File

@@ -207,7 +207,8 @@ code = '''
[cases.test_orphans_reentrant]
reentrant = true
# TODO fix this case, caused by non-DAG trees
if = '!(DEPTH == 3 && CACHE_SIZE != 64)'
# NOTE the second condition is required
if = '!(DEPTH == 3 && CACHE_SIZE != 64) && 2*FILES < BLOCK_COUNT'
defines = [
{FILES=6, DEPTH=1, CYCLES=20},
{FILES=26, DEPTH=1, CYCLES=20},
@@ -271,3 +272,69 @@ code = '''
lfs_unmount(&lfs) => 0;
'''
# non-reentrant testing for orphans, this is the same as reentrant
# testing, but we test way more states than we could under powerloss
[cases.test_orphans_nonreentrant]
# TODO fix this case, caused by non-DAG trees
# NOTE the second condition is required
if = '!(DEPTH == 3 && CACHE_SIZE != 64) && 2*FILES < BLOCK_COUNT'
defines = [
{FILES=6, DEPTH=1, CYCLES=2000},
{FILES=26, DEPTH=1, CYCLES=2000},
{FILES=3, DEPTH=3, CYCLES=2000},
]
code = '''
lfs_t lfs;
lfs_format(&lfs, cfg) => 0;
lfs_mount(&lfs, cfg) => 0;
uint32_t prng = 1;
const char alpha[] = "abcdefghijklmnopqrstuvwxyz";
for (unsigned i = 0; i < CYCLES; i++) {
// create random path
char full_path[256];
for (unsigned d = 0; d < DEPTH; d++) {
sprintf(&full_path[2*d], "/%c", alpha[TEST_PRNG(&prng) % FILES]);
}
// if it does not exist, we create it, else we destroy
struct lfs_info info;
int res = lfs_stat(&lfs, full_path, &info);
if (res == LFS_ERR_NOENT) {
// create each directory in turn, ignore if dir already exists
for (unsigned d = 0; d < DEPTH; d++) {
char path[1024];
strcpy(path, full_path);
path[2*d+2] = '\0';
int err = lfs_mkdir(&lfs, path);
assert(!err || err == LFS_ERR_EXIST);
}
for (unsigned d = 0; d < DEPTH; d++) {
char path[1024];
strcpy(path, full_path);
path[2*d+2] = '\0';
lfs_stat(&lfs, path, &info) => 0;
assert(strcmp(info.name, &path[2*d+1]) == 0);
assert(info.type == LFS_TYPE_DIR);
}
} else {
// is valid dir?
assert(strcmp(info.name, &full_path[2*(DEPTH-1)+1]) == 0);
assert(info.type == LFS_TYPE_DIR);
// try to delete path in reverse order, ignore if dir is not empty
for (int d = DEPTH-1; d >= 0; d--) {
char path[1024];
strcpy(path, full_path);
path[2*d+2] = '\0';
int err = lfs_remove(&lfs, path);
assert(!err || err == LFS_ERR_NOTEMPTY);
}
lfs_stat(&lfs, full_path, &info) => LFS_ERR_NOENT;
}
}
lfs_unmount(&lfs) => 0;
'''

View File

@@ -341,3 +341,171 @@ code = '''
}
lfs_unmount(&lfs) => 0;
'''
# non-reentrant testing for orphans, this is the same as reentrant
# testing, but we test way more states than we could under powerloss
[cases.test_relocations_nonreentrant]
# TODO fix this case, caused by non-DAG trees
# NOTE the second condition is required
if = '!(DEPTH == 3 && CACHE_SIZE != 64) && 2*FILES < BLOCK_COUNT'
defines = [
{FILES=6, DEPTH=1, CYCLES=2000, BLOCK_CYCLES=1},
{FILES=26, DEPTH=1, CYCLES=2000, BLOCK_CYCLES=1},
{FILES=3, DEPTH=3, CYCLES=2000, BLOCK_CYCLES=1},
]
code = '''
lfs_t lfs;
lfs_format(&lfs, cfg) => 0;
lfs_mount(&lfs, cfg) => 0;
uint32_t prng = 1;
const char alpha[] = "abcdefghijklmnopqrstuvwxyz";
for (unsigned i = 0; i < CYCLES; i++) {
// create random path
char full_path[256];
for (unsigned d = 0; d < DEPTH; d++) {
sprintf(&full_path[2*d], "/%c", alpha[TEST_PRNG(&prng) % FILES]);
}
// if it does not exist, we create it, else we destroy
struct lfs_info info;
int res = lfs_stat(&lfs, full_path, &info);
if (res == LFS_ERR_NOENT) {
// create each directory in turn, ignore if dir already exists
for (unsigned d = 0; d < DEPTH; d++) {
char path[1024];
strcpy(path, full_path);
path[2*d+2] = '\0';
int err = lfs_mkdir(&lfs, path);
assert(!err || err == LFS_ERR_EXIST);
}
for (unsigned d = 0; d < DEPTH; d++) {
char path[1024];
strcpy(path, full_path);
path[2*d+2] = '\0';
lfs_stat(&lfs, path, &info) => 0;
assert(strcmp(info.name, &path[2*d+1]) == 0);
assert(info.type == LFS_TYPE_DIR);
}
} else {
// is valid dir?
assert(strcmp(info.name, &full_path[2*(DEPTH-1)+1]) == 0);
assert(info.type == LFS_TYPE_DIR);
// try to delete path in reverse order, ignore if dir is not empty
for (unsigned d = DEPTH-1; d+1 > 0; d--) {
char path[1024];
strcpy(path, full_path);
path[2*d+2] = '\0';
int err = lfs_remove(&lfs, path);
assert(!err || err == LFS_ERR_NOTEMPTY);
}
lfs_stat(&lfs, full_path, &info) => LFS_ERR_NOENT;
}
}
lfs_unmount(&lfs) => 0;
'''
# non-reentrant testing for relocations, but now with random renames!
[cases.test_relocations_nonreentrant_renames]
# TODO fix this case, caused by non-DAG trees
# NOTE the second condition is required
if = '!(DEPTH == 3 && CACHE_SIZE != 64) && 2*FILES < BLOCK_COUNT'
defines = [
{FILES=6, DEPTH=1, CYCLES=2000, BLOCK_CYCLES=1},
{FILES=26, DEPTH=1, CYCLES=2000, BLOCK_CYCLES=1},
{FILES=3, DEPTH=3, CYCLES=2000, BLOCK_CYCLES=1},
]
code = '''
lfs_t lfs;
lfs_format(&lfs, cfg) => 0;
lfs_mount(&lfs, cfg) => 0;
uint32_t prng = 1;
const char alpha[] = "abcdefghijklmnopqrstuvwxyz";
for (unsigned i = 0; i < CYCLES; i++) {
// create random path
char full_path[256];
for (unsigned d = 0; d < DEPTH; d++) {
sprintf(&full_path[2*d], "/%c", alpha[TEST_PRNG(&prng) % FILES]);
}
// if it does not exist, we create it, else we destroy
struct lfs_info info;
int res = lfs_stat(&lfs, full_path, &info);
assert(!res || res == LFS_ERR_NOENT);
if (res == LFS_ERR_NOENT) {
// create each directory in turn, ignore if dir already exists
for (unsigned d = 0; d < DEPTH; d++) {
char path[1024];
strcpy(path, full_path);
path[2*d+2] = '\0';
int err = lfs_mkdir(&lfs, path);
assert(!err || err == LFS_ERR_EXIST);
}
for (unsigned d = 0; d < DEPTH; d++) {
char path[1024];
strcpy(path, full_path);
path[2*d+2] = '\0';
lfs_stat(&lfs, path, &info) => 0;
assert(strcmp(info.name, &path[2*d+1]) == 0);
assert(info.type == LFS_TYPE_DIR);
}
} else {
assert(strcmp(info.name, &full_path[2*(DEPTH-1)+1]) == 0);
assert(info.type == LFS_TYPE_DIR);
// create new random path
char new_path[256];
for (unsigned d = 0; d < DEPTH; d++) {
sprintf(&new_path[2*d], "/%c", alpha[TEST_PRNG(&prng) % FILES]);
}
// if new path does not exist, rename, otherwise destroy
res = lfs_stat(&lfs, new_path, &info);
assert(!res || res == LFS_ERR_NOENT);
if (res == LFS_ERR_NOENT) {
// stop once some dir is renamed
for (unsigned d = 0; d < DEPTH; d++) {
char path[1024];
strcpy(&path[2*d], &full_path[2*d]);
path[2*d+2] = '\0';
strcpy(&path[128+2*d], &new_path[2*d]);
path[128+2*d+2] = '\0';
int err = lfs_rename(&lfs, path, path+128);
assert(!err || err == LFS_ERR_NOTEMPTY);
if (!err) {
strcpy(path, path+128);
}
}
for (unsigned d = 0; d < DEPTH; d++) {
char path[1024];
strcpy(path, new_path);
path[2*d+2] = '\0';
lfs_stat(&lfs, path, &info) => 0;
assert(strcmp(info.name, &path[2*d+1]) == 0);
assert(info.type == LFS_TYPE_DIR);
}
lfs_stat(&lfs, full_path, &info) => LFS_ERR_NOENT;
} else {
// try to delete path in reverse order,
// ignore if dir is not empty
for (unsigned d = DEPTH-1; d+1 > 0; d--) {
char path[1024];
strcpy(path, full_path);
path[2*d+2] = '\0';
int err = lfs_remove(&lfs, path);
assert(!err || err == LFS_ERR_NOTEMPTY);
}
lfs_stat(&lfs, full_path, &info) => LFS_ERR_NOENT;
}
}
}
lfs_unmount(&lfs) => 0;
'''

109
tests/test_shrink.toml Normal file
View File

@@ -0,0 +1,109 @@
# simple shrink
[cases.test_shrink_simple]
defines.BLOCK_COUNT = [10, 15, 20]
defines.AFTER_BLOCK_COUNT = [5, 10, 15, 19]
if = "AFTER_BLOCK_COUNT <= BLOCK_COUNT"
code = '''
#ifdef LFS_SHRINKNONRELOCATING
lfs_t lfs;
lfs_format(&lfs, cfg) => 0;
lfs_mount(&lfs, cfg) => 0;
lfs_fs_grow(&lfs, AFTER_BLOCK_COUNT) => 0;
lfs_unmount(&lfs);
if (BLOCK_COUNT != AFTER_BLOCK_COUNT) {
lfs_mount(&lfs, cfg) => LFS_ERR_INVAL;
}
lfs_t lfs2 = lfs;
struct lfs_config cfg2 = *cfg;
cfg2.block_count = AFTER_BLOCK_COUNT;
lfs2.cfg = &cfg2;
lfs_mount(&lfs2, &cfg2) => 0;
lfs_unmount(&lfs2) => 0;
#endif
'''
# shrinking full
[cases.test_shrink_full]
defines.BLOCK_COUNT = [10, 15, 20]
defines.AFTER_BLOCK_COUNT = [5, 7, 10, 12, 15, 17, 20]
defines.FILES_COUNT = [7, 8, 9, 10]
if = "AFTER_BLOCK_COUNT <= BLOCK_COUNT && FILES_COUNT + 2 < BLOCK_COUNT"
code = '''
#ifdef LFS_SHRINKNONRELOCATING
lfs_t lfs;
lfs_format(&lfs, cfg) => 0;
// create FILES_COUNT files of BLOCK_SIZE - 50 bytes (to avoid inlining)
lfs_mount(&lfs, cfg) => 0;
for (int i = 0; i < FILES_COUNT + 1; i++) {
lfs_file_t file;
char path[1024];
sprintf(path, "file_%03d", i);
lfs_file_open(&lfs, &file, path,
LFS_O_WRONLY | LFS_O_CREAT | LFS_O_EXCL) => 0;
char wbuffer[BLOCK_SIZE];
memset(wbuffer, 'b', BLOCK_SIZE);
// Ensure one block is taken per file, but that files are not inlined.
lfs_size_t size = BLOCK_SIZE - 0x40;
sprintf(wbuffer, "Hi %03d", i);
lfs_file_write(&lfs, &file, wbuffer, size) => size;
lfs_file_close(&lfs, &file) => 0;
}
int err = lfs_fs_grow(&lfs, AFTER_BLOCK_COUNT);
if (err == 0) {
for (int i = 0; i < FILES_COUNT + 1; i++) {
lfs_file_t file;
char path[1024];
sprintf(path, "file_%03d", i);
lfs_file_open(&lfs, &file, path,
LFS_O_RDONLY ) => 0;
lfs_size_t size = BLOCK_SIZE - 0x40;
char wbuffer[size];
char wbuffer_ref[size];
// Ensure one block is taken per file, but that files are not inlined.
memset(wbuffer_ref, 'b', size);
sprintf(wbuffer_ref, "Hi %03d", i);
lfs_file_read(&lfs, &file, wbuffer, BLOCK_SIZE) => size;
lfs_file_close(&lfs, &file) => 0;
for (lfs_size_t j = 0; j < size; j++) {
wbuffer[j] => wbuffer_ref[j];
}
}
} else {
assert(err == LFS_ERR_NOTEMPTY);
}
lfs_unmount(&lfs) => 0;
if (err == 0 ) {
if ( AFTER_BLOCK_COUNT != BLOCK_COUNT ) {
lfs_mount(&lfs, cfg) => LFS_ERR_INVAL;
}
lfs_t lfs2 = lfs;
struct lfs_config cfg2 = *cfg;
cfg2.block_count = AFTER_BLOCK_COUNT;
lfs2.cfg = &cfg2;
lfs_mount(&lfs2, &cfg2) => 0;
for (int i = 0; i < FILES_COUNT + 1; i++) {
lfs_file_t file;
char path[1024];
sprintf(path, "file_%03d", i);
lfs_file_open(&lfs2, &file, path,
LFS_O_RDONLY ) => 0;
lfs_size_t size = BLOCK_SIZE - 0x40;
char wbuffer[size];
char wbuffer_ref[size];
// Ensure one block is taken per file, but that files are not inlined.
memset(wbuffer_ref, 'b', size);
sprintf(wbuffer_ref, "Hi %03d", i);
lfs_file_read(&lfs2, &file, wbuffer, BLOCK_SIZE) => size;
lfs_file_close(&lfs2, &file) => 0;
for (lfs_size_t j = 0; j < size; j++) {
wbuffer[j] => wbuffer_ref[j];
}
}
lfs_unmount(&lfs2);
}
#endif
'''

View File

@@ -524,6 +524,114 @@ code = '''
lfs_unmount(&lfs) => 0;
'''
# mount and grow the filesystem
[cases.test_superblocks_shrink]
defines.BLOCK_COUNT = 'ERASE_COUNT'
defines.BLOCK_COUNT_2 = ['ERASE_COUNT/2', 'ERASE_COUNT/4', '2']
defines.KNOWN_BLOCK_COUNT = [true, false]
code = '''
#ifdef LFS_SHRINKNONRELOCATING
lfs_t lfs;
lfs_format(&lfs, cfg) => 0;
if (KNOWN_BLOCK_COUNT) {
cfg->block_count = BLOCK_COUNT;
} else {
cfg->block_count = 0;
}
// mount with block_size < erase_size
lfs_mount(&lfs, cfg) => 0;
struct lfs_fsinfo fsinfo;
lfs_fs_stat(&lfs, &fsinfo) => 0;
assert(fsinfo.block_size == BLOCK_SIZE);
assert(fsinfo.block_count == BLOCK_COUNT);
lfs_unmount(&lfs) => 0;
// same size is a noop
lfs_mount(&lfs, cfg) => 0;
lfs_fs_grow(&lfs, BLOCK_COUNT) => 0;
lfs_fs_stat(&lfs, &fsinfo) => 0;
assert(fsinfo.block_size == BLOCK_SIZE);
assert(fsinfo.block_count == BLOCK_COUNT);
lfs_unmount(&lfs) => 0;
lfs_mount(&lfs, cfg) => 0;
lfs_fs_stat(&lfs, &fsinfo) => 0;
assert(fsinfo.block_size == BLOCK_SIZE);
assert(fsinfo.block_count == BLOCK_COUNT);
lfs_unmount(&lfs) => 0;
// grow to new size
lfs_mount(&lfs, cfg) => 0;
lfs_fs_grow(&lfs, BLOCK_COUNT_2) => 0;
lfs_fs_stat(&lfs, &fsinfo) => 0;
assert(fsinfo.block_size == BLOCK_SIZE);
assert(fsinfo.block_count == BLOCK_COUNT_2);
lfs_unmount(&lfs) => 0;
if (KNOWN_BLOCK_COUNT) {
cfg->block_count = BLOCK_COUNT_2;
} else {
cfg->block_count = 0;
}
lfs_mount(&lfs, cfg) => 0;
lfs_fs_stat(&lfs, &fsinfo) => 0;
assert(fsinfo.block_size == BLOCK_SIZE);
assert(fsinfo.block_count == BLOCK_COUNT_2);
lfs_unmount(&lfs) => 0;
// mounting with the previous size should fail
cfg->block_count = BLOCK_COUNT;
lfs_mount(&lfs, cfg) => LFS_ERR_INVAL;
if (KNOWN_BLOCK_COUNT) {
cfg->block_count = BLOCK_COUNT_2;
} else {
cfg->block_count = 0;
}
// same size is a noop
lfs_mount(&lfs, cfg) => 0;
lfs_fs_grow(&lfs, BLOCK_COUNT_2) => 0;
lfs_fs_stat(&lfs, &fsinfo) => 0;
assert(fsinfo.block_size == BLOCK_SIZE);
assert(fsinfo.block_count == BLOCK_COUNT_2);
lfs_unmount(&lfs) => 0;
lfs_mount(&lfs, cfg) => 0;
lfs_fs_stat(&lfs, &fsinfo) => 0;
assert(fsinfo.block_size == BLOCK_SIZE);
assert(fsinfo.block_count == BLOCK_COUNT_2);
lfs_unmount(&lfs) => 0;
// do some work
lfs_mount(&lfs, cfg) => 0;
lfs_fs_stat(&lfs, &fsinfo) => 0;
assert(fsinfo.block_size == BLOCK_SIZE);
assert(fsinfo.block_count == BLOCK_COUNT_2);
lfs_file_t file;
lfs_file_open(&lfs, &file, "test",
LFS_O_CREAT | LFS_O_EXCL | LFS_O_WRONLY) => 0;
lfs_file_write(&lfs, &file, "hello!", 6) => 6;
lfs_file_close(&lfs, &file) => 0;
lfs_unmount(&lfs) => 0;
lfs_mount(&lfs, cfg) => 0;
lfs_fs_stat(&lfs, &fsinfo) => 0;
assert(fsinfo.block_size == BLOCK_SIZE);
assert(fsinfo.block_count == BLOCK_COUNT_2);
lfs_file_open(&lfs, &file, "test", LFS_O_RDONLY) => 0;
uint8_t buffer[256];
lfs_file_read(&lfs, &file, buffer, sizeof(buffer)) => 6;
lfs_file_close(&lfs, &file) => 0;
assert(memcmp(buffer, "hello!", 6) == 0);
lfs_unmount(&lfs) => 0;
#endif
'''
# test that metadata_max does not cause problems for superblock compaction
[cases.test_superblocks_metadata_max]
defines.METADATA_MAX = [