Compare commits

...

148 Commits

Author SHA1 Message Date
Christopher Haster
ab59ab7f8d Bump default inline_max up to 1/4 block size
As noted by amgross, the current inline_max default (when littlefs
switches from inline files to CTZ skip-lists) does not match the
theoretical value in DESIGN.md.

The reason for this is 1/8 was chosen as a safer default during
development, due to concerns that 1/4 + mdirs splitting at 1/2 would
lead to poor distribution of large inline files in mdirs.

However, two things have happened since then:

1. Experiments have show "wasted" mdir space is less of a concern than
   initially thought. Extra mdir space contributes to logging, delays
   mdir compaction, and can overall lead to better performance.

2. inline_size was added as a configuration option, so if 1/4 is
   problematic users can always override it.

So bumping this back up to 1/4 may make sense.
2025-09-30 12:15:26 -05:00
Christopher Haster
8e251dd675 Merge pull request #1110 from Ryan-CW-Code/perf_gc
perf: gc might try to populate the lookahead buffer each time
2025-06-30 11:39:17 -05:00
Christopher Haster
25b9a4af85 Merge pull request #1109 from Ryan-CW-Code/never_read
refactor: value stored to 'diff' is never read
2025-06-30 11:39:05 -05:00
Christopher Haster
2acf939a00 Merge pull request #1106 from littlefs-project/fix-make-build-dep
make: Add missing BUILD_DEP include
2025-06-30 11:38:56 -05:00
ryancw
d5a86fd28d style: format code, limit to 80 columns. 2025-06-03 09:46:59 +08:00
ryancw
2349ac8c96 perf: gc might try to populate the lookahead buffer each time 2025-05-28 10:23:47 +08:00
ryancw
0755b00c21 refactor: value stored to 'diff' is never read 2025-05-27 20:00:29 +08:00
Christopher Haster
8365bbb7a2 make: Added missing BUILD_DEP include
This was preventing bench modifications from triggering relevant
bench-runner rebuilds.
2025-05-15 13:38:31 -05:00
Christopher Haster
16ceb67934 Merge pull request #1103 from littlefs-project/devel
Minor release: v2.11
2025-05-14 20:45:44 -05:00
Christopher Haster
8434536f0a Bumped minor version to v2.11 2025-05-13 13:18:31 -05:00
Christopher Haster
523319b685 Merge pull request #1104 from DvdGiessen/os-rename-between-filesystems
use shutil.move instead of os.rename to move file
2025-05-13 13:17:53 -05:00
Daniël van de Giessen
ba250a3075 use shutil.move instead of os.rename to move file
This prevents a "OSError: [Errno 18] Invalid cross-device link" if the temporary
file was created on different filesystem (such as a tmpfs mount).
2025-05-13 13:15:21 +02:00
Christopher Haster
8c458fa6bd Merge pull request #1094 from sosthene-nitrokey/shrink-fs
Add support for shrinking a filesystem
2025-05-13 00:45:32 -05:00
Christopher Haster
3149201ae5 Merge pull request #1091 from yamt/mach-o
adapt the linker sections usage to mach-o
2025-05-13 00:45:02 -05:00
Christopher Haster
6a43f3cdc3 Merge pull request #1090 from yamt/clang
drop a few unsupported CFLAGS for clang
2025-05-13 00:44:46 -05:00
Christopher Haster
d73fb8ef3c Merge pull request #1099 from littlefs-project/fix-remove-double-deorphan
Fix double deorphan caused by relocation mid dir remove
2025-05-13 00:44:26 -05:00
Christopher Haster
c1bf7cee84 Merge pull request #1100 from selimkeles/fix/bitshift_overflow
fix: added uint32_t cast to the bitshift places
2025-05-13 00:44:05 -05:00
Christopher Haster
b26bf3494c Merge pull request #1095 from DvdGiessen/lfs_crc
lfs_crc should be static if LFS_CRC is defined
2025-05-13 00:43:24 -05:00
Christopher Haster
0115cf6b74 gha: Dropped explicit CFLAGS from clang testing in CI
Thanks to yamt, GCC-specific flags should now be disabled if compiling
with clang. Dropping the explicit flags also doubles as a test that the
NO_GCC inference works.
2025-05-07 23:45:29 -05:00
Christopher Haster
bff4dfd1b1 Added NO_GCC to allow users to explicitly disable GCC-specific flags
This is the same as the implicit Clang => NO_GCC behavior introduced by
yamt, but with an explicit variable that can be assigned by users using
other, non-gcc, compilers:

  $ NO_GCC=1 make

Note, stack measurements are currently GCC specific:

  $ NO_GCC=1 make stack
  ... snip ...
  FileNotFoundError: [Errno 2] No such file or directory: 'lfs.ci'
  make: *** [Makefile:494: lfs.stack.csv] Error 1
2025-05-07 23:40:25 -05:00
Sosthène Guédon
edaaaf88ea Apply review comments 2025-05-07 10:38:43 +02:00
Sosthène Guédon
7d79423972 Rename SHRINKIFCHEAP to SHRINKNONRELOCATING 2025-05-07 10:34:24 +02:00
Sosthène Guédon
7782d3dfa3 Mention that shrinking is unlikely to work 2025-05-06 11:00:29 +02:00
selim.keles
f4a1bb328a fix: added uint32_t cast to the bitshift places
In 16 bit and 8 bit architectures, overflow and underflow issues were occuring while using functions lfs_frombe32 and lfs_fromle32
2025-05-05 13:38:05 +03:00
Sosthène Guédon
9b8f802b43 fixup! Add support for shrinking a filesystem 2025-05-05 11:37:39 +02:00
Christopher Haster
a3d6bec5f0 Fixed a double deorphan caused by relocation mid dir remove
Long story short: There is a specific case where removing a directory
can trigger a deorphan pass, but lfs_remove did not check for this,
would try to clean up the (already cleaned) directory orphan, and
trigger an assert:

  lfs.c:4890:assert: assert failed with false, expected eq true
      LFS_ASSERT(lfs_tag_size(lfs->gstate.tag) > 0x000 || orphans >= 0);

The specific case being a remove commit that triggers a relocation that
creates an orphan.

This is also possible in lfs_rename, but only if you're renaming a
directory that implies a remove, which is a pretty rare operation.

---

This was probably an oversight introduced in the non-recursive commit
logic rework.

Fortunately the fix is to just check if we even have an orphan before
trying to remove it. We can rely on this instead of the file type, so
this fix shouldn't even increase the code size.

Found and root-caused by Hugh-Baoa
2025-05-03 18:13:19 -05:00
Christopher Haster
0634d13e07 tests: Added non-reentrant variants of orphan/relocation tests
These are the same as the related reentrant variants, but by opting out
of powerloss testing, we can test a much larger number of states without
having to worry about the impact on powerloss testing runtime.

Bumped CYCLES from 20 -> 2000.

This reveals an orphan remove bug found by Hugh-Baoa.
2025-05-03 17:15:26 -05:00
Sosthène Guédon
2105e502c5 Add support for shrinking a filesystem
This PR adds a new `lfs_fs_shrink`, which functions similarly to
`lfs_fs_grow`, but supports reducing the block count.

This functions first checks that none of the removed block are in use.
If it is the case, it will fail.
2025-04-17 10:07:37 +02:00
Daniël van de Giessen
b823728420 lfs_crc should be static if LFS_CRC is defined 2025-04-16 18:17:21 +02:00
YAMAMOTO Takashi
0d861b7916 adapt the linker sections usage to mach-o
"make test" on macOS:

```
using runner: ./runners/test_runner
found 19 suites, 188 cases, 11242/11770 permutations

running test_alloc: 12/12 cases, 207/207 perms
running test_attrs: 4/4 cases, 20/20 perms
running test_badblocks: 4/4 cases, 300/300 perms
running test_bd: 5/5 cases, 85/85 perms
running test_compat: 17/17 cases, 205/205 perms
running test_dirs: 15/15 cases, 450/450 perms, 1756pls!
running test_entries: 8/8 cases, 32/32 perms
running test_evil: 8/8 cases, 105/105 perms
running test_exhaustion: 5/5 cases, 85/85 perms
running test_files: 10/10 cases, 7155/7155 perms, 9410pls!
running test_interspersed: 4/4 cases, 190/190 perms, 2835pls!
running test_move: 17/17 cases, 161/161 perms, 157pls!
running test_orphans: 6/6 cases, 50/50 perms, 846pls!
running test_paths: 33/33 cases, 325/325 perms
running test_powerloss: 2/2 cases, 21/21 perms
running test_relocations: 4/4 cases, 68/68 perms, 1612pls!
running test_seek: 10/10 cases, 195/195 perms, 1050pls!
running test_superblocks: 17/17 cases, 318/318 perms, 1437pls!
running test_truncate: 7/7 cases, 1270/1270 perms, 9691pls!

done: 11242/11242 passed, 0/11242 failed, 28794pls!, in 585.76s
```
2025-04-07 16:20:23 +09:00
YAMAMOTO Takashi
26bee8ad36 drop a few unsupported CFLAGS for clang 2025-04-07 16:06:01 +09:00
Christopher Haster
8ed63b27be Merge pull request #1084 from elupus/fix/packing
fix: avoid assuming struct packing
2025-03-20 01:26:11 -05:00
Christopher Haster
a666730044 Merge pull request #1078 from BrianPugh/unit-test-readme
Add a little bit of documentation on how to run tests.
2025-03-20 01:25:56 -05:00
Christopher Haster
47e738b788 Merge pull request #1071 from RocLoong/patch-1
print lfs_file_size overflow
2025-03-20 01:25:33 -05:00
Christopher Haster
81b0db0cdc Merge pull request #1070 from Noxet/filebd-wrong-cast
Changed cast to correct type when trace is enabled for filebd
2025-03-20 01:24:19 -05:00
Christopher Haster
63ab1ffb65 Merge pull request #1068 from littlefs-project/fix-dir-remove-read
Fix dir iteration being broken by concurrent removes
2025-03-20 01:24:04 -05:00
Christopher Haster
ca1081e7c4 Merge pull request #1065 from amubiera/fix-unsafe-use-of-bool
Fix for "unsafe use of type bool" warning when compiling with MSVC.
2025-03-20 01:23:35 -05:00
Christopher Haster
76027f1502 Merge pull request #1064 from tim-nordell-nimbelink/fix/script_syntax_warnings
scripts: Fixed several SyntaxWarning for python test helpers
2025-03-20 01:23:19 -05:00
Christopher Haster
61a1b0b496 Tweaked lfs_gstate_iszero for terseness 2025-03-18 02:39:28 -05:00
Joakim Plate
ffafb9cbb1 fix: avoid assuming struct packing
lfs_gstate_t was assumed to be a packed array of uint32_t,
but this is not always guaranteed. Access the fields directly
instead of attempting to loop over an array of uint32_t

Fixes clang tidy warnings about use of uninitialized memory
accessed.
2025-03-14 10:03:46 +01:00
Christopher Haster
5281a20f6c README.md: Tweaked testing documentation
- Showing some of the more useful flags.
- Showing the usual flow of bug -> reproduce -> gdb.
- Being a bit pedantic since this is the README.md.
2025-03-13 13:23:20 -05:00
Brian Pugh
f55520380d Add a little bit of documentation on how to run tests. 2025-02-27 17:41:29 -08:00
Rocloong
936919d134 LFS_TRACE: Fixed sign mismatch in lfs_file_size 2025-02-13 15:46:39 -06:00
Christopher Haster
d2c3a47627 gha: Added test-yes-trace build/test job to CI
To hopefully catch typos like the one found by Noxet in the future.

Nothing is actually testing that these trace statements compile
otherwise.
2025-02-06 01:20:29 -06:00
Jonathan Sönnerup
0320e7db0e Changed cast to correct type when trace is enabled for filebd 2025-02-05 16:16:53 +01:00
Christopher Haster
caba4f31df Fixed dir iteration being broken by concurrent removes
When removing a file, we mark all open handles as "removed" (
pair={-1,-1}) to avoid trying to later read metadata that no longer
exists. Unfortunately, this also includes open dir handles that happen
to be pointing at the removed file, causing them to return
LFS_ERR_CORRUPT on the next read.

The good news is this is _not_ actual filesystem corruption, only a
logic error in lfs_dir_read.

We actually already have logic in place to nudge the dir to the next id,
but it was unreachable with the existing logic. I suspect this worked at
one point but was broken during a refactor due to lack of testing.

---

Fortunately, all we need to do is _not_ clobber the handle if the
internal type is a dir. Then the dir-nudging logic can correctly take
over.

I've also added test_dirs_remove_read to test this and prevent another
regression, adapted from tests provided by tpwrules that identified the
original bug.

Found by tpwrules
2025-02-03 22:52:24 -06:00
Amilcar Ubiera
152d03043c Fix for "unsafe use of type bool" warning when compiling with MSVC. 2025-02-03 18:59:14 -05:00
Tim Nordell
8d01895b32 scripts: Fixed several SyntaxWarning for python test helpers
Many of these require a r'' string context to avoid errors like:

  scripts/test.py:105: SyntaxWarning: invalid escape sequence '\s'
2025-01-13 16:54:13 -06:00
Christopher Haster
0494ce7169 Merge pull request #1058 from littlefs-project/fix-seek-eob-cache
Fixed incorrect cache reuse when seeking from end-of-block
2024-12-20 09:02:13 -06:00
Christopher Haster
366100b140 Fixed incorrect cache reuse when seeking from end-of-block
In v2.5, we introduced an optimization to avoid rereading data when
seeking inside the file cache. Unfortunately this used a slightly
wrong condition to check if the cache was "live", which meant seeks from
end-of-blocks could end up with invalid caches and wrong data. Not
great.

The problem is the nuance of when a file's cache is "live":

1. The file is marked as LFS_F_READING or LFS_F_WRITING.

   But we can't reuse the cache when writing, so we only care about
   LFS_F_READING.

2. file->off != lfs->cfg->block_size (end-of-block).

   This is an optimization to avoid eagerly reading blocks we may not
   actually care about.

We weren't checking for the end-of-block case, which meant if you seeked
_from_ the end of a block to a seemingly valid location in the file
cache, you could end up with an invalid cache.

Note that end-of-block may not be powers-of-two due to CTZ skip-list
pointers.

---

The fix is to check for the end-of-block case in lfs_file_seek. Note
this now matches the need-new-block logic in lfs_file_flushedread.

This logic change may also make lfs_file_seek call lfs_file_flush more
often, but only in cases where lfs_file_flush is a noop.

I've also extended the test_seek tests to cover a few more boundary-read
cases and prevent a regression in the future.

Found by wjl and lrodorigo
2024-12-19 02:39:10 -06:00
Christopher Haster
630a0d87c2 Merge pull request #1050 from littlefs-project/devel
Minor release: v2.10
2024-12-11 16:56:45 -06:00
Christopher Haster
3d0386489b Bumped minor version to v2.10 2024-12-11 16:23:10 -06:00
Christopher Haster
b8e4433b34 Merge pull request #1052 from wangdongustc/assert_null_sync
Assert on NULL IO functions
2024-12-10 11:48:48 -06:00
Dong Wang
dae656aa53 Fix prettyasserts.py for pointer asserts 2024-12-10 22:54:58 +08:00
Dong Wang
469c863c18 Assert on NULL IO function 2024-12-10 22:54:54 +08:00
Christopher Haster
215613e41f gha: Fixed x86-only statuses
Looks like I missed a line during refactoring, resulted in only x86
sizes being reported in GitHub statuses.

If we wanted to limited these to one architecture, thumb would have
probably been a better pick.
2024-12-09 14:56:12 -06:00
Christopher Haster
2fcecc8894 Merge pull request #1046 from littlefs-project/fix-trailing-slashes
paths: Revisit path parsing, fix trailing slash behavior
2024-12-06 13:48:26 -06:00
Christopher Haster
78f9a5fcd3 Merge pull request #1038 from littlefs-project/link-ramcrc32bd-ramrsbd
Add links to ramcrc32bd and ramrsbd
2024-12-06 13:47:47 -06:00
Christopher Haster
83fe41b605 Merge pull request #1031 from littlefs-project/fix-enospc-issues
Fix metadata_max==prog_size commit->end calculation
2024-12-06 13:47:36 -06:00
Christopher Haster
d7a911923b Merge pull request #1027 from littlefs-project/fix-seek-overflow-ub
Fix seek undefined behavior on signed integer overflow
2024-12-06 13:47:20 -06:00
Christopher Haster
2ba4280a5e Merge pull request #997 from littlefs-project/fix-trace-format-again
Fix some more LFS_TRACE format specifiers
2024-12-06 13:47:06 -06:00
Christopher Haster
c961e1fe66 Merge pull request #1004 from yamt/user-define-header
Add an alternative way to override LFS_MALLOC etc
2024-12-06 13:45:56 -06:00
Christopher Haster
bd01a4c0ee Merge pull request #1013 from wdfk-prog/feature_2.9.3
Write the detect cycles function as a function to optimize code
2024-12-06 13:44:37 -06:00
Christopher Haster
999ef6656f paths: Changed CREAT with a trailing slash to return NOTDIR
- before: lfs_file_open("missing/") => LFS_ERR_ISDIR
- after:  lfs_file_open("missing/") => LFS_ERR_NOTDIR

As noted by bmcdonnell-fb, returning LFS_ERR_ISDIR here was inconsistent
with the case where the file exists:

  case                           before          after
  lfs_file_open("dir_a")      => LFS_ERR_ISDIR   LFS_ERR_ISDIR
  lfs_file_open("dir_a/")     => LFS_ERR_ISDIR   LFS_ERR_ISDIR
  lfs_file_open("reg_a/")     => LFS_ERR_NOTDIR  LFS_ERR_NOTDIR
  lfs_file_open("missing_a/") => LFS_ERR_ISDIR   LFS_ERR_NOTDIR

Note this is consistent with the behavior of lfs_stat:

  lfs_file_open("reg_a/") => LFS_ERR_NOTDIR
  lfs_stat("reg_a/")      => LFS_ERR_NOTDIR

And the only other function that can "create" files, lfs_rename:

  lfs_file_open("missing_a/")       => LFS_ERR_NOTDIR
  lfs_rename("reg_a", "missing_a/") => LFS_ERR_NOTDIR

There is some ongoing discussion about if these should return NOTDIR,
ISDIR, or INVAL, but this is at least an improvement over the
rename/open mismatch.
2024-11-25 15:40:44 -06:00
Christopher Haster
b735c8fd7f paths: Added tests over NOENT + trailing slash/dot
- test_paths_noent_trailing_slashes
- test_paths_noent_trailing_dots
- test_paths_noent_trailing_dotdots

These managed to slip through our path testing but should be tested, if
anything just to know exactly what errors these return.
2024-11-25 15:40:15 -06:00
Christopher Haster
30947054d4 paths: Extended tests to cover open with CREAT/EXCL
These flags change the behavior of open quite significantly. It's useful
to cover these in our path tests so the behavior is locked down.
2024-11-25 15:40:15 -06:00
Christopher Haster
80ca1ea300 paths: Reject empty paths
Before this, the empty path ("") was treated as an alias for the root.
This was unintentional and just a side-effect of how the path parser
worked.

Now, the empty path should always result in LFS_ERR_INVAL:

- before: lfs_stat("") => 0
- after:  lfs_stat("") => LFS_ERR_INVAL
2024-11-25 15:40:15 -06:00
Christopher Haster
815f0d85a5 paths: Fixed dots followed by dotdots
Unlike normal files, dots (".") should not change the depth when
attempting to skip dotdot ("..") entries.

A weird nuance in the path parser, but at least it had a relatively easy
fix.

Added test_paths_dot_dotdots to prevent a regression.
2024-11-25 15:40:15 -06:00
Christopher Haster
dc92dec6d3 paths: Reject dotdots above root
This changes the behavior of paths that attempt to navigate above root
to now return LFS_ERR_INVAL:

- before: lfs_stat("/../a") => 0
- after:  lfs_stat("/../a") => LFS_ERR_INVAL

This is a bit of an opinionated change while making other path
resolution tweaks.

In terms of POSIX-compatibility, it's a bit unclear exactly what dotdots
above the root should do.

POSIX notes:

> As a special case, in the root directory, dot-dot may refer to the
> root directory itself.

But the word choice of "may" implies it is up to the implementation.

I originally implement this as a root-loop simply because that is what
my Linux machine does, but I now think that's not the best option. Since
we're making other path-related tweaks, we might as well try to adopt
behavior that is, in my opinion, safer and less... weird...

This should also help make paths more consistent with future theoretical
openat-list APIs, where saturating at the current directory is sort of
the least expected behavior.
2024-11-25 15:40:07 -06:00
Christopher Haster
a6035071be paths: Fixed/doc trailing slash/dot POSIX incompatibilities
- lfs_mkdir now accepts trailing slashes:
  - before: lfs_mkdir("a/") => LFS_ERR_NOENT
  - after:  lfs_mkdir("a/") => 0

- lfs_stat, lfs_getattr, etc, now reject trailing slashes if the file is
  not a directory:
  - before: lfs_stat("reg_a/") => 0
  - after:  lfs_stat("reg_a/") => LFS_ERR_NOTDIR

  Note trailing slashes are accepted if the file is a directory:
  - before: lfs_stat("dir_a/") => 0
  - after:  lfs_stat("dir_a/") => 0

- lfs_file_open now returns LFS_ERR_NOTDIR if the file exists but the
  path contains trailing slashes:
  - before: lfs_file_open("reg_a/") => LFS_ERR_NOENT
  - after:  lfs_file_open("reg_a/") => LFS_ERR_NOTDIR

To make these work, the internal lfs_dir_find API required some
interesting changes:

- lfs_dir_find no longer sets id=0x3ff on not finding a parent entry in
  the path. Instead, lfs_path_islast can be used to determine if the
  modified path references a parent entry or child entry based on the
  remainder of the path string.

  Note this is only necessary for functions that create new entries
  (lfs_mkdir, lfs_rename, lfs_file_open).

- Trailing slashes mean we can no longer rely on the modified path being
  NULL-terminated. lfs_path_namelen provides an alternative to strlen
  that stops at slash or NULL.

- lfs_path_isdir also tells you if the modified path must reference a
  dir (contains trailing slashes). I considered handling this entirely
  in lfs_dir_find, but the behavior of entry-creating functions is too
  nuanced.

  At least lfs_dir_find returns LFS_ERR_NOTDIR if the file exists on
  disk.

Like strlen, lfs_path_namelen/islast/isdir are all O(n) where n is the
name length. This isn't great, but if you're using filenames large
enough for this to actually matter... uh... open an issue on GitHub and
we might improve this in the future.

---

There are a couple POSIX incompatibilities that I think are not
worth fixing:

- Root modifications return EINVAL instead of EBUSY:
  - littlefs: remove("/") => EINVAL
  - POSIX:    remove("/") => EBUSY
  Reason: This would be the only use of EBUSY in the system.

- We accept modifications of directories with trailing dots:
  - littlefs: remove("a/.") => 0
  - POSIX:    remove("a/.") => EBUSY
  Reason: Not worth implementing.

- We do not check for existence of directories followed by dotdots:
  - littlefs: stat("a/missing/..") => 0
  - POSIX:    stat("a/missing/..") => ENOENT
  Reason: Difficult to implement non-recursively.

- We accept modifications of directories with trailing dotdots:
  - littlefs: rename("a/b/..", "c") => 0
  - POSIX:    rename("a/b/..", "c") => EBUSY
  Reason: Not worth implementing.

These are at least now documented in tests/test_paths.toml, which isn't
the greatest location, but it's at least something until a better
document is created.

Note that these don't really belong in SPEC.md because path parsing is
a function of the driver and has no impact on disk.
2024-11-25 15:39:29 -06:00
Christopher Haster
232e736aae paths: Added trailing slashes and dots tests
As expected these are failing and will need some work to pass.

The issue with lfs_file_open allowing trailing slashes was found by
rob-zeno, and the issue with lfs_mkdir disallowing trailing slashes was
found by XinStellaris, PoppaChubby, pavel-kirienko, inf265, Xywzel,
steverpalmer, and likely others.
2024-11-23 19:03:36 -06:00
Christopher Haster
0de0389c6f paths: Reworked test_paths to cover more corner cases
This should be a superset of the previous test_paths test suite, while
covering a couple more things (more APIs, more path synonyms, utf8,
non-printable ascii, non-utf8, etc).

Not yet tested are some corner cases with known bugs, mainly around
trailing slashes.
2024-11-23 18:20:06 -06:00
Christopher Haster
1407db9556 Added links to ramcrc32bd and ramrsbd
These two small libraries provide examples of error-correction
compatible with littlefs (or any filesystem really).

It would be nice to eventually provide these as drop-in solutions, but
right now it's not really possible without breaking changes to
littlefs's block device API.

In the meantime, ramcrc32bd and ramrsbd at least provide example
implementations that can be adapted to users' own block devices.
2024-11-01 17:09:45 -05:00
Christopher Haster
ea431bd6ae Added some checks that metadata_max makes sense
Like the read/prog/block_size checks, these are just asserts. If these
invariants are broken the filesystem will break in surprising ways.
2024-10-04 13:45:57 -05:00
Christopher Haster
2d62d2f4c9 Fixed metadata_max==prog_size commit->end calculation
The inconsistency here between the use of block_size vs metadata_max was
suspicious. Turns out there's a bug when metadata_max == prog_size.

We correctly use metadata_max for the block_size/2 check, but we weren't
using it for the block_size-40 check. The second check seems unnecessary
after the first, but it protects against running out of space in a
commit for commit-related metadata (checksums, tail pointers, etc) when
we can't program half-blocks.

Turns out this is also needed when limiting metadata_max to a single
prog, otherwise we risk erroring with LFS_ERR_NOSPC early.

Found by ajheck, dpkristensen, NLLK, and likely others.
2024-10-04 13:45:43 -05:00
Christopher Haster
1f82c0f27f Added some metadata_max testing
- Added METADATA_MAX to test_runner.
- Added METADATA_MAX to bench_runner.
- Added a simple metadata_max test to test_superblocks, for lack of
  better location.

There have been several issues floating around related to metadata_max
and LFS_ERR_NOSPC which makes me think there's a bug in our metadata_max
logic.

metadata_max was a quick patch and is relatively untested, so an
undetected bug isn't too surprising. This commit adds at least some
testing over metadata_max.

Sure enough, the new test_superblocks_metadata_max test reveals a
curious LFS_ERR_NAMETOOLONG error that shouldn't be there.

More investigation needed.
2024-10-04 13:06:23 -05:00
wdfk-prog
a2c2e49e6b Write the detect cycles function as a function to optimize code 2024-10-04 10:37:25 +08:00
Christopher Haster
abaec45652 Fixed seek undefined behavior on signed integer overflow
In the previous implementation of lfs_file_seek, we calculated the new
offset using signed arithmetic before checking for possible
overflow/underflow conditions. This results in undefined behavior in C.

Fortunately for us, littlefs is now limited to 31-bit file sizes for API
reasons, so we don't have to be too clever here. Doing the arithmetic
with unsigned integers and just checking if we're in a valid range
afterwards should work.

Found by m-kostrzewa and lucic71
2024-09-24 14:01:20 -05:00
Christopher Haster
f1c430e779 Added some tests around seek integer overflow/underflow
Original tests provided by m-kostrzewa, these identify signed overflow
(undefined behavior) when compiled with -fsanitize=undefined.
2024-09-24 14:01:08 -05:00
YAMAMOTO Takashi
4a845be0be Rename LFS_USER_DEFINES to LFS_DEFINES 2024-09-24 12:29:13 -05:00
YAMAMOTO Takashi
e1636d05ab Add an alternative way to override LFS_MALLOC etc
With the existing method, (-DLFS_MALLOC=my_malloc)
users often had to use compiler options like -include, which
was not so portable.
This change introduces another way to provide partial overrides of
lfs_util.h using a user-provided header.
2024-09-24 12:29:13 -05:00
Christopher Haster
b78afe2518 Merge pull request #1026 from yamt/update-gh-actions
Update github actions to the latest versions
2024-09-24 12:25:04 -05:00
Christopher Haster
798073c2a7 gha: Dropped minor/patch version pinning of actions
With GitHub forcibly deprecating old versions of actions, pinning the
minor/patch version is more likely to cause breakage than not.
2024-09-20 16:05:15 -05:00
Christopher Haster
7db9e1663a gha: Switched to standard da for cross-workflow downloads
Looks like cross-workflow downloads has finally been added to the
standard download-artifact action, so we might as well switch to it to
reduce dependencies.

dawidd6's version was also missing the merge-multiple feature which is
necessary to work around breaking changes in download-artifact's v4
bump.

Weirdly it needs GITHUB_TOKEN for some reason? Not sure why this
couldn't be implicit.
2024-09-20 16:05:12 -05:00
Christopher Haster
2c4b262c35 gha: Merge artifacts on download
Turns out major versions break things.

Old behavior: Artifacts with same name are merged
New behavior: Artifacts with same name error

Using a pattern and merging on download should fix this at least on the
job-side. Though I do wonder if we'll start running into artifact limit
issues with the new way artifacts are handled...
2024-09-20 16:04:35 -05:00
YAMAMOTO Takashi
72a4b57f4e gha: Make the artifact names unique 2024-09-19 17:26:49 -05:00
YAMAMOTO Takashi
6e7269890a gha: Update github actions to the latest versions 2024-09-19 17:18:15 -05:00
Christopher Haster
ac207586ba Fixed some more LFS_TRACE format specifiers
- block_cycles is signed and should use PRId32
- flags is signed (which is a bit weird) and should be cast for %x

Unfortunately exactly what PRI* expands to is dependant on both the
compiler and the underlying architecture, so I don't think it's possible
for us to catch these mistakes with CI...

Found by stefano-zanotti
2024-06-25 16:08:00 -05:00
Christopher Haster
d01280e649 Merge pull request #968 from littlefs-project/link-pico-littlefs-usb
Add links to pico-littlefs-usb (FAT12 emulation) and mklittlefs
2024-04-29 16:21:49 -05:00
Christopher Haster
6e52140d51 Merge pull request #959 from littlefs-project/fix-expanded-magic
Duplicate the superblock entry during superblock expansion, fix missing magic
2024-04-29 14:26:38 -05:00
Christopher Haster
0bbb8bc88b Reorganized external project links a bit
These were grouped up a bit better at one point, but that sort of
drifted as new project were added:

1. Official repos (mainly littlefs-fuse)
2. Non-C reimplementations/wrappers
3. Utilities
4. Non-littlefs related projects

Eventually, maybe when these move out of the README.md, these categories
should probably be actually codified as headers or something.
2024-04-17 13:46:33 -05:00
Christopher Haster
78082336e7 Added a link to mklittlefs
Implemented by earlephilhower, mklittlefs is a command line interface
that seems to be used by the ESP8266 and RP2040 ecosystems. It deserves
a mention.

Also tweaked mklfs's description a bit.
2024-04-17 13:39:11 -05:00
Christopher Haster
8336ecd203 Added a link to pico-littlefs-usb (FAT12 emulation)
Implemented by oyama, pico-littlefs-usb provides an easy interface to
littlefs by emulating a FAT12 filesystem over USB.

There are some tradeoffs to this, but being able to mount a littlefs
device without installing additional drivers is very nice. Maybe in the
future devices could provide both a FAT and raw endpoint for
easy/advanced filesystem access.
2024-04-17 13:09:04 -05:00
Christopher Haster
68d28b5114 Merge pull request #966 from BrianPugh/fix-divide-by-zero-full-filesystem
Fix DivideByZero exception when filesystem is completely full.
2024-04-17 12:38:22 -05:00
Christopher Haster
1bc14933b7 Tweaked on-disk config comments for consistency
- Prefer "defaults to blablabla when zero" to hint that this is the
  default state when both explicitly set to zero and implicitly set to
  zero thanks to C's initializers.

- Prefer "disk" when referencing something stored "on disk". Other terms
  can quickly get ambiguous. Except maybe "block device"...
2024-04-17 00:16:20 -05:00
Christopher Haster
01b6a47ea8 Extended test_alloc to test inferred block_count
The block allocator is an area where inferred block counts (when
cfg.block_count=0) are more likely to cause problems.

As is shown by the recent divide-by-zero-exhaustion issue.
2024-04-17 00:04:56 -05:00
Brian Pugh
749a45650f Fix DivideByZero exception when filesystem is completely full. 2024-04-16 20:32:12 -07:00
Christopher Haster
11b036cc6c Prevented unnecessary superblock rewrites if old version in superblock chain
Because multiple, out-of-date superblocks can exist in our superblock
chain, we need to be careful to make sure newer superblock entries
override older superblock entries.

If we see an older on-disk minor version in the superblock chain, we
were correctly overriding the on-disk minor version, but we were also
leaving the "needs superblock" bit set in our consistency state.

This isn't a hard-error, but would lead to a superblock rewrite every
mount. The rewrite would make no progress, as the out-of-date version is
effectively immutable at this point, and just waste prog cycles.

This should fix that by clearing the "needs superblock" bit if we see a
newer on-disk minor version.
2024-03-19 00:49:28 -05:00
Christopher Haster
25ee90fdf1 Clarified what is accessible at specific superblock offsets in SPEC.md
It used to be the case that the entire superblock entry could be found
at specific offsets, but this was only possible while the superblock
entry was immutable. Now that the superblock entry is very mutable
(block-count changes, lfs2.0 -> lfs2.1 version bumps, etc), the correct
superblock entry may end up later in the metadata log.

At the very least, the "littlefs" magic string is still immutable and at
the specific offset offset=8. This is arguably the most useful
fixed-offset item.
2024-03-19 00:49:28 -05:00
Christopher Haster
a60a986c9c Duplicate the superblock entry during superblock expansion
The documentation does not match the implementation here. The intended
behavior of superblock expansion was to duplicate the current superblock
entry into the new superblock:

   .--------.  .--------.
  .|littlefs|->|littlefs|
  ||bs=4096 | ||bs=4096 |
  ||bc=256  | ||bc=256  |
  ||crc32   | ||root dir|
  ||        | ||crc32   |
  |'--------' |'--------'
  '--------'  '--------'

The main benefit is that we can rely on the magic string "littlefs"
always residing in blocks 0x{0,1}, even if the superblock chain has
multiple superblocks.

The downside is that earlier superblocks in the superblock chain may
contain out-of-date configuration. This is a bit annoying, and risks
hard-to-reach bugs, but in theory shouldn't break anything as long as
the filesystem is aware of this.

Unfortunately this was lost at some point during refactoring in the
early v2-alpha work. A lot of code was moving around in this stage, so
it's a bit hard to track down the change and if it was intentional. The
result is superblock expansion creates a valid linked-list of
superblocks, but only the last superblock contains a valid superblock
entry:

   .--------.  .--------.
  .|crc32   |->|littlefs|
  ||        | ||bs=4096 |
  ||        | ||bc=256  |
  ||        | ||root dir|
  ||        | ||crc32   |
  |'--------' |'--------'
  '--------'  '--------'

What's interesting is this isn't invalid as far as lfs_mount is
concerned. lfs_mount is happy as long as a superblock entry exists
anywhere in the superblock chain. This is good for compat flexibility,
but is the main reason this has gone unnoticed for so long.

---

With the benefit of more time to think about the problem, it may have
been more preferable to copy only the "littlefs" magic string and NOT
the superblock entry:

   .--------.  .--------.
  .|littlefs|->|littlefs|
  ||crc32c  | ||bs=4096 |
  ||        | ||bc=256  |
  ||        | ||root dir|
  ||        | ||crc32   |
  |'--------' |'--------'
  '--------'  '--------'

This would allow for simple "littlefs" magic string checks without the
risks associated with out-of-date superblock entries.

Unfortunately the current implementation errors if it finds a "littlefs"
magic string without an associated superblock entry, so such a change
would not be compatible with old drivers.

---

This commit tweaks superblock expansion to duplicate the superblock
entry instead of simply moving it to the new superblock. And adds tests
over the magic string "littlefs" both before and after superblock
expansion.

Found by rojer and Nikola Kosturski
2024-03-19 00:48:56 -05:00
Christopher Haster
4dd30c1b8f Merge pull request #948 from littlefs-project/fix-sync-ordering
Fix sync issue where data writes could appear before metadata writes
2024-03-08 16:49:59 -06:00
Christopher Haster
5c0d332ecd Merge pull request #939 from Graveflo/master
Add nim-littlefs to readme
2024-03-08 16:49:11 -06:00
Christopher Haster
cf68333a55 Merge pull request #937 from littlefs-project/fix-pending-rm-get-underflow
Fix synthetic move underflows in lfs_dir_get
2024-03-08 16:48:50 -06:00
Christopher Haster
7873d811a0 Fixed memory leak in emubd's out-of-order write emulation
We need to decrement the saved block state on sync, when we reset
out-of-order emulation. Otherwise we leak blocks out the wazoo.
2024-02-27 21:39:34 -06:00
Christopher Haster
fc2aa3350c Fixed issue with exhaustive + out-of-order powerloss testing
Unlike the heuristic based testing, exhaustive powerloss testing
effectively forks the current test and runs both the interrupted and
uninterrupted test states to completion. But emubd wasn't expecting
bd->cfg->powerloss_cb to return.

The fix here is to keep track to both the old+new out-of-order block
states and unrevert them if bd->cfg->powerloss_cb returns.

This may leak the temporary copy, but powerloss testing is already
inherently leaky.
2024-02-27 21:14:59 -06:00
Christopher Haster
6352185949 Fixed sync issue where data writes could appear before metadata writes
Long story short we aren't calling sync correctly in littlefs. This
fixes that.

Some forms of storage, mainly anything with an FTL, eMMC, SD, etc, do
not guarantee a strict write order for writes to different blocks. In
theory this is what bd sync is for, to tell the bd when it is important
for the writes to be ordered.

Currently, littlefs calls bd sync after committing metadata. This is
useful as it ensures that user code can rely on lfs_file_sync for
ordering external side-effects.

But this is insufficient for handling storage with out-of-order writes.

Consider the simple case of a file with one data block:

1. lfs_file_write(blablabla) => writes data into a new data block

2. lfs_file_sync() => commits metadata to point to the new data block

But with out-of-order writes, the bd is free to reorder things such that
the metadata is updated _before_ the data is written. If we lose power,
that would be bad.

The solution to this is to call bd sync twice: Once before we commit
the metadata to tell the bd that these writes must be ordered, and once
after we commit the metadata to allow ordering with user code.

As a small optimization, we only call bd sync if the current file is not
inlined and has actually been modified (LFS_F_DIRTY). It's possible for
inlined files to be interleaved with writes to other files.

Found by MFaehling and alex31
2024-02-27 14:00:10 -06:00
Christopher Haster
f2a6f45eef Added out-of-order write testing to emubd
Some forms of storage, mainly anything with an FTL, eMMC, SD, etc, do
not guarantee a strict write order for writes to different blocks. It
would be good to test that this doesn't break littlefs.

This adds LFS_EMUBD_POWERLOSS_OOO to lfs_emubd, which tells lfs_emubd to
try to break any order-dependent code on powerloss.

The behavior right now is a bit simple, but does result in test
breakage:

1. Save the state of the block on first write (erase really) after
   sync/init.

2. On powerloss, revert the first write to its original state.

This might be a bit confusing when debugging, since the block will
appear to time-travel, but doing anything fancier would make emubd quite
a bit more complicated.

You could also get a bit fancier with which/how many blocks to revert,
but this should at least be sufficient to make sure bd sync calls are in
the right place.
2024-02-27 13:59:37 -06:00
Ryan McConnell
2752d8c486 add nim-littlefs to readme 2024-02-07 02:53:16 -05:00
Christopher Haster
ddbfcaa722 Fixed synthetic move underflows in lfs_dir_get
By "luck" the previous code somehow managed to not be broken, though it
was possible to traverse the same file twice in lfs_fs_traverse/size
(which is not an error).

The problem was an underlying assumption in lfs_dir_get that it would
never be called when the requested id is pending removal because of a
powerloss. The assumption was either:

1. lfs_dir_find would need to be called first to find the id, and it
   would correctly toss out pending-rms with LFS_ERR_NOENT.

2. lfs_fs_mkconsistent would be implicitly called before any filesystem
   traversals, cleaning up any pending-rms. This is at least true for
   allocator scans.

But, as noted by andriyndev, both lfs_fs_traverse and lfs_fs_size can
call lfs_fs_get with a pending-rm id if called in a readonly context.

---

By "luck" this somehow manages to not break anything:

1. If the pending-rm id is >0, the id is decremented by 1 in lfs_fs_get,
   returning the previous file entry during traversal. Worst case, this
   reports any blocks owned by the previous file entry twice.

   Note this is not an error, lfs_fs_traverse/size may return the same
   block multiple times due to underlying copy-on-write structures.

2. More concerning, if the pending-rm id is 0, the id is decremented by
   1 in lfs_fs_get and underflows. This underflow propagates into the
   type field of the tag we are searching for, decrementing it from
   0x200 (LFS_TYPE_STRUCT) to 0x1ff (LFS_TYPE_INTERNAL(UNUSED)).

   Fortunately, since this happens to underflow to the INTERNAL tag
   type, the type intended to never exist on disk, we should never find
   a matching tag during our lfs_fs_get search. The result? lfs_dir_get
   returns LFS_ERR_NOENT, which is actually what we want.

Also note that LFS_ERR_NOENT does not terminate the mdir traversal
early. If it did we would have missed files instead of duplicating
files, which is a slightly worse situation.

---

The fix is to add an explicit check for pending-rms in lfs_dir_get, just
like in lfs_dir_find. This avoids relying on unintended underflow
propagation, and should make the internal API behavior more consistent.

This is especially important for potential future gc extensions.

Found by andriyndev
2024-02-04 15:12:31 -06:00
Christopher Haster
f53a0cc961 Merge pull request #929 from littlefs-project/devel
Minor release: v2.9
2024-01-23 12:33:13 -06:00
Christopher Haster
42910bc8e5 Bumped minor version to v2.9 2024-01-19 14:37:37 -06:00
Christopher Haster
a3e1d12ce1 Merge pull request #915 from littlefs-project/well-done
Rename internal functions _raw* -> _*_
2024-01-19 13:58:29 -06:00
Christopher Haster
a70870c628 Renamed internal functions _raw* -> _*_
So instead of lfs_file_rawopencfg, it's now lfs_file_opencfg_.

The "raw" prefix is annoying, doesn't really add meaning ("internal"
would have been better), and gets in the way of finding the relevant
function implementations.

I have been using _s as suffixes for unimportant name collisions in
other codebases, and it seems to work well at reducing wasted brain
cycles naming things. Adopting it here avoids the need for "raw"
prefixes.

It's quite a bit like the use of prime symbols to resolve name
collisions in math, e.g. x' = x + 1. Which is even supported in Haskell
and is quite nice there.

And the main benefit: Now if you search for the public API name, you get
the internal function first, which is probably what you care about.

Here is the exact script:

  sed -i 's/_raw\([a-z0-9_]*\)\>/_\1_/g' $(git ls-tree -r HEAD --name-only | grep '.*\.c')
2024-01-19 13:20:56 -06:00
Christopher Haster
ceb17a0f4a Merge pull request #917 from tomscii/fix_return_value_of_lfs_rename
Fix return value of lfs_rename()
2024-01-19 13:19:21 -06:00
Christopher Haster
a8a0905777 Merge pull request #916 from littlefs-project/ci-ubuntu-latest
Change CI to just run on ubuntu-latest
2024-01-19 13:19:07 -06:00
Christopher Haster
13d78616fe Merge pull request #914 from littlefs-project/inline-max
Add inline_max, to optionally limit the size of inlined files
2024-01-19 13:18:54 -06:00
Christopher Haster
8b8fd14187 Added inline_max, to optionally limit the size of inlined files
Inlined files live in metadata and decrease storage requirements, but
may be limited to improve metadata-related performance. This is
especially important given the current plague of metadata performance.

Though decreasing inline_max may make metadata more dense and increase
block usage, so it's important to benchmark if optimizing for speed.

The underlying limits of inlined files haven't changed:
1. Inlined files need to fit in RAM, so <= cache_size
2. Inlined files need to fit in a single attr, so <= attr_max
3. Inlined files need to fit in 1/8 of a block to avoid metadata
   overflow issues, this is after limiting by metadata_max,
   so <= min(metadata_max, block_size)/8

By default, the largest possible inline_max is used. This preserves
backwards compatibility and is probably a good default for most use
cases.

This does have the awkward effect of requiring inline_max=-1 to
indicate disabled inlined files, but I don't think there's a good
way around this.
2024-01-19 13:00:27 -06:00
Christopher Haster
09972a1710 Merge pull request #913 from littlefs-project/gc-compactions
Extend lfs_fs_gc to compact metadata, compact_thresh
2024-01-19 12:51:11 -06:00
Christopher Haster
ed7bd05435 Merge pull request #912 from littlefs-project/relaxed-lookahead
Relaxed lookahead alignment, other internal block alloc readability improvements
2024-01-19 12:27:14 -06:00
Christopher Haster
b5cd957f42 Extended lfs_fs_gc to compact metadata, compact_thresh
This extends lfs_fs_gc to now handle three things:

1. Calls mkconsistent if not already consistent
2. Compacts metadata > compact_thresh
3. Populates the block allocator

Which should be all of the janitorial work that can be done without
additional on-disk data structures.

Normally, metadata compaction occurs when an mdir is full, and results in
mdirs that are at most block_size/2.

Now, if you call lfs_fs_gc, littlefs will eagerly compact any mdirs that
exceed the compact_thresh configuration option. Because the resulting
mdirs are at most block_size/2, it only makes sense for compact_thresh to
be >= block_size/2 and <= block_size.

Additionally, there are some special values:

- compact_thresh=0  => defaults to ~88% block_size, may change
- compact_thresh=-1 => disables metadata compaction during lfs_fs_gc

Note that compact_thresh only affects lfs_fs_gc. Normal compactions
still only occur when full.
2024-01-19 12:25:45 -06:00
Christopher Haster
1195d606ae Merge pull request #909 from littlefs-project/easy-util-defines
Add some easier util overrides: LFS_MALLOC/FREE/CRC
2024-01-19 12:24:16 -06:00
Christopher Haster
1711bdef76 Merge pull request #886 from BrianPugh/macro-sanity-check
Add value-range checks for user-definable macros at compile-time
2024-01-19 12:23:36 -06:00
Christopher Haster
f522ed907a Added tests over rename type errors 2024-01-17 00:10:30 -06:00
Tom Szilagyi
4f32738cd6 Fix return value of lfs_rename()
When lfs_rename() is called trying to rename (move) a file to an
existing directory, LFS_ERR_ISDIR is (correctly) returned. However, in
the opposite case, if one tries to rename (move) a directory to a path
currently occupied by a regular file, LFS_ERR_NOTDIR should be
returned (since the error is that the destination is NOT a directory),
but in reality, LFS_ERR_ISDIR is returned in this case as well.

This commit fixes the code so that in the latter case, LFS_ERR_NOTDIR
is returned.
2024-01-17 00:06:52 -06:00
Christopher Haster
6691718b18 Restricted LFS_FILE_MAX to signed 32-bits, <2^31, <=2147483647
I think realistically no one is using this. It's already only partially
supported and untested.

Worst case, if someone does depend on this we can always revert.
2024-01-16 23:40:30 -06:00
Christopher Haster
1fefcbbcba Rearranged compile-time constant checks to live near lfs_init
lfs_init handles the checks/asserts of most configuration, moving these
checks near lfs_init attempts to keep all of these checks nearby each
other.

Also updated the comments to avoid somtimes-ambiguous range notation.

And removed negative bounds checks. Negative bounds should be obviously
incorrect, and 0 is _technically_ not illegal for any define (though
admittedly unlikely to be correct).
2024-01-16 23:39:51 -06:00
Christopher Haster
60567677b9 Relaxed alignment requirements for lfs_malloc
The only reason we needed this alignment was for the lookahead buffer.

Now that the lookahead buffer is relaxed to operate on bytes, we can
relax our malloc alignment requirement all the way down to the byte
level, since we mainly use lfs_malloc to allocate byte-level buffers.

This does introduce a risk that we might need word-level mallocs in the
future. If that happens we will need to decide if changing the malloc
alignment is a breaking change, or gate alignment requirements behind
user provided defines.

Found by HiFiPhile.
2024-01-16 00:27:07 -06:00
Christopher Haster
897b571318 Changed CI to just run on ubuntu-latest
If we already have to bump this version as GitHub phases out older
Ubuntu runners (which is reasonable), I don't really see the value of
pinning a specific version. We might as well just respond to any
broken dependencies caused by GitHub's implicit updates as they
happen...

It's not like CI is truly continuous.
2023-12-21 00:33:44 -06:00
Christopher Haster
3513ff1afc Merge pull request #911 from littlefs-project/fix-release-structs
Fix struct sizes missing from generated release notes
2023-12-21 00:08:16 -06:00
Christopher Haster
8a22bd6e67 Merge pull request #910 from littlefs-project/fix-superblock-expansion-thresh
Increase threshold for superblock expansion from ~50% -> ~88% full
2023-12-21 00:07:55 -06:00
Christopher Haster
9b82db72d8 Merge pull request #898 from zchen24/patch-1
Update DESIGN.md minor typo
2023-12-21 00:06:29 -06:00
Zihan Chen
99b84ee3db Update DESIGN.md, fix minor typo 2023-12-20 23:42:26 -06:00
Christopher Haster
b1b10c0e75 Relaxed lookahead buffer alignment
This drops the lookahead buffer from operating on 32-bit words to
operating on 8-bit bytes, and removes any alignment requirement. This
may have some minor performance impact, but it is unlikely to be
significant when you consider IO overhead.

The original motivation for 32-bit alignment was an attempt at
future-proofing in case we wanted some more complex on-disk data
structure. This never happened, and even if it did, it could have been
added via additional config options.

This has been a significant pain point for users, since providing
word-aligned byte-sized buffers in C can be a bit annoying.
2023-12-20 00:39:11 -06:00
Christopher Haster
1f9c3c04b1 Reworked the block allocator so the logic is hopefully simpler
Some of this is just better documentation, some of this is reworking the
logic to be more intention driven... if that makes sense...
2023-12-20 00:24:56 -06:00
Christopher Haster
7b68441888 Renamed a number of internal block-allocator fields
- Renamed lfs.free      -> lfs.lookahead
- Renamed lfs.free.off  -> lfs.lookahead.start
- Renamed lfs.free.i    -> lfs.lookahead.next
- Renamed lfs.free.ack  -> lfs.lookahead.ckpoint
- Renamed lfs_alloc_ack -> lfs_alloc_ckpoint

These have been named a bit confusingly, and I think the new names make
their relevant purposes a bit clearer.

At the very it's clear lfs.lookahead is related to the lookahead buffer.
(and doesn't imply a closed free-bitmap).
2023-12-20 00:17:08 -06:00
Christopher Haster
e91a29d2b5 Fixed struct sizes missing from generated release notes
This script was missed during a struct -> structs naming change
2023-12-19 22:00:18 -06:00
Christopher Haster
b9b95ab4bc Increase threshold for superblock expansion from ~50% -> ~88% full
Superblock expansion is an irreversible operation. In an effort to
prevent superblock expansion from claiming valuable scratch space
(important for small, <~8 block filesystems), littlefs prevents
superblock expansion when the disk is "mostly full".

In true computer-scientist fashion, this "mostly full" threshold was
set to ~50%.

As pointed out by gbolgradov and rojer, >~50% utilization is not
uncommon, and it can lead to a situation where superblock expansion does
not occur in a relatively healthy filesystem, causing focused wear at
the root.

To remedy this, the threshold is now increased to ~88% (7/8) full.

This may change in the future and should probably be eventually user
configurable.

Found by gbolgradov and rojer
2023-12-19 16:51:17 -06:00
Christopher Haster
9a620c730c Added LFS_CRC, easier override for lfs_crc
Now you can override littlefs's CRC implementation with some simple
defines:

  -DLFS_CRC=lfs_crc

The motivation for this is the same for LFS_MALLOC/LFS_FREE. I think
these are the main "system-level" utils that users want to override.

Don't override with this something that's not CRC32! Your filesystem
will no longer be compatible with other tools! This is only intended for
provided hardware acceleration!
2023-12-19 14:12:10 -06:00
Christopher Haster
a0c6c54345 Added LFS_MALLOC/FREE, easier overrides for lfs_malloc/free
Now you can override littlefs's malloc with some simple defines:

  -DLFS_MALLOC=my_malloc
  -DLFS_FREE=my_free

This is probably what most users expected when wanting to override
malloc/free in littlefs, but it hasn't been available, since instead
littlefs provides a file-level override of builtin utils.

The thinking was that there's just too many builtins that could be
overriden, lfs_max/min/alignup/npw2/etc/etc/etc, so allowing users to
just override the util file provides the best flexibility without a ton
of ifdefs.

But it's become clear this is awkward for users that just want to
replace malloc.

Maybe the original goal was too optimistic, maybe there's a better way
to structure this file, or maybe the best API is just a bunch of ifdefs,
I have no idea! This will hopefully continue to evolve.
2023-12-19 13:57:17 -06:00
Zihan Chen
10bcff1af8 Update DESIGN.md minor typo 2023-11-26 11:10:24 -08:00
Christopher Haster
c733d9ec57 Merge pull request #884 from DvdGiessen/static-functions
lfs_fs_raw* functions should be static
2023-10-31 13:26:35 -05:00
Brian Pugh
c531a5e88f Replace erroneous LFS_FILE_MAX upper bound 4294967296 to 4294967295 2023-10-30 11:18:20 -07:00
Brian Pugh
8f9427dd53 Add value-range checks for user-definable macros 2023-10-29 13:50:38 -07:00
Christopher Haster
8f3f32d1f3 Added -Wmissing-prototypes
This warning is useful for catching the easy mistake of missing the
keyword static on functions intended to be internal-only.

Missing the static keyword risks symbol polution and misses potential
compiler optimizations.

This is an interesting warning, while useful for libraries such as
littlefs, it's perfectly valid C to not predeclare all functions, and
common in final application binaries.

Relatedly, this warning is re-disabled for the test/bench runner. There
may be a better way to organize the CFLAGS, maybe into separate
LIB/RUNNER CFLAGS, but I'll leave this to future work if our CFLAGS grow
more complicated.

This was motivated by non-static internal-only functions leaking into a
release. Found and fixed by DvdGiessen.
2023-10-24 12:04:54 -05:00
Daniël van de Giessen
92fc780f71 lfs_fs_raw* functions should be static 2023-10-23 13:35:34 +02:00
Christopher Haster
f77214d1f0 Merge pull request #877 from littlefs-project/devel
Minor release: v2.8
2023-09-22 11:52:21 -05:00
Christopher Haster
f91c5bd687 Bumped minor version to v2.8 2023-09-21 13:02:09 -05:00
Christopher Haster
0eb52a2df1 Merge pull request #875 from littlefs-project/fs-gc
Add lfs_fs_gc to enable proactive finding of free blocks
2023-09-21 13:01:19 -05:00
35 changed files with 9357 additions and 756 deletions

View File

@@ -10,7 +10,7 @@ defaults:
jobs:
post-release:
runs-on: ubuntu-22.04
runs-on: ubuntu-latest
steps:
# trigger post-release in dependency repo, this indirection allows the
# dependency repo to be updated often without affecting this repo. At

View File

@@ -11,7 +11,7 @@ defaults:
jobs:
release:
runs-on: ubuntu-22.04
runs-on: ubuntu-latest
# need to manually check for a couple things
# - tests passed?
@@ -20,7 +20,7 @@ jobs:
github.event.workflow_run.head_sha == github.sha}}
steps:
- uses: actions/checkout@v2
- uses: actions/checkout@v4
with:
ref: ${{github.event.workflow_run.head_sha}}
# need workflow access since we push branches
@@ -30,26 +30,29 @@ jobs:
fetch-depth: 0
# try to get results from tests
- uses: dawidd6/action-download-artifact@v2
- uses: actions/download-artifact@v4
continue-on-error: true
with:
workflow: ${{github.event.workflow_run.name}}
run_id: ${{github.event.workflow_run.id}}
name: sizes
github-token: ${{secrets.GITHUB_TOKEN}}
run-id: ${{github.event.workflow_run.id}}
pattern: '{sizes,sizes-*}'
merge-multiple: true
path: sizes
- uses: dawidd6/action-download-artifact@v2
- uses: actions/download-artifact@v4
continue-on-error: true
with:
workflow: ${{github.event.workflow_run.name}}
run_id: ${{github.event.workflow_run.id}}
name: cov
github-token: ${{secrets.GITHUB_TOKEN}}
run-id: ${{github.event.workflow_run.id}}
pattern: '{cov,cov-*}'
merge-multiple: true
path: cov
- uses: dawidd6/action-download-artifact@v2
- uses: actions/download-artifact@v4
continue-on-error: true
with:
workflow: ${{github.event.workflow_run.name}}
run_id: ${{github.event.workflow_run.id}}
name: bench
github-token: ${{secrets.GITHUB_TOKEN}}
run-id: ${{github.event.workflow_run.id}}
pattern: '{bench,bench-*}'
merge-multiple: true
path: bench
- name: find-version
@@ -112,7 +115,7 @@ jobs:
table[$i,$j]=$c_camel
((j+=1))
for s in code stack struct
for s in code stack structs
do
f=sizes/thumb${c:+-$c}.$s.csv
[ -e $f ] && table[$i,$j]=$( \

View File

@@ -11,14 +11,15 @@ defaults:
jobs:
# forward custom statuses
status:
runs-on: ubuntu-22.04
runs-on: ubuntu-latest
steps:
- uses: dawidd6/action-download-artifact@v2
- uses: actions/download-artifact@v4
continue-on-error: true
with:
workflow: ${{github.event.workflow_run.name}}
run_id: ${{github.event.workflow_run.id}}
name: status
github-token: ${{secrets.GITHUB_TOKEN}}
run-id: ${{github.event.workflow_run.id}}
pattern: '{status,status-*}'
merge-multiple: true
path: status
- name: update-status
continue-on-error: true
@@ -60,19 +61,20 @@ jobs:
# forward custom pr-comments
comment:
runs-on: ubuntu-22.04
runs-on: ubuntu-latest
# only run on success (we don't want garbage comments!)
if: ${{github.event.workflow_run.conclusion == 'success'}}
steps:
# generated comment?
- uses: dawidd6/action-download-artifact@v2
- uses: actions/download-artifact@v4
continue-on-error: true
with:
workflow: ${{github.event.workflow_run.name}}
run_id: ${{github.event.workflow_run.id}}
name: comment
github-token: ${{secrets.GITHUB_TOKEN}}
run-id: ${{github.event.workflow_run.id}}
pattern: '{comment,comment-*}'
merge-multiple: true
path: comment
- name: update-comment
continue-on-error: true

View File

@@ -14,14 +14,14 @@ env:
jobs:
# run tests
test:
runs-on: ubuntu-22.04
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
arch: [x86_64, thumb, mips, powerpc]
steps:
- uses: actions/checkout@v2
- uses: actions/checkout@v4
- name: install
run: |
# need a few things
@@ -235,9 +235,9 @@ jobs:
# create size statuses
- name: upload-sizes
uses: actions/upload-artifact@v2
uses: actions/upload-artifact@v4
with:
name: sizes
name: sizes-${{matrix.arch}}
path: sizes
- name: status-sizes
run: |
@@ -273,16 +273,16 @@ jobs:
}' | tee status/$(basename $f .csv).json
done
- name: upload-status-sizes
uses: actions/upload-artifact@v2
uses: actions/upload-artifact@v4
with:
name: status
name: status-sizes-${{matrix.arch}}
path: status
retention-days: 1
# create cov statuses
- name: upload-cov
if: ${{matrix.arch == 'x86_64'}}
uses: actions/upload-artifact@v2
uses: actions/upload-artifact@v4
with:
name: cov
path: cov
@@ -317,11 +317,11 @@ jobs:
target_step: env.STEP,
}' | tee status/$(basename $f .csv)-$s.json
done
- name: upload-status-sizes
- name: upload-status-cov
if: ${{matrix.arch == 'x86_64'}}
uses: actions/upload-artifact@v2
uses: actions/upload-artifact@v4
with:
name: status
name: status-cov
path: status
retention-days: 1
@@ -329,14 +329,14 @@ jobs:
#
# this grows exponentially, so it doesn't turn out to be that many
test-pls:
runs-on: ubuntu-22.04
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
pls: [1, 2]
steps:
- uses: actions/checkout@v2
- uses: actions/checkout@v4
- name: install
run: |
# need a few things
@@ -359,9 +359,9 @@ jobs:
# run with LFS_NO_INTRINSICS to make sure that works
test-no-intrinsics:
runs-on: ubuntu-22.04
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- uses: actions/checkout@v4
- name: install
run: |
# need a few things
@@ -374,11 +374,50 @@ jobs:
run: |
CFLAGS="$CFLAGS -DLFS_NO_INTRINSICS" make test
test-shrink:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: install
run: |
# need a few things
sudo apt-get update -qq
sudo apt-get install -qq gcc python3 python3-pip
pip3 install toml
gcc --version
python3 --version
- name: test-no-intrinsics
run: |
CFLAGS="$CFLAGS -DLFS_SHRINKNONRELOCATING" make test
# run with all trace options enabled to at least make sure these
# all compile
test-yes-trace:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: install
run: |
# need a few things
sudo apt-get update -qq
sudo apt-get install -qq gcc python3 python3-pip
pip3 install toml
gcc --version
python3 --version
- name: test-yes-trace
run: |
CFLAGS="$CFLAGS \
-DLFS_YES_TRACE \
-DLFS_RAMBD_YES_TRACE \
-DLFS_FILEBD_YES_TRACE \
-DLFS_RAMBD_YES_TRACE" \
make test
# run LFS_MULTIVERSION tests
test-multiversion:
runs-on: ubuntu-22.04
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- uses: actions/checkout@v4
- name: install
run: |
# need a few things
@@ -393,9 +432,9 @@ jobs:
# run tests on the older version lfs2.0
test-lfs2_0:
runs-on: ubuntu-22.04
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- uses: actions/checkout@v4
- name: install
run: |
# need a few things
@@ -412,9 +451,9 @@ jobs:
# run under Valgrind to check for memory errors
test-valgrind:
runs-on: ubuntu-22.04
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- uses: actions/checkout@v4
- name: install
run: |
# need a few things
@@ -431,12 +470,11 @@ jobs:
TESTFLAGS="$TESTFLAGS --valgrind --context=1024 -Gdefault -Pnone" \
make test
# test that compilation is warning free under clang
# run with Clang, mostly to check for Clang-specific warnings
# compile/run with Clang, mostly to check for Clang-specific warnings
test-clang:
runs-on: ubuntu-22.04
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- uses: actions/checkout@v4
- name: install
run: |
# need a few things
@@ -446,20 +484,16 @@ jobs:
python3 --version
- name: test-clang
run: |
# override CFLAGS since Clang does not support -fcallgraph-info
# and -ftrack-macro-expansions
make \
CC=clang \
CFLAGS="$CFLAGS -MMD -g3 -I. -std=c99 -Wall -Wextra -pedantic" \
test
CC=clang \
make test
# run benchmarks
#
# note there's no real benefit to running these on multiple archs
bench:
runs-on: ubuntu-22.04
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- uses: actions/checkout@v4
- name: install
run: |
# need a few things
@@ -491,7 +525,7 @@ jobs:
# create bench statuses
- name: upload-bench
uses: actions/upload-artifact@v2
uses: actions/upload-artifact@v4
with:
name: bench
path: bench
@@ -525,20 +559,20 @@ jobs:
}' | tee status/$(basename $f .csv)-$s.json
done
- name: upload-status-bench
uses: actions/upload-artifact@v2
uses: actions/upload-artifact@v4
with:
name: status
name: status-bench
path: status
retention-days: 1
# run compatibility tests using the current master as the previous version
test-compat:
runs-on: ubuntu-22.04
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- uses: actions/checkout@v4
if: ${{github.event_name == 'pull_request'}}
# checkout the current pr target into lfsp
- uses: actions/checkout@v2
- uses: actions/checkout@v4
if: ${{github.event_name == 'pull_request'}}
with:
ref: ${{github.event.pull_request.base.ref}}
@@ -569,10 +603,10 @@ jobs:
# self-host with littlefs-fuse for a fuzz-like test
fuse:
runs-on: ubuntu-22.04
runs-on: ubuntu-latest
if: ${{!endsWith(github.ref, '-prefix')}}
steps:
- uses: actions/checkout@v2
- uses: actions/checkout@v4
- name: install
run: |
# need a few things
@@ -582,7 +616,7 @@ jobs:
gcc --version
python3 --version
fusermount -V
- uses: actions/checkout@v2
- uses: actions/checkout@v4
with:
repository: littlefs-project/littlefs-fuse
ref: v2
@@ -619,10 +653,10 @@ jobs:
# test migration using littlefs-fuse
migrate:
runs-on: ubuntu-22.04
runs-on: ubuntu-latest
if: ${{!endsWith(github.ref, '-prefix')}}
steps:
- uses: actions/checkout@v2
- uses: actions/checkout@v4
- name: install
run: |
# need a few things
@@ -632,12 +666,12 @@ jobs:
gcc --version
python3 --version
fusermount -V
- uses: actions/checkout@v2
- uses: actions/checkout@v4
with:
repository: littlefs-project/littlefs-fuse
ref: v2
path: v2
- uses: actions/checkout@v2
- uses: actions/checkout@v4
with:
repository: littlefs-project/littlefs-fuse
ref: v1
@@ -691,10 +725,10 @@ jobs:
# status related tasks that run after tests
status:
runs-on: ubuntu-22.04
runs-on: ubuntu-latest
needs: [test, bench]
steps:
- uses: actions/checkout@v2
- uses: actions/checkout@v4
if: ${{github.event_name == 'pull_request'}}
- name: install
if: ${{github.event_name == 'pull_request'}}
@@ -704,23 +738,26 @@ jobs:
pip3 install toml
gcc --version
python3 --version
- uses: actions/download-artifact@v2
- uses: actions/download-artifact@v4
if: ${{github.event_name == 'pull_request'}}
continue-on-error: true
with:
name: sizes
pattern: '{sizes,sizes-*}'
merge-multiple: true
path: sizes
- uses: actions/download-artifact@v2
- uses: actions/download-artifact@v4
if: ${{github.event_name == 'pull_request'}}
continue-on-error: true
with:
name: cov
pattern: '{cov,cov-*}'
merge-multiple: true
path: cov
- uses: actions/download-artifact@v2
- uses: actions/download-artifact@v4
if: ${{github.event_name == 'pull_request'}}
continue-on-error: true
with:
name: bench
pattern: '{bench,bench-*}'
merge-multiple: true
path: bench
# try to find results from tests
@@ -862,7 +899,7 @@ jobs:
body: $comment,
}' | tee comment/comment.json
- name: upload-comment
uses: actions/upload-artifact@v2
uses: actions/upload-artifact@v4
with:
name: comment
path: comment

View File

@@ -59,7 +59,7 @@ This leaves us with three major requirements for an embedded filesystem.
RAM to temporarily store filesystem metadata.
For ROM, this means we need to keep our design simple and reuse code paths
were possible. For RAM we have a stronger requirement, all RAM usage is
where possible. For RAM we have a stronger requirement, all RAM usage is
bounded. This means RAM usage does not grow as the filesystem changes in
size or number of files. This creates a unique challenge as even presumably
simple operations, such as traversing the filesystem, become surprisingly
@@ -626,7 +626,7 @@ log&#8322;_n_ pointers that skip to different preceding elements of the
skip-list.
The name comes from heavy use of the [CTZ instruction][wikipedia-ctz], which
lets us calculate the power-of-two factors efficiently. For a give block _n_,
lets us calculate the power-of-two factors efficiently. For a given block _n_,
that block contains ctz(_n_)+1 pointers.
```

View File

@@ -18,6 +18,12 @@ VALGRIND ?= valgrind
GDB ?= gdb
PERF ?= perf
# guess clang or gcc (clang sometimes masquerades as gcc because of
# course it does)
ifneq ($(shell $(CC) --version | grep clang),)
NO_GCC = 1
endif
SRC ?= $(filter-out $(wildcard *.t.* *.b.*),$(wildcard *.c))
OBJ := $(SRC:%.c=$(BUILDDIR)/%.o)
DEP := $(SRC:%.c=$(BUILDDIR)/%.d)
@@ -59,11 +65,15 @@ BENCH_PERF := $(BENCH_RUNNER:%=%.perf)
BENCH_TRACE := $(BENCH_RUNNER:%=%.trace)
BENCH_CSV := $(BENCH_RUNNER:%=%.csv)
CFLAGS += -fcallgraph-info=su
CFLAGS += -g3
CFLAGS += -I.
CFLAGS += -std=c99 -Wall -Wextra -pedantic
CFLAGS += -Wmissing-prototypes
ifndef NO_GCC
CFLAGS += -fcallgraph-info=su
CFLAGS += -ftrack-macro-expansion=0
endif
ifdef DEBUG
CFLAGS += -O0
else
@@ -354,6 +364,7 @@ summary-diff sizes-diff: $(OBJ) $(CI)
## Build the test-runner
.PHONY: test-runner build-test
test-runner build-test: CFLAGS+=-Wno-missing-prototypes
ifndef NO_COV
test-runner build-test: CFLAGS+=--coverage
endif
@@ -405,6 +416,7 @@ testmarks-diff: $(TEST_CSV)
## Build the bench-runner
.PHONY: bench-runner build-bench
bench-runner build-bench: CFLAGS+=-Wno-missing-prototypes
ifdef YES_COV
bench-runner build-bench: CFLAGS+=--coverage
endif
@@ -463,6 +475,7 @@ benchmarks-diff: $(BENCH_CSV)
# rules
-include $(DEP)
-include $(TEST_DEP)
-include $(BENCH_DEP)
.SUFFIXES:
.SECONDARY:

View File

@@ -199,6 +199,47 @@ The tests assume a Linux environment and can be started with make:
make test
```
Tests are implemented in C in the .toml files found in the `tests` directory.
When developing a feature or fixing a bug, it is frequently useful to run a
single test case or suite of tests:
``` bash
./scripts/test.py -l runners/test_runner # list available test suites
./scripts/test.py -L runners/test_runner test_dirs # list available test cases
./scripts/test.py runners/test_runner test_dirs # run a specific test suite
```
If an assert fails in a test, test.py will try to print information about the
failure:
``` bash
tests/test_dirs.toml:1:failure: test_dirs_root:1g12gg2 (PROG_SIZE=16, ERASE_SIZE=512) failed
tests/test_dirs.toml:5:assert: assert failed with 0, expected eq 42
lfs_mount(&lfs, cfg) => 42;
```
This includes the test id, which can be passed to test.py to run only that
specific test permutation:
``` bash
./scripts/test.py runners/test_runner test_dirs_root:1g12gg2 # run a specific test permutation
./scripts/test.py runners/test_runner test_dirs_root:1g12gg2 --gdb # drop into gdb on failure
```
Some other flags that may be useful:
```bash
./scripts/test.py runners/test_runner -b -j # run tests in parallel
./scripts/test.py runners/test_runner -v -O- # redirect stdout to stdout
./scripts/test.py runners/test_runner -ddisk # capture resulting disk image
```
See `-h/--help` for a full list of available flags:
``` bash
./scripts/test.py --help
```
## License
The littlefs is provided under the [BSD-3-Clause] license. See
@@ -231,11 +272,31 @@ License Identifiers that are here available: http://spdx.org/licenses/
to use littlefs in a Rust-friendly API, reaping the benefits of Rust's memory
safety and other guarantees.
- [nim-littlefs] - A Nim wrapper and API for littlefs. Includes a fuse
implementation based on [littlefs-fuse]
- [chamelon] - A pure-OCaml implementation of (most of) littlefs, designed for
use with the MirageOS library operating system project. It is interoperable
with the reference implementation, with some caveats.
- [littlefs-disk-img-viewer] - A memory-efficient web application for viewing
littlefs disk images in your web browser.
- [mklfs] - A command line tool built by the [Lua RTOS] guys for making
littlefs images from a host PC. Supports Windows, Mac OS, and Linux.
- [mklfs] - A command line tool for creating littlefs images. Used in the Lua
RTOS ecosystem.
- [mklittlefs] - A command line tool for creating littlefs images. Used in the
ESP8266 and RP2040 ecosystem.
- [pico-littlefs-usb] - An interface for littlefs that emulates a FAT12
filesystem over USB. Allows mounting littlefs on a host PC without additional
drivers.
- [ramcrc32bd] - An example block device using littlefs's 32-bit CRC for
error-correction.
- [ramrsbd] - An example block device using Reed-Solomon codes for
error-correction.
- [Mbed OS] - The easiest way to get started with littlefs is to jump into Mbed
which already has block device drivers for most forms of embedded storage.
@@ -254,23 +315,23 @@ License Identifiers that are here available: http://spdx.org/licenses/
for microcontroller-scale devices. Due to limitations of FAT it can't provide
power-loss resilience, but it does allow easy interop with PCs.
- [chamelon] - A pure-OCaml implementation of (most of) littlefs, designed for
use with the MirageOS library operating system project. It is interoperable
with the reference implementation, with some caveats.
[BSD-3-Clause]: https://spdx.org/licenses/BSD-3-Clause.html
[littlefs-disk-img-viewer]: https://github.com/tniessen/littlefs-disk-img-viewer
[littlefs-fuse]: https://github.com/geky/littlefs-fuse
[FUSE]: https://github.com/libfuse/libfuse
[littlefs-js]: https://github.com/geky/littlefs-js
[littlefs-js-demo]:http://littlefs.geky.net/demo.html
[littlefs-python]: https://pypi.org/project/littlefs-python/
[littlefs2-rust]: https://crates.io/crates/littlefs2
[nim-littlefs]: https://github.com/Graveflo/nim-littlefs
[chamelon]: https://github.com/yomimono/chamelon
[littlefs-disk-img-viewer]: https://github.com/tniessen/littlefs-disk-img-viewer
[mklfs]: https://github.com/whitecatboard/Lua-RTOS-ESP32/tree/master/components/mklfs/src
[Lua RTOS]: https://github.com/whitecatboard/Lua-RTOS-ESP32
[mklittlefs]: https://github.com/earlephilhower/mklittlefs
[pico-littlefs-usb]: https://github.com/oyama/pico-littlefs-usb
[ramcrc32bd]: https://github.com/geky/ramcrc32bd
[ramrsbd]: https://github.com/geky/ramrsbd
[Mbed OS]: https://github.com/armmbed/mbed-os
[LittleFileSystem]: https://os.mbed.com/docs/mbed-os/latest/apis/littlefilesystem.html
[SPIFFS]: https://github.com/pellepl/spiffs
[Dhara]: https://github.com/dlbeer/dhara
[ChaN's FatFs]: http://elm-chan.org/fsw/ff/00index_e.html
[littlefs-python]: https://pypi.org/project/littlefs-python/
[littlefs2-rust]: https://crates.io/crates/littlefs2
[chamelon]: https://github.com/yomimono/chamelon

View File

@@ -441,9 +441,10 @@ Superblock fields:
7. **Attr max (32-bits)** - Maximum size of file attributes in bytes.
The superblock must always be the first entry (id 0) in a metadata pair as well
as be the first entry written to the block. This means that the superblock
entry can be read from a device using offsets alone.
The superblock must always be the first entry (id 0) in the metadata pair, and
the name tag must always be the first tag in the metadata pair. This makes it
so that the magic string "littlefs" will always reside at offset=8 in a valid
littlefs superblock.
---
#### `0x2xx` LFS_TYPE_STRUCT

View File

@@ -129,6 +129,8 @@ int lfs_emubd_create(const struct lfs_config *cfg,
bd->proged = 0;
bd->erased = 0;
bd->power_cycles = bd->cfg->power_cycles;
bd->ooo_block = -1;
bd->ooo_data = NULL;
bd->disk = NULL;
if (bd->cfg->disk_path) {
@@ -195,6 +197,7 @@ int lfs_emubd_destroy(const struct lfs_config *cfg) {
free(bd->blocks);
// clean up other resources
lfs_emubd_decblock(bd->ooo_data);
if (bd->disk) {
bd->disk->rc -= 1;
if (bd->disk->rc == 0) {
@@ -209,6 +212,75 @@ int lfs_emubd_destroy(const struct lfs_config *cfg) {
}
// powerloss hook
static int lfs_emubd_powerloss(const struct lfs_config *cfg) {
lfs_emubd_t *bd = cfg->context;
// emulate out-of-order writes?
lfs_emubd_block_t *ooo_data = NULL;
if (bd->cfg->powerloss_behavior == LFS_EMUBD_POWERLOSS_OOO
&& bd->ooo_block != -1) {
// since writes between syncs are allowed to be out-of-order, it
// shouldn't hurt to restore the first write on powerloss, right?
ooo_data = bd->blocks[bd->ooo_block];
bd->blocks[bd->ooo_block] = lfs_emubd_incblock(bd->ooo_data);
// mirror to disk file?
if (bd->disk
&& (bd->blocks[bd->ooo_block]
|| bd->cfg->erase_value != -1)) {
off_t res1 = lseek(bd->disk->fd,
(off_t)bd->ooo_block*bd->cfg->erase_size,
SEEK_SET);
if (res1 < 0) {
return -errno;
}
ssize_t res2 = write(bd->disk->fd,
(bd->blocks[bd->ooo_block])
? bd->blocks[bd->ooo_block]->data
: bd->disk->scratch,
bd->cfg->erase_size);
if (res2 < 0) {
return -errno;
}
}
}
// simulate power loss
bd->cfg->powerloss_cb(bd->cfg->powerloss_data);
// if we continue, undo out-of-order write emulation
if (bd->cfg->powerloss_behavior == LFS_EMUBD_POWERLOSS_OOO
&& bd->ooo_block != -1) {
lfs_emubd_decblock(bd->blocks[bd->ooo_block]);
bd->blocks[bd->ooo_block] = ooo_data;
// mirror to disk file?
if (bd->disk
&& (bd->blocks[bd->ooo_block]
|| bd->cfg->erase_value != -1)) {
off_t res1 = lseek(bd->disk->fd,
(off_t)bd->ooo_block*bd->cfg->erase_size,
SEEK_SET);
if (res1 < 0) {
return -errno;
}
ssize_t res2 = write(bd->disk->fd,
(bd->blocks[bd->ooo_block])
? bd->blocks[bd->ooo_block]->data
: bd->disk->scratch,
bd->cfg->erase_size);
if (res2 < 0) {
return -errno;
}
}
}
return 0;
}
// block device API
@@ -344,8 +416,11 @@ int lfs_emubd_prog(const struct lfs_config *cfg, lfs_block_t block,
if (bd->power_cycles > 0) {
bd->power_cycles -= 1;
if (bd->power_cycles == 0) {
// simulate power loss
bd->cfg->powerloss_cb(bd->cfg->powerloss_data);
int err = lfs_emubd_powerloss(cfg);
if (err) {
LFS_EMUBD_TRACE("lfs_emubd_prog -> %d", err);
return err;
}
}
}
@@ -361,10 +436,17 @@ int lfs_emubd_erase(const struct lfs_config *cfg, lfs_block_t block) {
// check if erase is valid
LFS_ASSERT(block < bd->cfg->erase_count);
// emulate out-of-order writes? save first write
if (bd->cfg->powerloss_behavior == LFS_EMUBD_POWERLOSS_OOO
&& bd->ooo_block == -1) {
bd->ooo_block = block;
bd->ooo_data = lfs_emubd_incblock(bd->blocks[block]);
}
// get the block
lfs_emubd_block_t *b = lfs_emubd_mutblock(cfg, &bd->blocks[block]);
if (!b) {
LFS_EMUBD_TRACE("lfs_emubd_prog -> %d", LFS_ERR_NOMEM);
LFS_EMUBD_TRACE("lfs_emubd_erase -> %d", LFS_ERR_NOMEM);
return LFS_ERR_NOMEM;
}
@@ -430,8 +512,11 @@ int lfs_emubd_erase(const struct lfs_config *cfg, lfs_block_t block) {
if (bd->power_cycles > 0) {
bd->power_cycles -= 1;
if (bd->power_cycles == 0) {
// simulate power loss
bd->cfg->powerloss_cb(bd->cfg->powerloss_data);
int err = lfs_emubd_powerloss(cfg);
if (err) {
LFS_EMUBD_TRACE("lfs_emubd_erase -> %d", err);
return err;
}
}
}
@@ -441,17 +526,24 @@ int lfs_emubd_erase(const struct lfs_config *cfg, lfs_block_t block) {
int lfs_emubd_sync(const struct lfs_config *cfg) {
LFS_EMUBD_TRACE("lfs_emubd_sync(%p)", (void*)cfg);
lfs_emubd_t *bd = cfg->context;
// do nothing
(void)cfg;
// emulate out-of-order writes? reset first write, writes
// cannot be out-of-order across sync
if (bd->cfg->powerloss_behavior == LFS_EMUBD_POWERLOSS_OOO) {
lfs_emubd_decblock(bd->ooo_data);
bd->ooo_block = -1;
bd->ooo_data = NULL;
}
LFS_EMUBD_TRACE("lfs_emubd_sync -> %d", 0);
return 0;
}
/// Additional extended API for driving test features ///
static int lfs_emubd_rawcrc(const struct lfs_config *cfg,
static int lfs_emubd_crc_(const struct lfs_config *cfg,
lfs_block_t block, uint32_t *crc) {
lfs_emubd_t *bd = cfg->context;
@@ -480,7 +572,7 @@ int lfs_emubd_crc(const struct lfs_config *cfg,
lfs_block_t block, uint32_t *crc) {
LFS_EMUBD_TRACE("lfs_emubd_crc(%p, %"PRIu32", %p)",
(void*)cfg, block, crc);
int err = lfs_emubd_rawcrc(cfg, block, crc);
int err = lfs_emubd_crc_(cfg, block, crc);
LFS_EMUBD_TRACE("lfs_emubd_crc -> %d", err);
return err;
}
@@ -491,7 +583,7 @@ int lfs_emubd_bdcrc(const struct lfs_config *cfg, uint32_t *crc) {
uint32_t crc_ = 0xffffffff;
for (lfs_block_t i = 0; i < cfg->block_count; i++) {
uint32_t i_crc;
int err = lfs_emubd_rawcrc(cfg, i, &i_crc);
int err = lfs_emubd_crc_(cfg, i, &i_crc);
if (err) {
LFS_EMUBD_TRACE("lfs_emubd_bdcrc -> %d", err);
return err;
@@ -633,6 +725,8 @@ int lfs_emubd_copy(const struct lfs_config *cfg, lfs_emubd_t *copy) {
copy->proged = bd->proged;
copy->erased = bd->erased;
copy->power_cycles = bd->power_cycles;
copy->ooo_block = bd->ooo_block;
copy->ooo_data = lfs_emubd_incblock(bd->ooo_data);
copy->disk = bd->disk;
if (copy->disk) {
copy->disk->rc += 1;

View File

@@ -36,17 +36,18 @@ extern "C"
// Not that read-noop is not allowed. Read _must_ return a consistent (but
// may be arbitrary) value on every read.
typedef enum lfs_emubd_badblock_behavior {
LFS_EMUBD_BADBLOCK_PROGERROR,
LFS_EMUBD_BADBLOCK_ERASEERROR,
LFS_EMUBD_BADBLOCK_READERROR,
LFS_EMUBD_BADBLOCK_PROGNOOP,
LFS_EMUBD_BADBLOCK_ERASENOOP,
LFS_EMUBD_BADBLOCK_PROGERROR = 0, // Error on prog
LFS_EMUBD_BADBLOCK_ERASEERROR = 1, // Error on erase
LFS_EMUBD_BADBLOCK_READERROR = 2, // Error on read
LFS_EMUBD_BADBLOCK_PROGNOOP = 3, // Prog does nothing silently
LFS_EMUBD_BADBLOCK_ERASENOOP = 4, // Erase does nothing silently
} lfs_emubd_badblock_behavior_t;
// Mode determining how power-loss behaves during testing. For now this
// only supports a noop behavior, leaving the data on-disk untouched.
typedef enum lfs_emubd_powerloss_behavior {
LFS_EMUBD_POWERLOSS_NOOP,
LFS_EMUBD_POWERLOSS_NOOP = 0, // Progs are atomic
LFS_EMUBD_POWERLOSS_OOO = 1, // Blocks are written out-of-order
} lfs_emubd_powerloss_behavior_t;
// Type for measuring read/program/erase operations
@@ -152,6 +153,8 @@ typedef struct lfs_emubd {
lfs_emubd_io_t proged;
lfs_emubd_io_t erased;
lfs_emubd_powercycles_t power_cycles;
lfs_ssize_t ooo_block;
lfs_emubd_block_t *ooo_data;
lfs_emubd_disk_t *disk;
const struct lfs_emubd_config *cfg;

View File

@@ -133,7 +133,7 @@ int lfs_filebd_prog(const struct lfs_config *cfg, lfs_block_t block,
int lfs_filebd_erase(const struct lfs_config *cfg, lfs_block_t block) {
LFS_FILEBD_TRACE("lfs_filebd_erase(%p, 0x%"PRIx32" (%"PRIu32"))",
(void*)cfg, block, ((lfs_file_t*)cfg->context)->cfg->erase_size);
(void*)cfg, block, ((lfs_filebd_t*)cfg->context)->cfg->erase_size);
lfs_filebd_t *bd = cfg->context;
// check if erase is valid

888
lfs.c

File diff suppressed because it is too large Load Diff

100
lfs.h
View File

@@ -21,7 +21,7 @@ extern "C"
// Software library version
// Major (top-nibble), incremented on backwards incompatible changes
// Minor (bottom-nibble), incremented on feature additions
#define LFS_VERSION 0x00020007
#define LFS_VERSION 0x0002000b
#define LFS_VERSION_MAJOR (0xffff & (LFS_VERSION >> 16))
#define LFS_VERSION_MINOR (0xffff & (LFS_VERSION >> 0))
@@ -52,16 +52,15 @@ typedef uint32_t lfs_block_t;
#endif
// Maximum size of a file in bytes, may be redefined to limit to support other
// drivers. Limited on disk to <= 4294967296. However, above 2147483647 the
// functions lfs_file_seek, lfs_file_size, and lfs_file_tell will return
// incorrect values due to using signed integers. Stored in superblock and
// must be respected by other littlefs drivers.
// drivers. Limited on disk to <= 2147483647. Stored in superblock and must be
// respected by other littlefs drivers.
#ifndef LFS_FILE_MAX
#define LFS_FILE_MAX 2147483647
#endif
// Maximum size of custom attributes in bytes, may be redefined, but there is
// no real benefit to using a smaller LFS_ATTR_MAX. Limited to <= 1022.
// no real benefit to using a smaller LFS_ATTR_MAX. Limited to <= 1022. Stored
// in superblock and must be respected by other littlefs drivers.
#ifndef LFS_ATTR_MAX
#define LFS_ATTR_MAX 1022
#endif
@@ -205,7 +204,8 @@ struct lfs_config {
// program sizes.
lfs_size_t block_size;
// Number of erasable blocks on the device.
// Number of erasable blocks on the device. Defaults to block_count stored
// on disk when zero.
lfs_size_t block_count;
// Number of erase cycles before littlefs evicts metadata logs and moves
@@ -226,9 +226,20 @@ struct lfs_config {
// Size of the lookahead buffer in bytes. A larger lookahead buffer
// increases the number of blocks found during an allocation pass. The
// lookahead buffer is stored as a compact bitmap, so each byte of RAM
// can track 8 blocks. Must be a multiple of 8.
// can track 8 blocks.
lfs_size_t lookahead_size;
// Threshold for metadata compaction during lfs_fs_gc in bytes. Metadata
// pairs that exceed this threshold will be compacted during lfs_fs_gc.
// Defaults to ~88% block_size when zero, though the default may change
// in the future.
//
// Note this only affects lfs_fs_gc. Normal compactions still only occur
// when full.
//
// Set to -1 to disable metadata compaction during lfs_fs_gc.
lfs_size_t compact_thresh;
// Optional statically allocated read buffer. Must be cache_size.
// By default lfs_malloc is used to allocate this buffer.
void *read_buffer;
@@ -237,25 +248,24 @@ struct lfs_config {
// By default lfs_malloc is used to allocate this buffer.
void *prog_buffer;
// Optional statically allocated lookahead buffer. Must be lookahead_size
// and aligned to a 32-bit boundary. By default lfs_malloc is used to
// allocate this buffer.
// Optional statically allocated lookahead buffer. Must be lookahead_size.
// By default lfs_malloc is used to allocate this buffer.
void *lookahead_buffer;
// Optional upper limit on length of file names in bytes. No downside for
// larger names except the size of the info struct which is controlled by
// the LFS_NAME_MAX define. Defaults to LFS_NAME_MAX when zero. Stored in
// superblock and must be respected by other littlefs drivers.
// the LFS_NAME_MAX define. Defaults to LFS_NAME_MAX or name_max stored on
// disk when zero.
lfs_size_t name_max;
// Optional upper limit on files in bytes. No downside for larger files
// but must be <= LFS_FILE_MAX. Defaults to LFS_FILE_MAX when zero. Stored
// in superblock and must be respected by other littlefs drivers.
// but must be <= LFS_FILE_MAX. Defaults to LFS_FILE_MAX or file_max stored
// on disk when zero.
lfs_size_t file_max;
// Optional upper limit on custom attributes in bytes. No downside for
// larger attributes size but must be <= LFS_ATTR_MAX. Defaults to
// LFS_ATTR_MAX when zero.
// LFS_ATTR_MAX or attr_max stored on disk when zero.
lfs_size_t attr_max;
// Optional upper limit on total space given to metadata pairs in bytes. On
@@ -264,6 +274,15 @@ struct lfs_config {
// Defaults to block_size when zero.
lfs_size_t metadata_max;
// Optional upper limit on inlined files in bytes. Inlined files live in
// metadata and decrease storage requirements, but may be limited to
// improve metadata-related performance. Must be <= cache_size, <=
// attr_max, and <= block_size/4. Defaults to the largest possible
// inline_max when zero.
//
// Set to -1 to disable inlined files.
lfs_size_t inline_max;
#ifdef LFS_MULTIVERSION
// On-disk version to use when writing in the form of 16-bit major version
// + 16-bit minor version. This limiting metadata to what is supported by
@@ -430,19 +449,20 @@ typedef struct lfs {
lfs_gstate_t gdisk;
lfs_gstate_t gdelta;
struct lfs_free {
lfs_block_t off;
struct lfs_lookahead {
lfs_block_t start;
lfs_block_t size;
lfs_block_t i;
lfs_block_t ack;
uint32_t *buffer;
} free;
lfs_block_t next;
lfs_block_t ckpoint;
uint8_t *buffer;
} lookahead;
const struct lfs_config *cfg;
lfs_size_t block_count;
lfs_size_t name_max;
lfs_size_t file_max;
lfs_size_t attr_max;
lfs_size_t inline_max;
#ifdef LFS_MIGRATE
struct lfs1 *lfs1;
@@ -712,18 +732,6 @@ lfs_ssize_t lfs_fs_size(lfs_t *lfs);
// Returns a negative error code on failure.
int lfs_fs_traverse(lfs_t *lfs, int (*cb)(void*, lfs_block_t), void *data);
// Attempt to proactively find free blocks
//
// Calling this function is not required, but may allowing the offloading of
// the expensive block allocation scan to a less time-critical code path.
//
// Note: littlefs currently does not persist any found free blocks to disk.
// This may change in the future.
//
// Returns a negative error code on failure. Finding no free blocks is
// not an error.
int lfs_fs_gc(lfs_t *lfs);
#ifndef LFS_READONLY
// Attempt to make the filesystem consistent and ready for writing
//
@@ -736,11 +744,33 @@ int lfs_fs_gc(lfs_t *lfs);
int lfs_fs_mkconsistent(lfs_t *lfs);
#endif
#ifndef LFS_READONLY
// Attempt any janitorial work
//
// This currently:
// 1. Calls mkconsistent if not already consistent
// 2. Compacts metadata > compact_thresh
// 3. Populates the block allocator
//
// Though additional janitorial work may be added in the future.
//
// Calling this function is not required, but may allow the offloading of
// expensive janitorial work to a less time-critical code path.
//
// Returns a negative error code on failure. Accomplishing nothing is not
// an error.
int lfs_fs_gc(lfs_t *lfs);
#endif
#ifndef LFS_READONLY
// Grows the filesystem to a new size, updating the superblock with the new
// block count.
//
// Note: This is irreversible.
// If LFS_SHRINKNONRELOCATING is defined, this function will also accept
// block_counts smaller than the current configuration, after checking
// that none of the blocks that are being removed are in use.
// Note that littlefs's pseudorandom block allocation means that
// this is very unlikely to work in the general case.
//
// Returns a negative error code on failure.
int lfs_fs_grow(lfs_t *lfs, lfs_size_t block_count);

View File

@@ -11,6 +11,8 @@
#ifndef LFS_CONFIG
// If user provides their own CRC impl we don't need this
#ifndef LFS_CRC
// Software CRC implementation with small lookup table
uint32_t lfs_crc(uint32_t crc, const void *buffer, size_t size) {
static const uint32_t rtable[16] = {
@@ -29,6 +31,7 @@ uint32_t lfs_crc(uint32_t crc, const void *buffer, size_t size) {
return crc;
}
#endif
#endif

View File

@@ -8,6 +8,9 @@
#ifndef LFS_UTIL_H
#define LFS_UTIL_H
#define LFS_STRINGIZE(x) LFS_STRINGIZE2(x)
#define LFS_STRINGIZE2(x) #x
// Users can override lfs_util.h with their own configuration by defining
// LFS_CONFIG as a header file to include (-DLFS_CONFIG=lfs_config.h).
//
@@ -15,11 +18,26 @@
// provided by the config file. To start, I would suggest copying lfs_util.h
// and modifying as needed.
#ifdef LFS_CONFIG
#define LFS_STRINGIZE(x) LFS_STRINGIZE2(x)
#define LFS_STRINGIZE2(x) #x
#include LFS_STRINGIZE(LFS_CONFIG)
#else
// Alternatively, users can provide a header file which defines
// macros and other things consumed by littlefs.
//
// For example, provide my_defines.h, which contains
// something like:
//
// #include <stddef.h>
// extern void *my_malloc(size_t sz);
// #define LFS_MALLOC(sz) my_malloc(sz)
//
// And build littlefs with the header by defining LFS_DEFINES.
// (-DLFS_DEFINES=my_defines.h)
#ifdef LFS_DEFINES
#include LFS_STRINGIZE(LFS_DEFINES)
#endif
// System includes
#include <stdint.h>
#include <stdbool.h>
@@ -177,10 +195,10 @@ static inline uint32_t lfs_fromle32(uint32_t a) {
(defined(__BYTE_ORDER__) && defined(__ORDER_BIG_ENDIAN__) && __BYTE_ORDER__ == __ORDER_BIG_ENDIAN__))
return __builtin_bswap32(a);
#else
return (((uint8_t*)&a)[0] << 0) |
(((uint8_t*)&a)[1] << 8) |
(((uint8_t*)&a)[2] << 16) |
(((uint8_t*)&a)[3] << 24);
return ((uint32_t)((uint8_t*)&a)[0] << 0) |
((uint32_t)((uint8_t*)&a)[1] << 8) |
((uint32_t)((uint8_t*)&a)[2] << 16) |
((uint32_t)((uint8_t*)&a)[3] << 24);
#endif
}
@@ -200,10 +218,10 @@ static inline uint32_t lfs_frombe32(uint32_t a) {
(defined(__BYTE_ORDER__) && defined(__ORDER_BIG_ENDIAN__) && __BYTE_ORDER__ == __ORDER_BIG_ENDIAN__)
return a;
#else
return (((uint8_t*)&a)[0] << 24) |
(((uint8_t*)&a)[1] << 16) |
(((uint8_t*)&a)[2] << 8) |
(((uint8_t*)&a)[3] << 0);
return ((uint32_t)((uint8_t*)&a)[0] << 24) |
((uint32_t)((uint8_t*)&a)[1] << 16) |
((uint32_t)((uint8_t*)&a)[2] << 8) |
((uint32_t)((uint8_t*)&a)[3] << 0);
#endif
}
@@ -212,12 +230,22 @@ static inline uint32_t lfs_tobe32(uint32_t a) {
}
// Calculate CRC-32 with polynomial = 0x04c11db7
#ifdef LFS_CRC
static inline uint32_t lfs_crc(uint32_t crc, const void *buffer, size_t size) {
return LFS_CRC(crc, buffer, size);
}
#else
uint32_t lfs_crc(uint32_t crc, const void *buffer, size_t size);
#endif
// Allocate memory, only used if buffers are not provided to littlefs
// Note, memory must be 64-bit aligned
//
// littlefs current has no alignment requirements, as it only allocates
// byte-level buffers.
static inline void *lfs_malloc(size_t size) {
#ifndef LFS_NO_MALLOC
#if defined(LFS_MALLOC)
return LFS_MALLOC(size);
#elif !defined(LFS_NO_MALLOC)
return malloc(size);
#else
(void)size;
@@ -227,7 +255,9 @@ static inline void *lfs_malloc(size_t size) {
// Deallocate memory, only used if buffers are not provided to littlefs
static inline void lfs_free(void *p) {
#ifndef LFS_NO_MALLOC
#if defined(LFS_FREE)
LFS_FREE(p);
#elif !defined(LFS_NO_MALLOC)
free(p);
#else
(void)p;

View File

@@ -123,8 +123,13 @@ typedef struct bench_id {
// bench suites are linked into a custom ld section
#if defined(__APPLE__)
extern struct bench_suite __start__bench_suites __asm("section$start$__DATA$_bench_suites");
extern struct bench_suite __stop__bench_suites __asm("section$end$__DATA$_bench_suites");
#else
extern struct bench_suite __start__bench_suites;
extern struct bench_suite __stop__bench_suites;
#endif
const struct bench_suite *bench_suites = &__start__bench_suites;
#define BENCH_SUITE_COUNT \
@@ -1321,6 +1326,9 @@ void perm_run(
.block_cycles = BLOCK_CYCLES,
.cache_size = CACHE_SIZE,
.lookahead_size = LOOKAHEAD_SIZE,
.compact_thresh = COMPACT_THRESH,
.metadata_max = METADATA_MAX,
.inline_max = INLINE_MAX,
};
struct lfs_emubd_config bdcfg = {

View File

@@ -95,11 +95,14 @@ intmax_t bench_define(size_t define);
#define BLOCK_COUNT_i 5
#define CACHE_SIZE_i 6
#define LOOKAHEAD_SIZE_i 7
#define BLOCK_CYCLES_i 8
#define ERASE_VALUE_i 9
#define ERASE_CYCLES_i 10
#define BADBLOCK_BEHAVIOR_i 11
#define POWERLOSS_BEHAVIOR_i 12
#define COMPACT_THRESH_i 8
#define METADATA_MAX_i 9
#define INLINE_MAX_i 10
#define BLOCK_CYCLES_i 11
#define ERASE_VALUE_i 12
#define ERASE_CYCLES_i 13
#define BADBLOCK_BEHAVIOR_i 14
#define POWERLOSS_BEHAVIOR_i 15
#define READ_SIZE bench_define(READ_SIZE_i)
#define PROG_SIZE bench_define(PROG_SIZE_i)
@@ -109,6 +112,9 @@ intmax_t bench_define(size_t define);
#define BLOCK_COUNT bench_define(BLOCK_COUNT_i)
#define CACHE_SIZE bench_define(CACHE_SIZE_i)
#define LOOKAHEAD_SIZE bench_define(LOOKAHEAD_SIZE_i)
#define COMPACT_THRESH bench_define(COMPACT_THRESH_i)
#define METADATA_MAX bench_define(METADATA_MAX_i)
#define INLINE_MAX bench_define(INLINE_MAX_i)
#define BLOCK_CYCLES bench_define(BLOCK_CYCLES_i)
#define ERASE_VALUE bench_define(ERASE_VALUE_i)
#define ERASE_CYCLES bench_define(ERASE_CYCLES_i)
@@ -124,6 +130,9 @@ intmax_t bench_define(size_t define);
BENCH_DEF(BLOCK_COUNT, ERASE_COUNT/lfs_max(BLOCK_SIZE/ERASE_SIZE,1))\
BENCH_DEF(CACHE_SIZE, lfs_max(64,lfs_max(READ_SIZE,PROG_SIZE))) \
BENCH_DEF(LOOKAHEAD_SIZE, 16) \
BENCH_DEF(COMPACT_THRESH, 0) \
BENCH_DEF(METADATA_MAX, 0) \
BENCH_DEF(INLINE_MAX, 0) \
BENCH_DEF(BLOCK_CYCLES, -1) \
BENCH_DEF(ERASE_VALUE, 0xff) \
BENCH_DEF(ERASE_CYCLES, 0) \
@@ -131,7 +140,7 @@ intmax_t bench_define(size_t define);
BENCH_DEF(POWERLOSS_BEHAVIOR, LFS_EMUBD_POWERLOSS_NOOP)
#define BENCH_GEOMETRY_DEFINE_COUNT 4
#define BENCH_IMPLICIT_DEFINE_COUNT 13
#define BENCH_IMPLICIT_DEFINE_COUNT 16
#endif

View File

@@ -136,8 +136,13 @@ typedef struct test_id {
// test suites are linked into a custom ld section
#if defined(__APPLE__)
extern struct test_suite __start__test_suites __asm("section$start$__DATA$_test_suites");
extern struct test_suite __stop__test_suites __asm("section$end$__DATA$_test_suites");
#else
extern struct test_suite __start__test_suites;
extern struct test_suite __stop__test_suites;
#endif
const struct test_suite *test_suites = &__start__test_suites;
#define TEST_SUITE_COUNT \
@@ -1346,6 +1351,9 @@ static void run_powerloss_none(
.block_cycles = BLOCK_CYCLES,
.cache_size = CACHE_SIZE,
.lookahead_size = LOOKAHEAD_SIZE,
.compact_thresh = COMPACT_THRESH,
.metadata_max = METADATA_MAX,
.inline_max = INLINE_MAX,
#ifdef LFS_MULTIVERSION
.disk_version = DISK_VERSION,
#endif
@@ -1422,6 +1430,9 @@ static void run_powerloss_linear(
.block_cycles = BLOCK_CYCLES,
.cache_size = CACHE_SIZE,
.lookahead_size = LOOKAHEAD_SIZE,
.compact_thresh = COMPACT_THRESH,
.metadata_max = METADATA_MAX,
.inline_max = INLINE_MAX,
#ifdef LFS_MULTIVERSION
.disk_version = DISK_VERSION,
#endif
@@ -1515,6 +1526,9 @@ static void run_powerloss_log(
.block_cycles = BLOCK_CYCLES,
.cache_size = CACHE_SIZE,
.lookahead_size = LOOKAHEAD_SIZE,
.compact_thresh = COMPACT_THRESH,
.metadata_max = METADATA_MAX,
.inline_max = INLINE_MAX,
#ifdef LFS_MULTIVERSION
.disk_version = DISK_VERSION,
#endif
@@ -1606,6 +1620,9 @@ static void run_powerloss_cycles(
.block_cycles = BLOCK_CYCLES,
.cache_size = CACHE_SIZE,
.lookahead_size = LOOKAHEAD_SIZE,
.compact_thresh = COMPACT_THRESH,
.metadata_max = METADATA_MAX,
.inline_max = INLINE_MAX,
#ifdef LFS_MULTIVERSION
.disk_version = DISK_VERSION,
#endif
@@ -1795,6 +1812,9 @@ static void run_powerloss_exhaustive(
.block_cycles = BLOCK_CYCLES,
.cache_size = CACHE_SIZE,
.lookahead_size = LOOKAHEAD_SIZE,
.compact_thresh = COMPACT_THRESH,
.metadata_max = METADATA_MAX,
.inline_max = INLINE_MAX,
#ifdef LFS_MULTIVERSION
.disk_version = DISK_VERSION,
#endif

View File

@@ -88,12 +88,15 @@ intmax_t test_define(size_t define);
#define BLOCK_COUNT_i 5
#define CACHE_SIZE_i 6
#define LOOKAHEAD_SIZE_i 7
#define BLOCK_CYCLES_i 8
#define ERASE_VALUE_i 9
#define ERASE_CYCLES_i 10
#define BADBLOCK_BEHAVIOR_i 11
#define POWERLOSS_BEHAVIOR_i 12
#define DISK_VERSION_i 13
#define COMPACT_THRESH_i 8
#define METADATA_MAX_i 9
#define INLINE_MAX_i 10
#define BLOCK_CYCLES_i 11
#define ERASE_VALUE_i 12
#define ERASE_CYCLES_i 13
#define BADBLOCK_BEHAVIOR_i 14
#define POWERLOSS_BEHAVIOR_i 15
#define DISK_VERSION_i 16
#define READ_SIZE TEST_DEFINE(READ_SIZE_i)
#define PROG_SIZE TEST_DEFINE(PROG_SIZE_i)
@@ -103,6 +106,9 @@ intmax_t test_define(size_t define);
#define BLOCK_COUNT TEST_DEFINE(BLOCK_COUNT_i)
#define CACHE_SIZE TEST_DEFINE(CACHE_SIZE_i)
#define LOOKAHEAD_SIZE TEST_DEFINE(LOOKAHEAD_SIZE_i)
#define COMPACT_THRESH TEST_DEFINE(COMPACT_THRESH_i)
#define METADATA_MAX TEST_DEFINE(METADATA_MAX_i)
#define INLINE_MAX TEST_DEFINE(INLINE_MAX_i)
#define BLOCK_CYCLES TEST_DEFINE(BLOCK_CYCLES_i)
#define ERASE_VALUE TEST_DEFINE(ERASE_VALUE_i)
#define ERASE_CYCLES TEST_DEFINE(ERASE_CYCLES_i)
@@ -119,6 +125,9 @@ intmax_t test_define(size_t define);
TEST_DEF(BLOCK_COUNT, ERASE_COUNT/lfs_max(BLOCK_SIZE/ERASE_SIZE,1)) \
TEST_DEF(CACHE_SIZE, lfs_max(64,lfs_max(READ_SIZE,PROG_SIZE))) \
TEST_DEF(LOOKAHEAD_SIZE, 16) \
TEST_DEF(COMPACT_THRESH, 0) \
TEST_DEF(METADATA_MAX, 0) \
TEST_DEF(INLINE_MAX, 0) \
TEST_DEF(BLOCK_CYCLES, -1) \
TEST_DEF(ERASE_VALUE, 0xff) \
TEST_DEF(ERASE_CYCLES, 0) \
@@ -127,7 +136,7 @@ intmax_t test_define(size_t define);
TEST_DEF(DISK_VERSION, 0)
#define TEST_GEOMETRY_DEFINE_COUNT 4
#define TEST_IMPLICIT_DEFINE_COUNT 14
#define TEST_IMPLICIT_DEFINE_COUNT 17
#endif

View File

@@ -404,12 +404,15 @@ def compile(bench_paths, **args):
f.writeln()
# create suite struct
#
f.writeln('#if defined(__APPLE__)')
f.writeln('__attribute__((section("__DATA,_bench_suites")))')
f.writeln('#else')
# note we place this in the custom bench_suites section with
# minimum alignment, otherwise GCC ups the alignment to
# 32-bytes for some reason
f.writeln('__attribute__((section("_bench_suites"), '
'aligned(1)))')
f.writeln('#endif')
f.writeln('const struct bench_suite __bench__%s__suite = {'
% suite.name)
f.writeln(4*' '+'.name = "%s",' % suite.name)

View File

@@ -73,7 +73,7 @@ def changefile(from_prefix, to_prefix, from_path, to_path, *,
shutil.copystat(from_path, to_path)
if to_path_temp:
os.rename(to_path, from_path)
shutil.move(to_path, from_path)
elif from_path != '-':
os.remove(from_path)

View File

@@ -35,10 +35,10 @@ LEXEMES = {
'assert': ['assert'],
'arrow': ['=>'],
'string': [r'"(?:\\.|[^"])*"', r"'(?:\\.|[^'])\'"],
'paren': ['\(', '\)'],
'paren': [r'\(', r'\)'],
'cmp': CMP.keys(),
'logic': ['\&\&', '\|\|'],
'sep': [':', ';', '\{', '\}', ','],
'logic': [r'\&\&', r'\|\|'],
'sep': [':', ';', r'\{', r'\}', ','],
'op': ['->'], # specifically ops that conflict with cmp
}
@@ -86,6 +86,13 @@ def write_header(f, limit=LIMIT):
f.writeln("}")
f.writeln()
f.writeln("__attribute__((unused))")
f.writeln("static void __pretty_assert_print_ptr(")
f.writeln(" const void *v, size_t size) {")
f.writeln(" (void)size;")
f.writeln(" printf(\"%p\", v);")
f.writeln("}")
f.writeln()
f.writeln("__attribute__((unused))")
f.writeln("static void __pretty_assert_print_mem(")
f.writeln(" const void *v, size_t size) {")
f.writeln(" const uint8_t *v_ = v;")
@@ -183,6 +190,23 @@ def write_header(f, limit=LIMIT):
f.writeln(" _rh, strlen(_rh)); \\")
f.writeln(" } \\")
f.writeln("} while (0)")
for op, cmp in sorted(CMP.items()):
# Only EQ and NE are supported when compared to NULL.
if cmp not in ['eq', 'ne']:
continue
f.writeln("#define __PRETTY_ASSERT_PTR_%s(lh, rh) do { \\"
% cmp.upper())
f.writeln(" const void *_lh = (const void*)(uintptr_t)lh; \\")
f.writeln(" const void *_rh = (const void*)(uintptr_t)rh; \\")
f.writeln(" if (!(_lh %s _rh)) { \\" % op)
f.writeln(" __pretty_assert_fail( \\")
f.writeln(" __FILE__, __LINE__, \\")
f.writeln(" __pretty_assert_print_ptr, \"%s\", \\"
% cmp)
f.writeln(" (const void*){_lh}, 0, \\")
f.writeln(" (const void*){_rh}, 0); \\")
f.writeln(" } \\")
f.writeln("} while (0)")
f.writeln()
f.writeln()
@@ -301,6 +325,8 @@ def p_assert(p):
cmp = p.expect('cmp') ; p.accept('ws')
rh = p_expr(p) ; p.accept('ws')
p.expect(')')
if rh == 'NULL' or lh == 'NULL':
return mkassert('ptr', CMP[cmp], lh, rh)
return mkassert('int', CMP[cmp], lh, rh)
except ParseFailure:
p.pop(state)

View File

@@ -102,9 +102,9 @@ class TestCase:
# the runner itself.
for v_ in csplit(v):
m = re.search(r'\brange\b\s*\('
'(?P<start>[^,\s]*)'
'\s*(?:,\s*(?P<stop>[^,\s]*)'
'\s*(?:,\s*(?P<step>[^,\s]*)\s*)?)?\)',
r'(?P<start>[^,\s]*)'
r'\s*(?:,\s*(?P<stop>[^,\s]*)'
r'\s*(?:,\s*(?P<step>[^,\s]*)\s*)?)?\)',
v_)
if m:
start = (int(m.group('start'), 0)
@@ -163,8 +163,8 @@ class TestSuite:
code_linenos = []
for i, line in enumerate(f):
match = re.match(
'(?P<case>\[\s*cases\s*\.\s*(?P<name>\w+)\s*\])'
'|' '(?P<code>code\s*=)',
r'(?P<case>\[\s*cases\s*\.\s*(?P<name>\w+)\s*\])'
r'|' r'(?P<code>code\s*=)',
line)
if match and match.group('case'):
case_linenos.append((i+1, match.group('name')))
@@ -412,12 +412,15 @@ def compile(test_paths, **args):
f.writeln()
# create suite struct
#
f.writeln('#if defined(__APPLE__)')
f.writeln('__attribute__((section("__DATA,_test_suites")))')
f.writeln('#else')
# note we place this in the custom test_suites section with
# minimum alignment, otherwise GCC ups the alignment to
# 32-bytes for some reason
f.writeln('__attribute__((section("_test_suites"), '
'aligned(1)))')
f.writeln('#endif')
f.writeln('const struct test_suite __test__%s__suite = {'
% suite.name)
f.writeln(4*' '+'.name = "%s",' % suite.name)
@@ -602,9 +605,9 @@ def find_perms(runner_, ids=[], **args):
errors='replace',
close_fds=False)
pattern = re.compile(
'^(?P<case>[^\s]+)'
'\s+(?P<flags>[^\s]+)'
'\s+(?P<filtered>\d+)/(?P<perms>\d+)')
r'^(?P<case>[^\s]+)'
r'\s+(?P<flags>[^\s]+)'
r'\s+(?P<filtered>\d+)/(?P<perms>\d+)')
# skip the first line
for line in it.islice(proc.stdout, 1, None):
m = pattern.match(line)
@@ -632,8 +635,8 @@ def find_perms(runner_, ids=[], **args):
errors='replace',
close_fds=False)
pattern = re.compile(
'^(?P<case>[^\s]+)'
'\s+(?P<path>[^:]+):(?P<lineno>\d+)')
r'^(?P<case>[^\s]+)'
r'\s+(?P<path>[^:]+):(?P<lineno>\d+)')
# skip the first line
for line in it.islice(proc.stdout, 1, None):
m = pattern.match(line)
@@ -676,8 +679,8 @@ def find_path(runner_, id, **args):
errors='replace',
close_fds=False)
pattern = re.compile(
'^(?P<case>[^\s]+)'
'\s+(?P<path>[^:]+):(?P<lineno>\d+)')
r'^(?P<case>[^\s]+)'
r'\s+(?P<path>[^:]+):(?P<lineno>\d+)')
# skip the first line
for line in it.islice(proc.stdout, 1, None):
m = pattern.match(line)
@@ -706,7 +709,7 @@ def find_defines(runner_, id, **args):
errors='replace',
close_fds=False)
defines = co.OrderedDict()
pattern = re.compile('^(?P<define>\w+)=(?P<value>.+)')
pattern = re.compile(r'^(?P<define>\w+)=(?P<value>.+)')
for line in proc.stdout:
m = pattern.match(line)
if m:
@@ -781,12 +784,12 @@ def run_stage(name, runner_, ids, stdout_, trace_, output_, **args):
failures = []
killed = False
pattern = re.compile('^(?:'
'(?P<op>running|finished|skipped|powerloss) '
'(?P<id>(?P<case>[^:]+)[^\s]*)'
'|' '(?P<path>[^:]+):(?P<lineno>\d+):(?P<op_>assert):'
' *(?P<message>.*)'
')$')
pattern = re.compile(r'^(?:'
r'(?P<op>running|finished|skipped|powerloss) '
r'(?P<id>(?P<case>[^:]+)[^\s]*)'
r'|' r'(?P<path>[^:]+):(?P<lineno>\d+):(?P<op_>assert):'
r' *(?P<message>.*)'
r')$')
locals = th.local()
children = set()

View File

@@ -7,17 +7,23 @@ if = 'BLOCK_CYCLES == -1'
defines.FILES = 3
defines.SIZE = '(((BLOCK_SIZE-8)*(BLOCK_COUNT-6)) / FILES)'
defines.GC = [false, true]
defines.COMPACT_THRESH = ['-1', '0', 'BLOCK_SIZE/2']
defines.INFER_BC = [false, true]
code = '''
const char *names[] = {"bacon", "eggs", "pancakes"};
lfs_file_t files[FILES];
lfs_t lfs;
lfs_format(&lfs, cfg) => 0;
lfs_mount(&lfs, cfg) => 0;
struct lfs_config cfg_ = *cfg;
if (INFER_BC) {
cfg_.block_count = 0;
}
lfs_mount(&lfs, &cfg_) => 0;
lfs_mkdir(&lfs, "breakfast") => 0;
lfs_unmount(&lfs) => 0;
lfs_mount(&lfs, cfg) => 0;
lfs_mount(&lfs, &cfg_) => 0;
for (int n = 0; n < FILES; n++) {
char path[1024];
sprintf(path, "breakfast/%s", names[n]);
@@ -38,7 +44,7 @@ code = '''
}
lfs_unmount(&lfs) => 0;
lfs_mount(&lfs, cfg) => 0;
lfs_mount(&lfs, &cfg_) => 0;
for (int n = 0; n < FILES; n++) {
char path[1024];
sprintf(path, "breakfast/%s", names[n]);
@@ -60,17 +66,23 @@ code = '''
defines.FILES = 3
defines.SIZE = '(((BLOCK_SIZE-8)*(BLOCK_COUNT-6)) / FILES)'
defines.GC = [false, true]
defines.COMPACT_THRESH = ['-1', '0', 'BLOCK_SIZE/2']
defines.INFER_BC = [false, true]
code = '''
const char *names[] = {"bacon", "eggs", "pancakes"};
lfs_t lfs;
lfs_format(&lfs, cfg) => 0;
lfs_mount(&lfs, cfg) => 0;
struct lfs_config cfg_ = *cfg;
if (INFER_BC) {
cfg_.block_count = 0;
}
lfs_mount(&lfs, &cfg_) => 0;
lfs_mkdir(&lfs, "breakfast") => 0;
lfs_unmount(&lfs) => 0;
for (int n = 0; n < FILES; n++) {
lfs_mount(&lfs, cfg) => 0;
lfs_mount(&lfs, &cfg_) => 0;
char path[1024];
sprintf(path, "breakfast/%s", names[n]);
lfs_file_t file;
@@ -89,7 +101,7 @@ code = '''
lfs_unmount(&lfs) => 0;
}
lfs_mount(&lfs, cfg) => 0;
lfs_mount(&lfs, &cfg_) => 0;
for (int n = 0; n < FILES; n++) {
char path[1024];
sprintf(path, "breakfast/%s", names[n]);
@@ -111,19 +123,24 @@ code = '''
defines.FILES = 3
defines.SIZE = '(((BLOCK_SIZE-8)*(BLOCK_COUNT-6)) / FILES)'
defines.CYCLES = [1, 10]
defines.INFER_BC = [false, true]
code = '''
const char *names[] = {"bacon", "eggs", "pancakes"};
lfs_file_t files[FILES];
lfs_t lfs;
lfs_format(&lfs, cfg) => 0;
struct lfs_config cfg_ = *cfg;
if (INFER_BC) {
cfg_.block_count = 0;
}
for (int c = 0; c < CYCLES; c++) {
lfs_mount(&lfs, cfg) => 0;
lfs_mount(&lfs, &cfg_) => 0;
lfs_mkdir(&lfs, "breakfast") => 0;
lfs_unmount(&lfs) => 0;
lfs_mount(&lfs, cfg) => 0;
lfs_mount(&lfs, &cfg_) => 0;
for (int n = 0; n < FILES; n++) {
char path[1024];
sprintf(path, "breakfast/%s", names[n]);
@@ -141,7 +158,7 @@ code = '''
}
lfs_unmount(&lfs) => 0;
lfs_mount(&lfs, cfg) => 0;
lfs_mount(&lfs, &cfg_) => 0;
for (int n = 0; n < FILES; n++) {
char path[1024];
sprintf(path, "breakfast/%s", names[n]);
@@ -157,7 +174,7 @@ code = '''
}
lfs_unmount(&lfs) => 0;
lfs_mount(&lfs, cfg) => 0;
lfs_mount(&lfs, &cfg_) => 0;
for (int n = 0; n < FILES; n++) {
char path[1024];
sprintf(path, "breakfast/%s", names[n]);
@@ -173,19 +190,24 @@ code = '''
defines.FILES = 3
defines.SIZE = '(((BLOCK_SIZE-8)*(BLOCK_COUNT-6)) / FILES)'
defines.CYCLES = [1, 10]
defines.INFER_BC = [false, true]
code = '''
const char *names[] = {"bacon", "eggs", "pancakes"};
lfs_t lfs;
lfs_format(&lfs, cfg) => 0;
struct lfs_config cfg_ = *cfg;
if (INFER_BC) {
cfg_.block_count = 0;
}
for (int c = 0; c < CYCLES; c++) {
lfs_mount(&lfs, cfg) => 0;
lfs_mount(&lfs, &cfg_) => 0;
lfs_mkdir(&lfs, "breakfast") => 0;
lfs_unmount(&lfs) => 0;
for (int n = 0; n < FILES; n++) {
lfs_mount(&lfs, cfg) => 0;
lfs_mount(&lfs, &cfg_) => 0;
char path[1024];
sprintf(path, "breakfast/%s", names[n]);
lfs_file_t file;
@@ -230,10 +252,15 @@ code = '''
# exhaustion test
[cases.test_alloc_exhaustion]
defines.INFER_BC = [false, true]
code = '''
lfs_t lfs;
lfs_format(&lfs, cfg) => 0;
lfs_mount(&lfs, cfg) => 0;
struct lfs_config cfg_ = *cfg;
if (INFER_BC) {
cfg_.block_count = 0;
}
lfs_mount(&lfs, &cfg_) => 0;
lfs_file_t file;
lfs_file_open(&lfs, &file, "exhaustion", LFS_O_WRONLY | LFS_O_CREAT);
size_t size = strlen("exhaustion");
@@ -261,7 +288,7 @@ code = '''
lfs_file_close(&lfs, &file) => 0;
lfs_unmount(&lfs) => 0;
lfs_mount(&lfs, cfg) => 0;
lfs_mount(&lfs, &cfg_) => 0;
lfs_file_open(&lfs, &file, "exhaustion", LFS_O_RDONLY);
size = strlen("exhaustion");
lfs_file_size(&lfs, &file) => size;
@@ -274,10 +301,15 @@ code = '''
# exhaustion wraparound test
[cases.test_alloc_exhaustion_wraparound]
defines.SIZE = '(((BLOCK_SIZE-8)*(BLOCK_COUNT-4)) / 3)'
defines.INFER_BC = [false, true]
code = '''
lfs_t lfs;
lfs_format(&lfs, cfg) => 0;
lfs_mount(&lfs, cfg) => 0;
struct lfs_config cfg_ = *cfg;
if (INFER_BC) {
cfg_.block_count = 0;
}
lfs_mount(&lfs, &cfg_) => 0;
lfs_file_t file;
lfs_file_open(&lfs, &file, "padding", LFS_O_WRONLY | LFS_O_CREAT);
@@ -315,7 +347,7 @@ code = '''
lfs_file_close(&lfs, &file) => 0;
lfs_unmount(&lfs) => 0;
lfs_mount(&lfs, cfg) => 0;
lfs_mount(&lfs, &cfg_) => 0;
lfs_file_open(&lfs, &file, "exhaustion", LFS_O_RDONLY);
size = strlen("exhaustion");
lfs_file_size(&lfs, &file) => size;
@@ -328,10 +360,15 @@ code = '''
# dir exhaustion test
[cases.test_alloc_dir_exhaustion]
defines.INFER_BC = [false, true]
code = '''
lfs_t lfs;
lfs_format(&lfs, cfg) => 0;
lfs_mount(&lfs, cfg) => 0;
struct lfs_config cfg_ = *cfg;
if (INFER_BC) {
cfg_.block_count = 0;
}
lfs_mount(&lfs, &cfg_) => 0;
// find out max file size
lfs_mkdir(&lfs, "exhaustiondir") => 0;

View File

@@ -181,6 +181,10 @@ code = '''
defines.N = [5, 11]
if = 'BLOCK_COUNT >= 4*N'
reentrant = true
defines.POWERLOSS_BEHAVIOR = [
'LFS_EMUBD_POWERLOSS_NOOP',
'LFS_EMUBD_POWERLOSS_OOO',
]
code = '''
lfs_t lfs;
int err = lfs_mount(&lfs, cfg);
@@ -439,6 +443,10 @@ code = '''
defines.N = [5, 25]
if = 'N < BLOCK_COUNT/2'
reentrant = true
defines.POWERLOSS_BEHAVIOR = [
'LFS_EMUBD_POWERLOSS_NOOP',
'LFS_EMUBD_POWERLOSS_OOO',
]
code = '''
lfs_t lfs;
int err = lfs_mount(&lfs, cfg);
@@ -717,6 +725,82 @@ code = '''
lfs_unmount(&lfs) => 0;
'''
[cases.test_dirs_remove_read]
defines.N = 10
if = 'N < BLOCK_COUNT/2'
code = '''
lfs_t lfs;
lfs_format(&lfs, cfg) => 0;
lfs_mount(&lfs, cfg) => 0;
lfs_mkdir(&lfs, "prickly-pear") => 0;
for (int i = 0; i < N; i++) {
char path[1024];
sprintf(path, "prickly-pear/cactus%03d", i);
lfs_mkdir(&lfs, path) => 0;
}
lfs_dir_t dir;
lfs_dir_open(&lfs, &dir, "prickly-pear") => 0;
struct lfs_info info;
lfs_dir_read(&lfs, &dir, &info) => 1;
assert(info.type == LFS_TYPE_DIR);
assert(strcmp(info.name, ".") == 0);
lfs_dir_read(&lfs, &dir, &info) => 1;
assert(info.type == LFS_TYPE_DIR);
assert(strcmp(info.name, "..") == 0);
for (int i = 0; i < N; i++) {
char path[1024];
sprintf(path, "cactus%03d", i);
lfs_dir_read(&lfs, &dir, &info) => 1;
assert(info.type == LFS_TYPE_DIR);
assert(strcmp(info.name, path) == 0);
}
lfs_dir_read(&lfs, &dir, &info) => 0;
lfs_dir_close(&lfs, &dir) => 0;
lfs_unmount(&lfs);
for (lfs_size_t k = 0; k < N; k++) {
for (lfs_size_t j = 0; j < N; j++) {
lfs_mount(&lfs, cfg) => 0;
lfs_dir_open(&lfs, &dir, "prickly-pear") => 0;
lfs_dir_read(&lfs, &dir, &info) => 1;
assert(info.type == LFS_TYPE_DIR);
assert(strcmp(info.name, ".") == 0);
lfs_dir_read(&lfs, &dir, &info) => 1;
assert(info.type == LFS_TYPE_DIR);
assert(strcmp(info.name, "..") == 0);
// iterate over dirs < j
for (unsigned i = 0; i < j; i++) {
char path[1024];
sprintf(path, "cactus%03d", i);
lfs_dir_read(&lfs, &dir, &info) => 1;
assert(info.type == LFS_TYPE_DIR);
assert(strcmp(info.name, path) == 0);
}
// remove k while iterating
char path[1024];
sprintf(path, "prickly-pear/cactus%03d", k);
lfs_remove(&lfs, path) => 0;
// iterate over dirs >= j
for (unsigned i = j; i < ((k >= j) ? N-1 : N); i++) {
char path[1024];
sprintf(path, "cactus%03d", (k >= j && i >= k) ? i+1 : i);
lfs_dir_read(&lfs, &dir, &info) => 1;
assert(info.type == LFS_TYPE_DIR);
assert(strcmp(info.name, path) == 0);
}
lfs_dir_read(&lfs, &dir, &info) => 0;
lfs_dir_close(&lfs, &dir) => 0;
// recreate k
sprintf(path, "prickly-pear/cactus%03d", k);
lfs_mkdir(&lfs, path) => 0;
lfs_unmount(&lfs) => 0;
}
}
'''
[cases.test_dirs_other_errors]
code = '''
lfs_t lfs;
@@ -747,6 +831,11 @@ code = '''
lfs_file_open(&lfs, &file, "potato",
LFS_O_WRONLY | LFS_O_CREAT) => LFS_ERR_ISDIR;
lfs_file_open(&lfs, &file, "tacoto", LFS_O_WRONLY | LFS_O_CREAT) => 0;
lfs_file_close(&lfs, &file) => 0;
lfs_rename(&lfs, "tacoto", "potato") => LFS_ERR_ISDIR;
lfs_rename(&lfs, "potato", "tacoto") => LFS_ERR_NOTDIR;
lfs_mkdir(&lfs, "/") => LFS_ERR_EXIST;
lfs_file_open(&lfs, &file, "/",
LFS_O_WRONLY | LFS_O_CREAT | LFS_O_EXCL) => LFS_ERR_EXIST;
@@ -770,6 +859,10 @@ code = '''
lfs_dir_read(&lfs, &dir, &info) => 1;
assert(info.type == LFS_TYPE_DIR);
assert(strcmp(info.name, "potato") == 0);
lfs_dir_read(&lfs, &dir, &info) => 1;
assert(info.type == LFS_TYPE_REG);
assert(strcmp(info.name, "tacoto") == 0);
assert(info.size == 0);
lfs_dir_read(&lfs, &dir, &info) => 0;
lfs_dir_close(&lfs, &dir) => 0;
@@ -790,6 +883,10 @@ code = '''
lfs_dir_read(&lfs, &dir, &info) => 1;
assert(info.type == LFS_TYPE_DIR);
assert(strcmp(info.name, "potato") == 0);
lfs_dir_read(&lfs, &dir, &info) => 1;
assert(info.type == LFS_TYPE_REG);
assert(strcmp(info.name, "tacoto") == 0);
assert(info.size == 0);
lfs_dir_read(&lfs, &dir, &info) => 0;
lfs_dir_close(&lfs, &dir) => 0;
lfs_unmount(&lfs) => 0;

View File

@@ -1,5 +1,6 @@
[cases.test_files_simple]
defines.INLINE_MAX = [0, -1, 8]
code = '''
lfs_t lfs;
lfs_format(&lfs, cfg) => 0;
@@ -25,6 +26,7 @@ code = '''
[cases.test_files_large]
defines.SIZE = [32, 8192, 262144, 0, 7, 8193]
defines.CHUNKSIZE = [31, 16, 33, 1, 1023]
defines.INLINE_MAX = [0, -1, 8]
code = '''
lfs_t lfs;
lfs_format(&lfs, cfg) => 0;
@@ -67,6 +69,7 @@ code = '''
defines.SIZE1 = [32, 8192, 131072, 0, 7, 8193]
defines.SIZE2 = [32, 8192, 131072, 0, 7, 8193]
defines.CHUNKSIZE = [31, 16, 1]
defines.INLINE_MAX = [0, -1, 8]
code = '''
lfs_t lfs;
lfs_format(&lfs, cfg) => 0;
@@ -152,6 +155,7 @@ code = '''
defines.SIZE1 = [32, 8192, 131072, 0, 7, 8193]
defines.SIZE2 = [32, 8192, 131072, 0, 7, 8193]
defines.CHUNKSIZE = [31, 16, 1]
defines.INLINE_MAX = [0, -1, 8]
code = '''
lfs_t lfs;
lfs_format(&lfs, cfg) => 0;
@@ -232,6 +236,7 @@ code = '''
defines.SIZE1 = [32, 8192, 131072, 0, 7, 8193]
defines.SIZE2 = [32, 8192, 131072, 0, 7, 8193]
defines.CHUNKSIZE = [31, 16, 1]
defines.INLINE_MAX = [0, -1, 8]
code = '''
lfs_t lfs;
lfs_format(&lfs, cfg) => 0;
@@ -303,7 +308,12 @@ code = '''
[cases.test_files_reentrant_write]
defines.SIZE = [32, 0, 7, 2049]
defines.CHUNKSIZE = [31, 16, 65]
defines.INLINE_MAX = [0, -1, 8]
reentrant = true
defines.POWERLOSS_BEHAVIOR = [
'LFS_EMUBD_POWERLOSS_NOOP',
'LFS_EMUBD_POWERLOSS_OOO',
]
code = '''
lfs_t lfs;
int err = lfs_mount(&lfs, cfg);
@@ -354,11 +364,20 @@ code = '''
[cases.test_files_reentrant_write_sync]
defines = [
# append (O(n))
{MODE='LFS_O_APPEND', SIZE=[32, 0, 7, 2049], CHUNKSIZE=[31, 16, 65]},
{MODE='LFS_O_APPEND',
SIZE=[32, 0, 7, 2049],
CHUNKSIZE=[31, 16, 65],
INLINE_MAX=[0, -1, 8]},
# truncate (O(n^2))
{MODE='LFS_O_TRUNC', SIZE=[32, 0, 7, 200], CHUNKSIZE=[31, 16, 65]},
{MODE='LFS_O_TRUNC',
SIZE=[32, 0, 7, 200],
CHUNKSIZE=[31, 16, 65],
INLINE_MAX=[0, -1, 8]},
# rewrite (O(n^2))
{MODE=0, SIZE=[32, 0, 7, 200], CHUNKSIZE=[31, 16, 65]},
{MODE=0,
SIZE=[32, 0, 7, 200],
CHUNKSIZE=[31, 16, 65],
INLINE_MAX=[0, -1, 8]},
]
reentrant = true
code = '''
@@ -485,6 +504,10 @@ code = '''
[cases.test_files_many_power_loss]
defines.N = 300
reentrant = true
defines.POWERLOSS_BEHAVIOR = [
'LFS_EMUBD_POWERLOSS_NOOP',
'LFS_EMUBD_POWERLOSS_OOO',
]
code = '''
lfs_t lfs;
int err = lfs_mount(&lfs, cfg);

View File

@@ -195,6 +195,10 @@ code = '''
defines.SIZE = [10, 100]
defines.FILES = [4, 10, 26]
reentrant = true
defines.POWERLOSS_BEHAVIOR = [
'LFS_EMUBD_POWERLOSS_NOOP',
'LFS_EMUBD_POWERLOSS_OOO',
]
code = '''
lfs_t lfs;
lfs_file_t files[FILES];

View File

@@ -357,6 +357,10 @@ code = '''
[cases.test_move_reentrant_file]
reentrant = true
defines.POWERLOSS_BEHAVIOR = [
'LFS_EMUBD_POWERLOSS_NOOP',
'LFS_EMUBD_POWERLOSS_OOO',
]
code = '''
lfs_t lfs;
int err = lfs_mount(&lfs, cfg);
@@ -839,6 +843,10 @@ code = '''
[cases.test_reentrant_dir]
reentrant = true
defines.POWERLOSS_BEHAVIOR = [
'LFS_EMUBD_POWERLOSS_NOOP',
'LFS_EMUBD_POWERLOSS_OOO',
]
code = '''
lfs_t lfs;
int err = lfs_mount(&lfs, cfg);

View File

@@ -98,7 +98,7 @@ code = '''
lfs_mount(&lfs, cfg) => 0;
// create an orphan
lfs_mdir_t orphan;
lfs_alloc_ack(&lfs);
lfs_alloc_ckpoint(&lfs);
lfs_dir_alloc(&lfs, &orphan) => 0;
lfs_dir_commit(&lfs, &orphan, NULL, 0) => 0;
@@ -170,7 +170,7 @@ code = '''
lfs_mount(&lfs, cfg) => 0;
// create an orphan
lfs_mdir_t orphan;
lfs_alloc_ack(&lfs);
lfs_alloc_ckpoint(&lfs);
lfs_dir_alloc(&lfs, &orphan) => 0;
lfs_dir_commit(&lfs, &orphan, NULL, 0) => 0;
@@ -207,7 +207,8 @@ code = '''
[cases.test_orphans_reentrant]
reentrant = true
# TODO fix this case, caused by non-DAG trees
if = '!(DEPTH == 3 && CACHE_SIZE != 64)'
# NOTE the second condition is required
if = '!(DEPTH == 3 && CACHE_SIZE != 64) && 2*FILES < BLOCK_COUNT'
defines = [
{FILES=6, DEPTH=1, CYCLES=20},
{FILES=26, DEPTH=1, CYCLES=20},
@@ -271,3 +272,69 @@ code = '''
lfs_unmount(&lfs) => 0;
'''
# non-reentrant testing for orphans, this is the same as reentrant
# testing, but we test way more states than we could under powerloss
[cases.test_orphans_nonreentrant]
# TODO fix this case, caused by non-DAG trees
# NOTE the second condition is required
if = '!(DEPTH == 3 && CACHE_SIZE != 64) && 2*FILES < BLOCK_COUNT'
defines = [
{FILES=6, DEPTH=1, CYCLES=2000},
{FILES=26, DEPTH=1, CYCLES=2000},
{FILES=3, DEPTH=3, CYCLES=2000},
]
code = '''
lfs_t lfs;
lfs_format(&lfs, cfg) => 0;
lfs_mount(&lfs, cfg) => 0;
uint32_t prng = 1;
const char alpha[] = "abcdefghijklmnopqrstuvwxyz";
for (unsigned i = 0; i < CYCLES; i++) {
// create random path
char full_path[256];
for (unsigned d = 0; d < DEPTH; d++) {
sprintf(&full_path[2*d], "/%c", alpha[TEST_PRNG(&prng) % FILES]);
}
// if it does not exist, we create it, else we destroy
struct lfs_info info;
int res = lfs_stat(&lfs, full_path, &info);
if (res == LFS_ERR_NOENT) {
// create each directory in turn, ignore if dir already exists
for (unsigned d = 0; d < DEPTH; d++) {
char path[1024];
strcpy(path, full_path);
path[2*d+2] = '\0';
int err = lfs_mkdir(&lfs, path);
assert(!err || err == LFS_ERR_EXIST);
}
for (unsigned d = 0; d < DEPTH; d++) {
char path[1024];
strcpy(path, full_path);
path[2*d+2] = '\0';
lfs_stat(&lfs, path, &info) => 0;
assert(strcmp(info.name, &path[2*d+1]) == 0);
assert(info.type == LFS_TYPE_DIR);
}
} else {
// is valid dir?
assert(strcmp(info.name, &full_path[2*(DEPTH-1)+1]) == 0);
assert(info.type == LFS_TYPE_DIR);
// try to delete path in reverse order, ignore if dir is not empty
for (int d = DEPTH-1; d >= 0; d--) {
char path[1024];
strcpy(path, full_path);
path[2*d+2] = '\0';
int err = lfs_remove(&lfs, path);
assert(!err || err == LFS_ERR_NOTEMPTY);
}
lfs_stat(&lfs, full_path, &info) => LFS_ERR_NOENT;
}
}
lfs_unmount(&lfs) => 0;
'''

File diff suppressed because it is too large Load Diff

View File

@@ -341,3 +341,171 @@ code = '''
}
lfs_unmount(&lfs) => 0;
'''
# non-reentrant testing for orphans, this is the same as reentrant
# testing, but we test way more states than we could under powerloss
[cases.test_relocations_nonreentrant]
# TODO fix this case, caused by non-DAG trees
# NOTE the second condition is required
if = '!(DEPTH == 3 && CACHE_SIZE != 64) && 2*FILES < BLOCK_COUNT'
defines = [
{FILES=6, DEPTH=1, CYCLES=2000, BLOCK_CYCLES=1},
{FILES=26, DEPTH=1, CYCLES=2000, BLOCK_CYCLES=1},
{FILES=3, DEPTH=3, CYCLES=2000, BLOCK_CYCLES=1},
]
code = '''
lfs_t lfs;
lfs_format(&lfs, cfg) => 0;
lfs_mount(&lfs, cfg) => 0;
uint32_t prng = 1;
const char alpha[] = "abcdefghijklmnopqrstuvwxyz";
for (unsigned i = 0; i < CYCLES; i++) {
// create random path
char full_path[256];
for (unsigned d = 0; d < DEPTH; d++) {
sprintf(&full_path[2*d], "/%c", alpha[TEST_PRNG(&prng) % FILES]);
}
// if it does not exist, we create it, else we destroy
struct lfs_info info;
int res = lfs_stat(&lfs, full_path, &info);
if (res == LFS_ERR_NOENT) {
// create each directory in turn, ignore if dir already exists
for (unsigned d = 0; d < DEPTH; d++) {
char path[1024];
strcpy(path, full_path);
path[2*d+2] = '\0';
int err = lfs_mkdir(&lfs, path);
assert(!err || err == LFS_ERR_EXIST);
}
for (unsigned d = 0; d < DEPTH; d++) {
char path[1024];
strcpy(path, full_path);
path[2*d+2] = '\0';
lfs_stat(&lfs, path, &info) => 0;
assert(strcmp(info.name, &path[2*d+1]) == 0);
assert(info.type == LFS_TYPE_DIR);
}
} else {
// is valid dir?
assert(strcmp(info.name, &full_path[2*(DEPTH-1)+1]) == 0);
assert(info.type == LFS_TYPE_DIR);
// try to delete path in reverse order, ignore if dir is not empty
for (unsigned d = DEPTH-1; d+1 > 0; d--) {
char path[1024];
strcpy(path, full_path);
path[2*d+2] = '\0';
int err = lfs_remove(&lfs, path);
assert(!err || err == LFS_ERR_NOTEMPTY);
}
lfs_stat(&lfs, full_path, &info) => LFS_ERR_NOENT;
}
}
lfs_unmount(&lfs) => 0;
'''
# non-reentrant testing for relocations, but now with random renames!
[cases.test_relocations_nonreentrant_renames]
# TODO fix this case, caused by non-DAG trees
# NOTE the second condition is required
if = '!(DEPTH == 3 && CACHE_SIZE != 64) && 2*FILES < BLOCK_COUNT'
defines = [
{FILES=6, DEPTH=1, CYCLES=2000, BLOCK_CYCLES=1},
{FILES=26, DEPTH=1, CYCLES=2000, BLOCK_CYCLES=1},
{FILES=3, DEPTH=3, CYCLES=2000, BLOCK_CYCLES=1},
]
code = '''
lfs_t lfs;
lfs_format(&lfs, cfg) => 0;
lfs_mount(&lfs, cfg) => 0;
uint32_t prng = 1;
const char alpha[] = "abcdefghijklmnopqrstuvwxyz";
for (unsigned i = 0; i < CYCLES; i++) {
// create random path
char full_path[256];
for (unsigned d = 0; d < DEPTH; d++) {
sprintf(&full_path[2*d], "/%c", alpha[TEST_PRNG(&prng) % FILES]);
}
// if it does not exist, we create it, else we destroy
struct lfs_info info;
int res = lfs_stat(&lfs, full_path, &info);
assert(!res || res == LFS_ERR_NOENT);
if (res == LFS_ERR_NOENT) {
// create each directory in turn, ignore if dir already exists
for (unsigned d = 0; d < DEPTH; d++) {
char path[1024];
strcpy(path, full_path);
path[2*d+2] = '\0';
int err = lfs_mkdir(&lfs, path);
assert(!err || err == LFS_ERR_EXIST);
}
for (unsigned d = 0; d < DEPTH; d++) {
char path[1024];
strcpy(path, full_path);
path[2*d+2] = '\0';
lfs_stat(&lfs, path, &info) => 0;
assert(strcmp(info.name, &path[2*d+1]) == 0);
assert(info.type == LFS_TYPE_DIR);
}
} else {
assert(strcmp(info.name, &full_path[2*(DEPTH-1)+1]) == 0);
assert(info.type == LFS_TYPE_DIR);
// create new random path
char new_path[256];
for (unsigned d = 0; d < DEPTH; d++) {
sprintf(&new_path[2*d], "/%c", alpha[TEST_PRNG(&prng) % FILES]);
}
// if new path does not exist, rename, otherwise destroy
res = lfs_stat(&lfs, new_path, &info);
assert(!res || res == LFS_ERR_NOENT);
if (res == LFS_ERR_NOENT) {
// stop once some dir is renamed
for (unsigned d = 0; d < DEPTH; d++) {
char path[1024];
strcpy(&path[2*d], &full_path[2*d]);
path[2*d+2] = '\0';
strcpy(&path[128+2*d], &new_path[2*d]);
path[128+2*d+2] = '\0';
int err = lfs_rename(&lfs, path, path+128);
assert(!err || err == LFS_ERR_NOTEMPTY);
if (!err) {
strcpy(path, path+128);
}
}
for (unsigned d = 0; d < DEPTH; d++) {
char path[1024];
strcpy(path, new_path);
path[2*d+2] = '\0';
lfs_stat(&lfs, path, &info) => 0;
assert(strcmp(info.name, &path[2*d+1]) == 0);
assert(info.type == LFS_TYPE_DIR);
}
lfs_stat(&lfs, full_path, &info) => LFS_ERR_NOENT;
} else {
// try to delete path in reverse order,
// ignore if dir is not empty
for (unsigned d = DEPTH-1; d+1 > 0; d--) {
char path[1024];
strcpy(path, full_path);
path[2*d+2] = '\0';
int err = lfs_remove(&lfs, path);
assert(!err || err == LFS_ERR_NOTEMPTY);
}
lfs_stat(&lfs, full_path, &info) => LFS_ERR_NOENT;
}
}
}
lfs_unmount(&lfs) => 0;
'''

View File

@@ -137,6 +137,130 @@ code = '''
lfs_unmount(&lfs) => 0;
'''
# boundary seek and reads
[cases.test_seek_boundary_read]
defines.COUNT = 132
code = '''
lfs_t lfs;
lfs_format(&lfs, cfg) => 0;
lfs_mount(&lfs, cfg) => 0;
lfs_file_t file;
lfs_file_open(&lfs, &file, "kitty",
LFS_O_WRONLY | LFS_O_CREAT | LFS_O_APPEND) => 0;
size_t size = strlen("kittycatcat");
uint8_t buffer[1024];
memcpy(buffer, "kittycatcat", size);
for (int j = 0; j < COUNT; j++) {
lfs_file_write(&lfs, &file, buffer, size);
}
lfs_file_close(&lfs, &file) => 0;
lfs_unmount(&lfs) => 0;
lfs_mount(&lfs, cfg) => 0;
lfs_file_open(&lfs, &file, "kitty", LFS_O_RDONLY) => 0;
size = strlen("kittycatcat");
const lfs_soff_t offsets[] = {
512,
1024-4,
512+1,
1024-4+1,
512-1,
1024-4-1,
512-strlen("kittycatcat"),
1024-4-strlen("kittycatcat"),
512-strlen("kittycatcat")+1,
1024-4-strlen("kittycatcat")+1,
512-strlen("kittycatcat")-1,
1024-4-strlen("kittycatcat")-1,
strlen("kittycatcat")*(COUNT-2)-1,
};
for (unsigned i = 0; i < sizeof(offsets) / sizeof(offsets[0]); i++) {
lfs_soff_t off = offsets[i];
// read @ offset
lfs_file_seek(&lfs, &file, off, LFS_SEEK_SET) => off;
lfs_file_read(&lfs, &file, buffer, size) => size;
memcmp(buffer,
&"kittycatcatkittycatcat"[off % strlen("kittycatcat")],
size) => 0;
// read after
lfs_file_seek(&lfs, &file, off+strlen("kittycatcat")+1, LFS_SEEK_SET)
=> off+strlen("kittycatcat")+1;
lfs_file_read(&lfs, &file, buffer, size) => size;
memcmp(buffer,
&"kittycatcatkittycatcat"[(off+1) % strlen("kittycatcat")],
size) => 0;
// read before
lfs_file_seek(&lfs, &file, off-strlen("kittycatcat")-1, LFS_SEEK_SET)
=> off-strlen("kittycatcat")-1;
lfs_file_read(&lfs, &file, buffer, size) => size;
memcmp(buffer,
&"kittycatcatkittycatcat"[(off-1) % strlen("kittycatcat")],
size) => 0;
// read @ 0
lfs_file_seek(&lfs, &file, 0, LFS_SEEK_SET) => 0;
lfs_file_read(&lfs, &file, buffer, size) => size;
memcmp(buffer, "kittycatcat", size) => 0;
// read @ offset
lfs_file_seek(&lfs, &file, off, LFS_SEEK_SET) => off;
lfs_file_read(&lfs, &file, buffer, size) => size;
memcmp(buffer,
&"kittycatcatkittycatcat"[off % strlen("kittycatcat")],
size) => 0;
// read after
lfs_file_seek(&lfs, &file, off+strlen("kittycatcat")+1, LFS_SEEK_SET)
=> off+strlen("kittycatcat")+1;
lfs_file_read(&lfs, &file, buffer, size) => size;
memcmp(buffer,
&"kittycatcatkittycatcat"[(off+1) % strlen("kittycatcat")],
size) => 0;
// read before
lfs_file_seek(&lfs, &file, off-strlen("kittycatcat")-1, LFS_SEEK_SET)
=> off-strlen("kittycatcat")-1;
lfs_file_read(&lfs, &file, buffer, size) => size;
memcmp(buffer,
&"kittycatcatkittycatcat"[(off-1) % strlen("kittycatcat")],
size) => 0;
// sync
lfs_file_sync(&lfs, &file) => 0;
// read @ 0
lfs_file_seek(&lfs, &file, 0, LFS_SEEK_SET) => 0;
lfs_file_read(&lfs, &file, buffer, size) => size;
memcmp(buffer, "kittycatcat", size) => 0;
// read @ offset
lfs_file_seek(&lfs, &file, off, LFS_SEEK_SET) => off;
lfs_file_read(&lfs, &file, buffer, size) => size;
memcmp(buffer,
&"kittycatcatkittycatcat"[off % strlen("kittycatcat")],
size) => 0;
// read after
lfs_file_seek(&lfs, &file, off+strlen("kittycatcat")+1, LFS_SEEK_SET)
=> off+strlen("kittycatcat")+1;
lfs_file_read(&lfs, &file, buffer, size) => size;
memcmp(buffer,
&"kittycatcatkittycatcat"[(off+1) % strlen("kittycatcat")],
size) => 0;
// read before
lfs_file_seek(&lfs, &file, off-strlen("kittycatcat")-1, LFS_SEEK_SET)
=> off-strlen("kittycatcat")-1;
lfs_file_read(&lfs, &file, buffer, size) => size;
memcmp(buffer,
&"kittycatcatkittycatcat"[(off-1) % strlen("kittycatcat")],
size) => 0;
}
lfs_file_close(&lfs, &file) => 0;
lfs_unmount(&lfs) => 0;
'''
# boundary seek and writes
[cases.test_seek_boundary_write]
defines.COUNT = 132
@@ -160,31 +284,54 @@ code = '''
lfs_file_open(&lfs, &file, "kitty", LFS_O_RDWR) => 0;
size = strlen("hedgehoghog");
const lfs_soff_t offsets[] = {512, 1020, 513, 1021, 511, 1019, 1441};
const lfs_soff_t offsets[] = {
512,
1024-4,
512+1,
1024-4+1,
512-1,
1024-4-1,
512-strlen("kittycatcat"),
1024-4-strlen("kittycatcat"),
512-strlen("kittycatcat")+1,
1024-4-strlen("kittycatcat")+1,
512-strlen("kittycatcat")-1,
1024-4-strlen("kittycatcat")-1,
strlen("kittycatcat")*(COUNT-2)-1,
};
for (unsigned i = 0; i < sizeof(offsets) / sizeof(offsets[0]); i++) {
lfs_soff_t off = offsets[i];
// write @ offset
memcpy(buffer, "hedgehoghog", size);
lfs_file_seek(&lfs, &file, off, LFS_SEEK_SET) => off;
lfs_file_write(&lfs, &file, buffer, size) => size;
// read @ offset
lfs_file_seek(&lfs, &file, off, LFS_SEEK_SET) => off;
lfs_file_read(&lfs, &file, buffer, size) => size;
memcmp(buffer, "hedgehoghog", size) => 0;
// read @ 0
lfs_file_seek(&lfs, &file, 0, LFS_SEEK_SET) => 0;
lfs_file_read(&lfs, &file, buffer, size) => size;
memcmp(buffer, "kittycatcat", size) => 0;
// read @ offset
lfs_file_seek(&lfs, &file, off, LFS_SEEK_SET) => off;
lfs_file_read(&lfs, &file, buffer, size) => size;
memcmp(buffer, "hedgehoghog", size) => 0;
lfs_file_sync(&lfs, &file) => 0;
// read @ 0
lfs_file_seek(&lfs, &file, 0, LFS_SEEK_SET) => 0;
lfs_file_read(&lfs, &file, buffer, size) => size;
memcmp(buffer, "kittycatcat", size) => 0;
// read @ offset
lfs_file_seek(&lfs, &file, off, LFS_SEEK_SET) => off;
lfs_file_read(&lfs, &file, buffer, size) => size;
memcmp(buffer, "hedgehoghog", size) => 0;
@@ -329,6 +476,10 @@ code = '''
# must be power-of-2 for quadratic probing to be exhaustive
defines.COUNT = [4, 64, 128]
reentrant = true
defines.POWERLOSS_BEHAVIOR = [
'LFS_EMUBD_POWERLOSS_NOOP',
'LFS_EMUBD_POWERLOSS_OOO',
]
code = '''
lfs_t lfs;
int err = lfs_mount(&lfs, cfg);
@@ -401,3 +552,111 @@ code = '''
lfs_file_close(&lfs, &file) => 0;
lfs_unmount(&lfs) => 0;
'''
# test possible overflow/underflow conditions
#
# note these need -fsanitize=undefined to consistently detect
# overflow/underflow conditions
[cases.test_seek_filemax]
code = '''
lfs_t lfs;
lfs_format(&lfs, cfg) => 0;
lfs_mount(&lfs, cfg) => 0;
lfs_file_t file;
lfs_file_open(&lfs, &file, "kitty",
LFS_O_WRONLY | LFS_O_CREAT | LFS_O_APPEND) => 0;
uint8_t buffer[1024];
strcpy((char*)buffer, "kittycatcat");
size_t size = strlen((char*)buffer);
lfs_file_write(&lfs, &file, buffer, size) => size;
// seek with LFS_SEEK_SET
lfs_file_seek(&lfs, &file, LFS_FILE_MAX, LFS_SEEK_SET) => LFS_FILE_MAX;
// seek with LFS_SEEK_CUR
lfs_file_seek(&lfs, &file, 0, LFS_SEEK_CUR) => LFS_FILE_MAX;
// the file hasn't changed size, so seek end takes us back to the offset=0
lfs_file_seek(&lfs, &file, +10, LFS_SEEK_END) => size+10;
lfs_file_close(&lfs, &file) => 0;
lfs_unmount(&lfs) => 0;
'''
[cases.test_seek_underflow]
code = '''
lfs_t lfs;
lfs_format(&lfs, cfg) => 0;
lfs_mount(&lfs, cfg) => 0;
lfs_file_t file;
lfs_file_open(&lfs, &file, "kitty",
LFS_O_WRONLY | LFS_O_CREAT | LFS_O_APPEND) => 0;
uint8_t buffer[1024];
strcpy((char*)buffer, "kittycatcat");
size_t size = strlen((char*)buffer);
lfs_file_write(&lfs, &file, buffer, size) => size;
// underflow with LFS_SEEK_CUR, should error
lfs_file_seek(&lfs, &file, -(size+10), LFS_SEEK_CUR) => LFS_ERR_INVAL;
lfs_file_seek(&lfs, &file, -LFS_FILE_MAX, LFS_SEEK_CUR) => LFS_ERR_INVAL;
lfs_file_seek(&lfs, &file, -(size+LFS_FILE_MAX), LFS_SEEK_CUR)
=> LFS_ERR_INVAL;
// underflow with LFS_SEEK_END, should error
lfs_file_seek(&lfs, &file, -(size+10), LFS_SEEK_END) => LFS_ERR_INVAL;
lfs_file_seek(&lfs, &file, -LFS_FILE_MAX, LFS_SEEK_END) => LFS_ERR_INVAL;
lfs_file_seek(&lfs, &file, -(size+LFS_FILE_MAX), LFS_SEEK_END)
=> LFS_ERR_INVAL;
// file pointer should not have changed
lfs_file_tell(&lfs, &file) => size;
lfs_file_close(&lfs, &file) => 0;
lfs_unmount(&lfs) => 0;
'''
[cases.test_seek_overflow]
code = '''
lfs_t lfs;
lfs_format(&lfs, cfg) => 0;
lfs_mount(&lfs, cfg) => 0;
lfs_file_t file;
lfs_file_open(&lfs, &file, "kitty",
LFS_O_WRONLY | LFS_O_CREAT | LFS_O_APPEND) => 0;
uint8_t buffer[1024];
strcpy((char*)buffer, "kittycatcat");
size_t size = strlen((char*)buffer);
lfs_file_write(&lfs, &file, buffer, size) => size;
// seek to LFS_FILE_MAX
lfs_file_seek(&lfs, &file, LFS_FILE_MAX, LFS_SEEK_SET) => LFS_FILE_MAX;
// overflow with LFS_SEEK_CUR, should error
lfs_file_seek(&lfs, &file, +10, LFS_SEEK_CUR) => LFS_ERR_INVAL;
lfs_file_seek(&lfs, &file, +LFS_FILE_MAX, LFS_SEEK_CUR) => LFS_ERR_INVAL;
// LFS_SEEK_SET/END don't care about the current file position, but we can
// still overflow with a large offset
// overflow with LFS_SEEK_SET, should error
lfs_file_seek(&lfs, &file,
+((uint32_t)LFS_FILE_MAX+10),
LFS_SEEK_SET) => LFS_ERR_INVAL;
lfs_file_seek(&lfs, &file,
+((uint32_t)LFS_FILE_MAX+(uint32_t)LFS_FILE_MAX),
LFS_SEEK_SET) => LFS_ERR_INVAL;
// overflow with LFS_SEEK_END, should error
lfs_file_seek(&lfs, &file, +(LFS_FILE_MAX-size+10), LFS_SEEK_END)
=> LFS_ERR_INVAL;
lfs_file_seek(&lfs, &file, +(LFS_FILE_MAX-size+LFS_FILE_MAX), LFS_SEEK_END)
=> LFS_ERR_INVAL;
// file pointer should not have changed
lfs_file_tell(&lfs, &file) => LFS_FILE_MAX;
lfs_file_close(&lfs, &file) => 0;
lfs_unmount(&lfs) => 0;
'''

109
tests/test_shrink.toml Normal file
View File

@@ -0,0 +1,109 @@
# simple shrink
[cases.test_shrink_simple]
defines.BLOCK_COUNT = [10, 15, 20]
defines.AFTER_BLOCK_COUNT = [5, 10, 15, 19]
if = "AFTER_BLOCK_COUNT <= BLOCK_COUNT"
code = '''
#ifdef LFS_SHRINKNONRELOCATING
lfs_t lfs;
lfs_format(&lfs, cfg) => 0;
lfs_mount(&lfs, cfg) => 0;
lfs_fs_grow(&lfs, AFTER_BLOCK_COUNT) => 0;
lfs_unmount(&lfs);
if (BLOCK_COUNT != AFTER_BLOCK_COUNT) {
lfs_mount(&lfs, cfg) => LFS_ERR_INVAL;
}
lfs_t lfs2 = lfs;
struct lfs_config cfg2 = *cfg;
cfg2.block_count = AFTER_BLOCK_COUNT;
lfs2.cfg = &cfg2;
lfs_mount(&lfs2, &cfg2) => 0;
lfs_unmount(&lfs2) => 0;
#endif
'''
# shrinking full
[cases.test_shrink_full]
defines.BLOCK_COUNT = [10, 15, 20]
defines.AFTER_BLOCK_COUNT = [5, 7, 10, 12, 15, 17, 20]
defines.FILES_COUNT = [7, 8, 9, 10]
if = "AFTER_BLOCK_COUNT <= BLOCK_COUNT && FILES_COUNT + 2 < BLOCK_COUNT"
code = '''
#ifdef LFS_SHRINKNONRELOCATING
lfs_t lfs;
lfs_format(&lfs, cfg) => 0;
// create FILES_COUNT files of BLOCK_SIZE - 50 bytes (to avoid inlining)
lfs_mount(&lfs, cfg) => 0;
for (int i = 0; i < FILES_COUNT + 1; i++) {
lfs_file_t file;
char path[1024];
sprintf(path, "file_%03d", i);
lfs_file_open(&lfs, &file, path,
LFS_O_WRONLY | LFS_O_CREAT | LFS_O_EXCL) => 0;
char wbuffer[BLOCK_SIZE];
memset(wbuffer, 'b', BLOCK_SIZE);
// Ensure one block is taken per file, but that files are not inlined.
lfs_size_t size = BLOCK_SIZE - 0x40;
sprintf(wbuffer, "Hi %03d", i);
lfs_file_write(&lfs, &file, wbuffer, size) => size;
lfs_file_close(&lfs, &file) => 0;
}
int err = lfs_fs_grow(&lfs, AFTER_BLOCK_COUNT);
if (err == 0) {
for (int i = 0; i < FILES_COUNT + 1; i++) {
lfs_file_t file;
char path[1024];
sprintf(path, "file_%03d", i);
lfs_file_open(&lfs, &file, path,
LFS_O_RDONLY ) => 0;
lfs_size_t size = BLOCK_SIZE - 0x40;
char wbuffer[size];
char wbuffer_ref[size];
// Ensure one block is taken per file, but that files are not inlined.
memset(wbuffer_ref, 'b', size);
sprintf(wbuffer_ref, "Hi %03d", i);
lfs_file_read(&lfs, &file, wbuffer, BLOCK_SIZE) => size;
lfs_file_close(&lfs, &file) => 0;
for (lfs_size_t j = 0; j < size; j++) {
wbuffer[j] => wbuffer_ref[j];
}
}
} else {
assert(err == LFS_ERR_NOTEMPTY);
}
lfs_unmount(&lfs) => 0;
if (err == 0 ) {
if ( AFTER_BLOCK_COUNT != BLOCK_COUNT ) {
lfs_mount(&lfs, cfg) => LFS_ERR_INVAL;
}
lfs_t lfs2 = lfs;
struct lfs_config cfg2 = *cfg;
cfg2.block_count = AFTER_BLOCK_COUNT;
lfs2.cfg = &cfg2;
lfs_mount(&lfs2, &cfg2) => 0;
for (int i = 0; i < FILES_COUNT + 1; i++) {
lfs_file_t file;
char path[1024];
sprintf(path, "file_%03d", i);
lfs_file_open(&lfs2, &file, path,
LFS_O_RDONLY ) => 0;
lfs_size_t size = BLOCK_SIZE - 0x40;
char wbuffer[size];
char wbuffer_ref[size];
// Ensure one block is taken per file, but that files are not inlined.
memset(wbuffer_ref, 'b', size);
sprintf(wbuffer_ref, "Hi %03d", i);
lfs_file_read(&lfs2, &file, wbuffer, BLOCK_SIZE) => size;
lfs_file_close(&lfs2, &file) => 0;
for (lfs_size_t j = 0; j < size; j++) {
wbuffer[j] => wbuffer_ref[j];
}
}
lfs_unmount(&lfs2);
}
#endif
'''

View File

@@ -14,6 +14,24 @@ code = '''
lfs_unmount(&lfs) => 0;
'''
# make sure the magic string "littlefs" is always at offset=8
[cases.test_superblocks_magic]
code = '''
lfs_t lfs;
lfs_format(&lfs, cfg) => 0;
// check our magic string
//
// note if we lose power we may not have the magic string in both blocks!
// but we don't lose power in this test so we can assert the magic string
// is present in both
uint8_t magic[lfs_max(16, READ_SIZE)];
cfg->read(cfg, 0, 0, magic, lfs_max(16, READ_SIZE)) => 0;
assert(memcmp(&magic[8], "littlefs", 8) == 0);
cfg->read(cfg, 1, 0, magic, lfs_max(16, READ_SIZE)) => 0;
assert(memcmp(&magic[8], "littlefs", 8) == 0);
'''
# mount/unmount from interpretting a previous superblock block_count
[cases.test_superblocks_mount_unknown_block_count]
code = '''
@@ -28,10 +46,13 @@ code = '''
lfs_unmount(&lfs) => 0;
'''
# reentrant format
[cases.test_superblocks_reentrant_format]
reentrant = true
defines.POWERLOSS_BEHAVIOR = [
'LFS_EMUBD_POWERLOSS_NOOP',
'LFS_EMUBD_POWERLOSS_OOO',
]
code = '''
lfs_t lfs;
int err = lfs_mount(&lfs, cfg);
@@ -131,6 +152,39 @@ code = '''
lfs_unmount(&lfs) => 0;
'''
# make sure the magic string "littlefs" is always at offset=8
[cases.test_superblocks_magic_expand]
defines.BLOCK_CYCLES = [32, 33, 1]
defines.N = [10, 100, 1000]
code = '''
lfs_t lfs;
lfs_format(&lfs, cfg) => 0;
lfs_mount(&lfs, cfg) => 0;
for (int i = 0; i < N; i++) {
lfs_file_t file;
lfs_file_open(&lfs, &file, "dummy",
LFS_O_WRONLY | LFS_O_CREAT | LFS_O_EXCL) => 0;
lfs_file_close(&lfs, &file) => 0;
struct lfs_info info;
lfs_stat(&lfs, "dummy", &info) => 0;
assert(strcmp(info.name, "dummy") == 0);
assert(info.type == LFS_TYPE_REG);
lfs_remove(&lfs, "dummy") => 0;
}
lfs_unmount(&lfs) => 0;
// check our magic string
//
// note if we lose power we may not have the magic string in both blocks!
// but we don't lose power in this test so we can assert the magic string
// is present in both
uint8_t magic[lfs_max(16, READ_SIZE)];
cfg->read(cfg, 0, 0, magic, lfs_max(16, READ_SIZE)) => 0;
assert(memcmp(&magic[8], "littlefs", 8) == 0);
cfg->read(cfg, 1, 0, magic, lfs_max(16, READ_SIZE)) => 0;
assert(memcmp(&magic[8], "littlefs", 8) == 0);
'''
# expanding superblock with power cycle
[cases.test_superblocks_expand_power_cycle]
defines.BLOCK_CYCLES = [32, 33, 1]
@@ -174,6 +228,10 @@ code = '''
defines.BLOCK_CYCLES = [2, 1]
defines.N = 24
reentrant = true
defines.POWERLOSS_BEHAVIOR = [
'LFS_EMUBD_POWERLOSS_NOOP',
'LFS_EMUBD_POWERLOSS_OOO',
]
code = '''
lfs_t lfs;
int err = lfs_mount(&lfs, cfg);
@@ -213,6 +271,7 @@ code = '''
lfs_unmount(&lfs) => 0;
'''
# mount with unknown block_count
[cases.test_superblocks_unknown_blocks]
code = '''
@@ -464,3 +523,138 @@ code = '''
assert(memcmp(buffer, "hello!", 6) == 0);
lfs_unmount(&lfs) => 0;
'''
# mount and grow the filesystem
[cases.test_superblocks_shrink]
defines.BLOCK_COUNT = 'ERASE_COUNT'
defines.BLOCK_COUNT_2 = ['ERASE_COUNT/2', 'ERASE_COUNT/4', '2']
defines.KNOWN_BLOCK_COUNT = [true, false]
code = '''
#ifdef LFS_SHRINKNONRELOCATING
lfs_t lfs;
lfs_format(&lfs, cfg) => 0;
if (KNOWN_BLOCK_COUNT) {
cfg->block_count = BLOCK_COUNT;
} else {
cfg->block_count = 0;
}
// mount with block_size < erase_size
lfs_mount(&lfs, cfg) => 0;
struct lfs_fsinfo fsinfo;
lfs_fs_stat(&lfs, &fsinfo) => 0;
assert(fsinfo.block_size == BLOCK_SIZE);
assert(fsinfo.block_count == BLOCK_COUNT);
lfs_unmount(&lfs) => 0;
// same size is a noop
lfs_mount(&lfs, cfg) => 0;
lfs_fs_grow(&lfs, BLOCK_COUNT) => 0;
lfs_fs_stat(&lfs, &fsinfo) => 0;
assert(fsinfo.block_size == BLOCK_SIZE);
assert(fsinfo.block_count == BLOCK_COUNT);
lfs_unmount(&lfs) => 0;
lfs_mount(&lfs, cfg) => 0;
lfs_fs_stat(&lfs, &fsinfo) => 0;
assert(fsinfo.block_size == BLOCK_SIZE);
assert(fsinfo.block_count == BLOCK_COUNT);
lfs_unmount(&lfs) => 0;
// grow to new size
lfs_mount(&lfs, cfg) => 0;
lfs_fs_grow(&lfs, BLOCK_COUNT_2) => 0;
lfs_fs_stat(&lfs, &fsinfo) => 0;
assert(fsinfo.block_size == BLOCK_SIZE);
assert(fsinfo.block_count == BLOCK_COUNT_2);
lfs_unmount(&lfs) => 0;
if (KNOWN_BLOCK_COUNT) {
cfg->block_count = BLOCK_COUNT_2;
} else {
cfg->block_count = 0;
}
lfs_mount(&lfs, cfg) => 0;
lfs_fs_stat(&lfs, &fsinfo) => 0;
assert(fsinfo.block_size == BLOCK_SIZE);
assert(fsinfo.block_count == BLOCK_COUNT_2);
lfs_unmount(&lfs) => 0;
// mounting with the previous size should fail
cfg->block_count = BLOCK_COUNT;
lfs_mount(&lfs, cfg) => LFS_ERR_INVAL;
if (KNOWN_BLOCK_COUNT) {
cfg->block_count = BLOCK_COUNT_2;
} else {
cfg->block_count = 0;
}
// same size is a noop
lfs_mount(&lfs, cfg) => 0;
lfs_fs_grow(&lfs, BLOCK_COUNT_2) => 0;
lfs_fs_stat(&lfs, &fsinfo) => 0;
assert(fsinfo.block_size == BLOCK_SIZE);
assert(fsinfo.block_count == BLOCK_COUNT_2);
lfs_unmount(&lfs) => 0;
lfs_mount(&lfs, cfg) => 0;
lfs_fs_stat(&lfs, &fsinfo) => 0;
assert(fsinfo.block_size == BLOCK_SIZE);
assert(fsinfo.block_count == BLOCK_COUNT_2);
lfs_unmount(&lfs) => 0;
// do some work
lfs_mount(&lfs, cfg) => 0;
lfs_fs_stat(&lfs, &fsinfo) => 0;
assert(fsinfo.block_size == BLOCK_SIZE);
assert(fsinfo.block_count == BLOCK_COUNT_2);
lfs_file_t file;
lfs_file_open(&lfs, &file, "test",
LFS_O_CREAT | LFS_O_EXCL | LFS_O_WRONLY) => 0;
lfs_file_write(&lfs, &file, "hello!", 6) => 6;
lfs_file_close(&lfs, &file) => 0;
lfs_unmount(&lfs) => 0;
lfs_mount(&lfs, cfg) => 0;
lfs_fs_stat(&lfs, &fsinfo) => 0;
assert(fsinfo.block_size == BLOCK_SIZE);
assert(fsinfo.block_count == BLOCK_COUNT_2);
lfs_file_open(&lfs, &file, "test", LFS_O_RDONLY) => 0;
uint8_t buffer[256];
lfs_file_read(&lfs, &file, buffer, sizeof(buffer)) => 6;
lfs_file_close(&lfs, &file) => 0;
assert(memcmp(buffer, "hello!", 6) == 0);
lfs_unmount(&lfs) => 0;
#endif
'''
# test that metadata_max does not cause problems for superblock compaction
[cases.test_superblocks_metadata_max]
defines.METADATA_MAX = [
'lfs_max(512, PROG_SIZE)',
'lfs_max(BLOCK_SIZE/2, PROG_SIZE)',
'BLOCK_SIZE'
]
defines.N = [10, 100, 1000]
code = '''
lfs_t lfs;
lfs_format(&lfs, cfg) => 0;
lfs_mount(&lfs, cfg) => 0;
for (int i = 0; i < N; i++) {
lfs_file_t file;
char name[256];
sprintf(name, "hello%03x", i);
lfs_file_open(&lfs, &file, name,
LFS_O_WRONLY | LFS_O_CREAT | LFS_O_EXCL) => 0;
lfs_file_close(&lfs, &file) => 0;
struct lfs_info info;
lfs_stat(&lfs, name, &info) => 0;
assert(strcmp(info.name, name) == 0);
assert(info.type == LFS_TYPE_REG);
}
lfs_unmount(&lfs) => 0;
'''

View File

@@ -231,6 +231,10 @@ defines.SMALLSIZE = [4, 512]
defines.MEDIUMSIZE = [0, 3, 4, 5, 31, 32, 33, 511, 512, 513, 1023, 1024, 1025]
defines.LARGESIZE = 2048
reentrant = true
defines.POWERLOSS_BEHAVIOR = [
'LFS_EMUBD_POWERLOSS_NOOP',
'LFS_EMUBD_POWERLOSS_OOO',
]
code = '''
lfs_t lfs;
int err = lfs_mount(&lfs, cfg);