Compare commits

...

10 Commits

Author SHA1 Message Date
Christopher Haster
93ecd2f2c1 Limited release script to only draft releases
We probably always want a human involved in the final step of publishing
a release. 1. To allow editing of the release notes before notifying
users, and 2. to avoid any issues with rogue scripts (who doesn't want
1000 release notifications?).

Especially now that releases may trigger additional post-release
scripts.
2024-04-17 12:43:33 -05:00
Christopher Haster
68d28b5114 Merge pull request #966 from BrianPugh/fix-divide-by-zero-full-filesystem
Fix DivideByZero exception when filesystem is completely full.
2024-04-17 12:38:22 -05:00
Christopher Haster
1bc14933b7 Tweaked on-disk config comments for consistency
- Prefer "defaults to blablabla when zero" to hint that this is the
  default state when both explicitly set to zero and implicitly set to
  zero thanks to C's initializers.

- Prefer "disk" when referencing something stored "on disk". Other terms
  can quickly get ambiguous. Except maybe "block device"...
2024-04-17 00:16:20 -05:00
Christopher Haster
01b6a47ea8 Extended test_alloc to test inferred block_count
The block allocator is an area where inferred block counts (when
cfg.block_count=0) are more likely to cause problems.

As is shown by the recent divide-by-zero-exhaustion issue.
2024-04-17 00:04:56 -05:00
Brian Pugh
749a45650f Fix DivideByZero exception when filesystem is completely full. 2024-04-16 20:32:12 -07:00
Christopher Haster
4dd30c1b8f Merge pull request #948 from littlefs-project/fix-sync-ordering
Fix sync issue where data writes could appear before metadata writes
2024-03-08 16:49:59 -06:00
Christopher Haster
5c0d332ecd Merge pull request #939 from Graveflo/master
Add nim-littlefs to readme
2024-03-08 16:49:11 -06:00
Christopher Haster
cf68333a55 Merge pull request #937 from littlefs-project/fix-pending-rm-get-underflow
Fix synthetic move underflows in lfs_dir_get
2024-03-08 16:48:50 -06:00
Ryan McConnell
2752d8c486 add nim-littlefs to readme 2024-02-07 02:53:16 -05:00
Christopher Haster
ddbfcaa722 Fixed synthetic move underflows in lfs_dir_get
By "luck" the previous code somehow managed to not be broken, though it
was possible to traverse the same file twice in lfs_fs_traverse/size
(which is not an error).

The problem was an underlying assumption in lfs_dir_get that it would
never be called when the requested id is pending removal because of a
powerloss. The assumption was either:

1. lfs_dir_find would need to be called first to find the id, and it
   would correctly toss out pending-rms with LFS_ERR_NOENT.

2. lfs_fs_mkconsistent would be implicitly called before any filesystem
   traversals, cleaning up any pending-rms. This is at least true for
   allocator scans.

But, as noted by andriyndev, both lfs_fs_traverse and lfs_fs_size can
call lfs_fs_get with a pending-rm id if called in a readonly context.

---

By "luck" this somehow manages to not break anything:

1. If the pending-rm id is >0, the id is decremented by 1 in lfs_fs_get,
   returning the previous file entry during traversal. Worst case, this
   reports any blocks owned by the previous file entry twice.

   Note this is not an error, lfs_fs_traverse/size may return the same
   block multiple times due to underlying copy-on-write structures.

2. More concerning, if the pending-rm id is 0, the id is decremented by
   1 in lfs_fs_get and underflows. This underflow propagates into the
   type field of the tag we are searching for, decrementing it from
   0x200 (LFS_TYPE_STRUCT) to 0x1ff (LFS_TYPE_INTERNAL(UNUSED)).

   Fortunately, since this happens to underflow to the INTERNAL tag
   type, the type intended to never exist on disk, we should never find
   a matching tag during our lfs_fs_get search. The result? lfs_dir_get
   returns LFS_ERR_NOENT, which is actually what we want.

Also note that LFS_ERR_NOENT does not terminate the mdir traversal
early. If it did we would have missed files instead of duplicating
files, which is a slightly worse situation.

---

The fix is to add an explicit check for pending-rms in lfs_dir_get, just
like in lfs_dir_find. This avoids relying on unintended underflow
propagation, and should make the internal API behavior more consistent.

This is especially important for potential future gc extensions.

Found by andriyndev
2024-02-04 15:12:31 -06:00
5 changed files with 74 additions and 30 deletions

View File

@@ -254,7 +254,7 @@ jobs:
tag_name: env.LFS_VERSION,
name: env.LFS_VERSION | rtrimstr(".0"),
target_commitish: "${{github.event.workflow_run.head_sha}}",
draft: env.LFS_VERSION | endswith(".0"),
draft: true,
body: $release,
}' | tee /dev/stderr)"

View File

@@ -258,6 +258,9 @@ License Identifiers that are here available: http://spdx.org/licenses/
use with the MirageOS library operating system project. It is interoperable
with the reference implementation, with some caveats.
- [nim-littlefs] - A Nim wrapper and API for littlefs. Includes a fuse
implementation based on [littlefs-fuse]
[BSD-3-Clause]: https://spdx.org/licenses/BSD-3-Clause.html
[littlefs-disk-img-viewer]: https://github.com/tniessen/littlefs-disk-img-viewer
[littlefs-fuse]: https://github.com/geky/littlefs-fuse
@@ -274,3 +277,4 @@ License Identifiers that are here available: http://spdx.org/licenses/
[littlefs-python]: https://pypi.org/project/littlefs-python/
[littlefs2-rust]: https://crates.io/crates/littlefs2
[chamelon]: https://github.com/yomimono/chamelon
[nim-littlefs]: https://github.com/Graveflo/nim-littlefs

13
lfs.c
View File

@@ -688,7 +688,7 @@ static int lfs_alloc(lfs_t *lfs, lfs_block_t *block) {
if (lfs->lookahead.ckpoint <= 0) {
LFS_ERROR("No more free space 0x%"PRIx32,
(lfs->lookahead.start + lfs->lookahead.next)
% lfs->cfg->block_count);
% lfs->block_count);
return LFS_ERR_NOSPC;
}
@@ -710,11 +710,14 @@ static lfs_stag_t lfs_dir_getslice(lfs_t *lfs, const lfs_mdir_t *dir,
lfs_tag_t ntag = dir->etag;
lfs_stag_t gdiff = 0;
// synthetic moves
if (lfs_gstate_hasmovehere(&lfs->gdisk, dir->pair) &&
lfs_tag_id(gmask) != 0 &&
lfs_tag_id(lfs->gdisk.tag) <= lfs_tag_id(gtag)) {
// synthetic moves
gdiff -= LFS_MKTAG(0, 1, 0);
lfs_tag_id(gmask) != 0) {
if (lfs_tag_id(lfs->gdisk.tag) == lfs_tag_id(gtag)) {
return LFS_ERR_NOENT;
} else if (lfs_tag_id(lfs->gdisk.tag) < lfs_tag_id(gtag)) {
gdiff -= LFS_MKTAG(0, 1, 0);
}
}
// iterate over dir block backwards (for faster lookups)

16
lfs.h
View File

@@ -59,7 +59,8 @@ typedef uint32_t lfs_block_t;
#endif
// Maximum size of custom attributes in bytes, may be redefined, but there is
// no real benefit to using a smaller LFS_ATTR_MAX. Limited to <= 1022.
// no real benefit to using a smaller LFS_ATTR_MAX. Limited to <= 1022. Stored
// in superblock and must be respected by other littlefs drivers.
#ifndef LFS_ATTR_MAX
#define LFS_ATTR_MAX 1022
#endif
@@ -203,7 +204,8 @@ struct lfs_config {
// program sizes.
lfs_size_t block_size;
// Number of erasable blocks on the device.
// Number of erasable blocks on the device. Defaults to block_count stored
// on disk when zero.
lfs_size_t block_count;
// Number of erase cycles before littlefs evicts metadata logs and moves
@@ -252,18 +254,18 @@ struct lfs_config {
// Optional upper limit on length of file names in bytes. No downside for
// larger names except the size of the info struct which is controlled by
// the LFS_NAME_MAX define. Defaults to LFS_NAME_MAX when zero. Stored in
// superblock and must be respected by other littlefs drivers.
// the LFS_NAME_MAX define. Defaults to LFS_NAME_MAX or name_max stored on
// disk when zero.
lfs_size_t name_max;
// Optional upper limit on files in bytes. No downside for larger files
// but must be <= LFS_FILE_MAX. Defaults to LFS_FILE_MAX when zero. Stored
// in superblock and must be respected by other littlefs drivers.
// but must be <= LFS_FILE_MAX. Defaults to LFS_FILE_MAX or file_max stored
// on disk when zero.
lfs_size_t file_max;
// Optional upper limit on custom attributes in bytes. No downside for
// larger attributes size but must be <= LFS_ATTR_MAX. Defaults to
// LFS_ATTR_MAX when zero.
// LFS_ATTR_MAX or attr_max stored on disk when zero.
lfs_size_t attr_max;
// Optional upper limit on total space given to metadata pairs in bytes. On

View File

@@ -8,17 +8,22 @@ defines.FILES = 3
defines.SIZE = '(((BLOCK_SIZE-8)*(BLOCK_COUNT-6)) / FILES)'
defines.GC = [false, true]
defines.COMPACT_THRESH = ['-1', '0', 'BLOCK_SIZE/2']
defines.INFER_BC = [false, true]
code = '''
const char *names[] = {"bacon", "eggs", "pancakes"};
lfs_file_t files[FILES];
lfs_t lfs;
lfs_format(&lfs, cfg) => 0;
lfs_mount(&lfs, cfg) => 0;
struct lfs_config cfg_ = *cfg;
if (INFER_BC) {
cfg_.block_count = 0;
}
lfs_mount(&lfs, &cfg_) => 0;
lfs_mkdir(&lfs, "breakfast") => 0;
lfs_unmount(&lfs) => 0;
lfs_mount(&lfs, cfg) => 0;
lfs_mount(&lfs, &cfg_) => 0;
for (int n = 0; n < FILES; n++) {
char path[1024];
sprintf(path, "breakfast/%s", names[n]);
@@ -39,7 +44,7 @@ code = '''
}
lfs_unmount(&lfs) => 0;
lfs_mount(&lfs, cfg) => 0;
lfs_mount(&lfs, &cfg_) => 0;
for (int n = 0; n < FILES; n++) {
char path[1024];
sprintf(path, "breakfast/%s", names[n]);
@@ -62,17 +67,22 @@ defines.FILES = 3
defines.SIZE = '(((BLOCK_SIZE-8)*(BLOCK_COUNT-6)) / FILES)'
defines.GC = [false, true]
defines.COMPACT_THRESH = ['-1', '0', 'BLOCK_SIZE/2']
defines.INFER_BC = [false, true]
code = '''
const char *names[] = {"bacon", "eggs", "pancakes"};
lfs_t lfs;
lfs_format(&lfs, cfg) => 0;
lfs_mount(&lfs, cfg) => 0;
struct lfs_config cfg_ = *cfg;
if (INFER_BC) {
cfg_.block_count = 0;
}
lfs_mount(&lfs, &cfg_) => 0;
lfs_mkdir(&lfs, "breakfast") => 0;
lfs_unmount(&lfs) => 0;
for (int n = 0; n < FILES; n++) {
lfs_mount(&lfs, cfg) => 0;
lfs_mount(&lfs, &cfg_) => 0;
char path[1024];
sprintf(path, "breakfast/%s", names[n]);
lfs_file_t file;
@@ -91,7 +101,7 @@ code = '''
lfs_unmount(&lfs) => 0;
}
lfs_mount(&lfs, cfg) => 0;
lfs_mount(&lfs, &cfg_) => 0;
for (int n = 0; n < FILES; n++) {
char path[1024];
sprintf(path, "breakfast/%s", names[n]);
@@ -113,19 +123,24 @@ code = '''
defines.FILES = 3
defines.SIZE = '(((BLOCK_SIZE-8)*(BLOCK_COUNT-6)) / FILES)'
defines.CYCLES = [1, 10]
defines.INFER_BC = [false, true]
code = '''
const char *names[] = {"bacon", "eggs", "pancakes"};
lfs_file_t files[FILES];
lfs_t lfs;
lfs_format(&lfs, cfg) => 0;
struct lfs_config cfg_ = *cfg;
if (INFER_BC) {
cfg_.block_count = 0;
}
for (int c = 0; c < CYCLES; c++) {
lfs_mount(&lfs, cfg) => 0;
lfs_mount(&lfs, &cfg_) => 0;
lfs_mkdir(&lfs, "breakfast") => 0;
lfs_unmount(&lfs) => 0;
lfs_mount(&lfs, cfg) => 0;
lfs_mount(&lfs, &cfg_) => 0;
for (int n = 0; n < FILES; n++) {
char path[1024];
sprintf(path, "breakfast/%s", names[n]);
@@ -143,7 +158,7 @@ code = '''
}
lfs_unmount(&lfs) => 0;
lfs_mount(&lfs, cfg) => 0;
lfs_mount(&lfs, &cfg_) => 0;
for (int n = 0; n < FILES; n++) {
char path[1024];
sprintf(path, "breakfast/%s", names[n]);
@@ -159,7 +174,7 @@ code = '''
}
lfs_unmount(&lfs) => 0;
lfs_mount(&lfs, cfg) => 0;
lfs_mount(&lfs, &cfg_) => 0;
for (int n = 0; n < FILES; n++) {
char path[1024];
sprintf(path, "breakfast/%s", names[n]);
@@ -175,19 +190,24 @@ code = '''
defines.FILES = 3
defines.SIZE = '(((BLOCK_SIZE-8)*(BLOCK_COUNT-6)) / FILES)'
defines.CYCLES = [1, 10]
defines.INFER_BC = [false, true]
code = '''
const char *names[] = {"bacon", "eggs", "pancakes"};
lfs_t lfs;
lfs_format(&lfs, cfg) => 0;
struct lfs_config cfg_ = *cfg;
if (INFER_BC) {
cfg_.block_count = 0;
}
for (int c = 0; c < CYCLES; c++) {
lfs_mount(&lfs, cfg) => 0;
lfs_mount(&lfs, &cfg_) => 0;
lfs_mkdir(&lfs, "breakfast") => 0;
lfs_unmount(&lfs) => 0;
for (int n = 0; n < FILES; n++) {
lfs_mount(&lfs, cfg) => 0;
lfs_mount(&lfs, &cfg_) => 0;
char path[1024];
sprintf(path, "breakfast/%s", names[n]);
lfs_file_t file;
@@ -232,10 +252,15 @@ code = '''
# exhaustion test
[cases.test_alloc_exhaustion]
defines.INFER_BC = [false, true]
code = '''
lfs_t lfs;
lfs_format(&lfs, cfg) => 0;
lfs_mount(&lfs, cfg) => 0;
struct lfs_config cfg_ = *cfg;
if (INFER_BC) {
cfg_.block_count = 0;
}
lfs_mount(&lfs, &cfg_) => 0;
lfs_file_t file;
lfs_file_open(&lfs, &file, "exhaustion", LFS_O_WRONLY | LFS_O_CREAT);
size_t size = strlen("exhaustion");
@@ -263,7 +288,7 @@ code = '''
lfs_file_close(&lfs, &file) => 0;
lfs_unmount(&lfs) => 0;
lfs_mount(&lfs, cfg) => 0;
lfs_mount(&lfs, &cfg_) => 0;
lfs_file_open(&lfs, &file, "exhaustion", LFS_O_RDONLY);
size = strlen("exhaustion");
lfs_file_size(&lfs, &file) => size;
@@ -276,10 +301,15 @@ code = '''
# exhaustion wraparound test
[cases.test_alloc_exhaustion_wraparound]
defines.SIZE = '(((BLOCK_SIZE-8)*(BLOCK_COUNT-4)) / 3)'
defines.INFER_BC = [false, true]
code = '''
lfs_t lfs;
lfs_format(&lfs, cfg) => 0;
lfs_mount(&lfs, cfg) => 0;
struct lfs_config cfg_ = *cfg;
if (INFER_BC) {
cfg_.block_count = 0;
}
lfs_mount(&lfs, &cfg_) => 0;
lfs_file_t file;
lfs_file_open(&lfs, &file, "padding", LFS_O_WRONLY | LFS_O_CREAT);
@@ -317,7 +347,7 @@ code = '''
lfs_file_close(&lfs, &file) => 0;
lfs_unmount(&lfs) => 0;
lfs_mount(&lfs, cfg) => 0;
lfs_mount(&lfs, &cfg_) => 0;
lfs_file_open(&lfs, &file, "exhaustion", LFS_O_RDONLY);
size = strlen("exhaustion");
lfs_file_size(&lfs, &file) => size;
@@ -330,10 +360,15 @@ code = '''
# dir exhaustion test
[cases.test_alloc_dir_exhaustion]
defines.INFER_BC = [false, true]
code = '''
lfs_t lfs;
lfs_format(&lfs, cfg) => 0;
lfs_mount(&lfs, cfg) => 0;
struct lfs_config cfg_ = *cfg;
if (INFER_BC) {
cfg_.block_count = 0;
}
lfs_mount(&lfs, &cfg_) => 0;
// find out max file size
lfs_mkdir(&lfs, "exhaustiondir") => 0;