forked from Imagelibrary/littlefs
Implemented scratch file basics
"Scratch files" are a new file type added to solve the zero-sized
file problem. Though they have a few other uses that may be quite
valuable.
The "zero-sized file problem" is a common surprise for users, where what
seems like a simple file create+write operation:
lfs_file_open(&lfs, &file, "hi",
LFS_O_WRONLY | LFS_O_CREAT | LFS_O_EXCL);
lfs_file_write(&lfs, &file, "hello!", strlen("hello!"));
lfs_file_close(&lfs, &file);
Can end up create a zero-sized file under powerloss, breaking user
assumptions and their code.
The tricky thing is that this is actually correct behavior as defined by
POSIX. `open` with O_CREAT creats a file entry immediately, which is
initially zero-sized. And the fact that power can be lost between `open`
and `close` isn't really avoidable.
But this is a common enough footgun that it's probably worth deviating
from POSIX here.
But how to avoid zero-sized files exactly? First thought: Delay the file
creation until sync/close, tracking uncreated files in-device until
then. This solves the problem and avoids any intermediary state if we
lose power, but came with a number of headaches:
1. Since we delay file creation, we don't immediately write the filename
to disk on open. This implies we need to keep the filename allocated
in RAM until the first sync/close call.
The requirement to keep the filename allocated for new files until
first sync/close could be added to open, and with the option to call
sync immediately to save the filename (and accept the risk of
zero-sized files), I don't think it would be _that_ bad of an API.
But it would still be pretty bad. Extra bad because 1. there's no
way to warn on misuse at compile-time, 2. use-after-free bugs have a
tendency to go unnoticed annoyingly often, 3. it's a regression from
the previous API, and 4. who the heck reads the more-or-less same
`open` documentation for every filesystem they adopt.
2. Without an allocated mid, tracking files internally gets a lot
harder. The best option I could think of was to keep the opened-file
linked-list sorted by mid + (in-device) file name.
This did not feel like a great solutiona and was going to add more
code cost.
3. Handling mdir splits containing uncreated files adds another
headache. Complicated lfsr_mdir_estimate further as it needs to
decide in which mdir the uncreated files will end up, and potentially
split on a filename that isn't even created yet.
4. Since the number of uncreated files can be potentially unbounded, you
can't prevent an mdir from filling up with only uncreated files. On
disk this ends up looking like an "empty" mdir, which need specially
handling in littlefs to reclaim after powerloss.
Support for empty mdirs -- the orphaned mdir scan -- was already
added earlier. We already scan each mdir to build gstate, so it
doesn't really add much cost.
Notice that last bullet point? We already scan each mdir during mount.
Why not, instead of scanning for orphaned mdirs, scan for orphaned
files?
So this leads to the idea of "scratch files". Instead of actually
delaying file creation, fake it. Create a scratch file during open, and
on the first sync/close, convert it to a regular file. If we lose power,
scan for scratch files during mount, and remove them on first write.
Some tradeoffs:
1. The orphan scan for scratch files is a bit more expensive than for
mdirs on storage with large block sizes. We need to look at each file
entry vs just each mdir, which pushed the runtime up to O(BlogB) vs
O(B).
Though if you also consider large mtrees, the worst case is still
O(nlogn).
2. Creating intermediate scratch files adds another commit to file
creation.
This is probably not a big issue for flash, but may be more of a
concern on devices with large prog sizes.
3. Scratch files complicate unrelated mkdir/rename/etc code a bit, since
we need to consider what happens when the dest is a scratch file.
But the end result is simple. And simple is good. Both for
implementation headaches, and code size. Even if the on-disk state is
conceptually more complicated.
You may have noticed these scratch files are basically isomorphic to
just setting an "uncreated" flag on the file, and that's true. There may
have been a simpler route to end up with the design, but hey, as long as
it works.
As a plus, scratch files present a solution for a couple other things:
1. Removing an open file can become a scratch file until closed.
2. Scratch files can be used as temporary files. Open a file with
O_DESYNC and never call sync and you have yourself a temporary file.
Maybe in the future we should add O_TMPFILE to avoid the need for
unique filenames, but that is low priority.
This commit is contained in:
Christopher Haster
@@ -25,6 +25,7 @@ TAG_GRMDELTA = 0x0100
|
||||
TAG_NAME = 0x0200
|
||||
TAG_REG = 0x0201
|
||||
TAG_DIR = 0x0202
|
||||
TAG_SCRATCH = 0x0203
|
||||
TAG_BOOKMARK = 0x0204
|
||||
TAG_STRUCT = 0x0300
|
||||
TAG_DATA = 0x0300
|
||||
@@ -218,6 +219,7 @@ def tagrepr(tag, w, size, off=None):
|
||||
'name' if (tag & 0xfff) == TAG_NAME
|
||||
else 'reg' if (tag & 0xfff) == TAG_REG
|
||||
else 'dir' if (tag & 0xfff) == TAG_DIR
|
||||
else 'scratch' if (tag & 0xfff) == TAG_SCRATCH
|
||||
else 'bookmark' if (tag & 0xfff) == TAG_BOOKMARK
|
||||
else 'name 0x%02x' % (tag & 0xff),
|
||||
' w%d' % w if w else '',
|
||||
@@ -1179,7 +1181,7 @@ class GState:
|
||||
yield grepr(tag, data), tag, data
|
||||
|
||||
def frepr(mdir, rid, tag):
|
||||
if tag == TAG_REG:
|
||||
if tag == TAG_REG or tag == TAG_SCRATCH:
|
||||
size = 0
|
||||
structs = []
|
||||
# inlined data?
|
||||
@@ -1205,7 +1207,9 @@ def frepr(mdir, rid, tag):
|
||||
weight, block, trunk, cksum = frombtree(data)
|
||||
size = max(size, weight)
|
||||
structs.append('btree 0x%x.%x' % (block, trunk))
|
||||
return 'reg %s' % ', '.join(it.chain(['%d' % size], structs))
|
||||
return '%s %s' % (
|
||||
'scratch' if tag == TAG_SCRATCH else 'reg',
|
||||
', '.join(it.chain(['%d' % size], structs)))
|
||||
|
||||
elif tag == TAG_DIR:
|
||||
# read the did
|
||||
@@ -1979,6 +1983,9 @@ def main(disk, mroots=None, *,
|
||||
# skip bookmarks
|
||||
if tag == TAG_BOOKMARK:
|
||||
continue
|
||||
# skip scratch files
|
||||
if tag == TAG_SCRATCH:
|
||||
continue
|
||||
# skip grmed entries
|
||||
if (max(mbid-max(mw-1, 0), 0), rid) in gstate.grm:
|
||||
continue
|
||||
@@ -2027,7 +2034,9 @@ def main(disk, mroots=None, *,
|
||||
print('%s%12s %*s %-*s %s%s%s' % (
|
||||
'\x1b[31m' if color and not grmed and notes
|
||||
else '\x1b[90m'
|
||||
if color and (grmed or tag == TAG_BOOKMARK)
|
||||
if color and (grmed
|
||||
or tag == TAG_BOOKMARK
|
||||
or tag == TAG_SCRATCH)
|
||||
else '',
|
||||
'{%s}:' % ','.join('%04x' % block
|
||||
for block in it.chain([mdir.block],
|
||||
@@ -2043,7 +2052,10 @@ def main(disk, mroots=None, *,
|
||||
frepr(mdir, rid, tag),
|
||||
' (%s)' % ', '.join(notes) if notes else '',
|
||||
'\x1b[m' if color and (
|
||||
notes or grmed or tag == TAG_BOOKMARK)
|
||||
notes
|
||||
or grmed
|
||||
or tag == TAG_BOOKMARK
|
||||
or tag == TAG_SCRATCH)
|
||||
else ''))
|
||||
pmbid = mbid
|
||||
|
||||
@@ -2087,7 +2099,8 @@ def main(disk, mroots=None, *,
|
||||
line))
|
||||
|
||||
# print file contents?
|
||||
if tag == TAG_REG and args.get('structs'):
|
||||
if ((tag == TAG_REG or tag == TAG_SCRATCH)
|
||||
and args.get('structs')):
|
||||
# inlined sprout?
|
||||
done, rid_, tag_, w_, j, d, data, _ = mdir.lookup(
|
||||
rid, TAG_DATA)
|
||||
|
||||
Reference in New Issue
Block a user