MagnitudeOS/littlefs
1
0
Fork 0
forked from Imagelibrary/littlefs
Code Pull Requests Activity

scripts: Reworked tracebd.py, needs cleanup

Browse Source 
It's a mess but it's working. Still a number of TODOs to cleanup...

This adopts all of the changes in dbgbmap.py/dbgbmapd3.py, block
grouping, nested curves, Canvas, Attrs, etc:

- Like dbgbmap.py, we now group by block first before applying space
  filling curves, using nested space filling curves to render byte-level
  operations.

  Python's ft.lru_cache really shines here.

  The previous behavior is still available via -u/--contiguous

- Adopted most features in dbgbmap.py, so --to-scale, -t/--tiny, custom
  --title strings, etc.

- Adopted Attrs so now chars/coloring can be customized with
  -./--add-char, -,/--add-wear-char, -C/--add-color,
  -G/--add-wear-color.

- Renamed -R/--reset -> --volatile, which is a much better name.

- Wear is now colored cyan -> white -> read, which is a bit more
  visually interesting. And we're not using cyan in any scripts yet.

In addition to the new stuff, there were a few simplifications:

- We no longer support sub-char -n/--lines with -:/--dots or
  -⣿/--braille. Too complicated, required Canvas state hacks to get
  working, and wasn't super useful.

  We probably want to avoid doing too much cleverness with -:/--dots and
  -⣿/--braille since we can't color sub-chars.

- Dropped -@/--blocks byte-level range stuff. This was just not worth
  the amount of complexity it added. -@/--blocks is now limited to
  simple block ranges. High-level scripts should stick to high-level
  options.

- No fancy/complicated Bmap class. The bmap object is just a dict of
  TraceBlocks which contain RangeSets for relevant operations.

  Actually the new RangeSet class deserves a mention but this commit
  message is probably already too long.

  RangeSet is a decently efficient set of, well, ranges, that can be
  merged and queried. In a lower-level language it should be implemented
  as a binary tree, but in Python we're just using a sorted list because
  we're probably not going to be able to beat O(n) list operations.

- Wear is tracked at the block level, no reason to overcomplicate this.

- We no longer resize based on new info. Instead we either expect a
  -b/--block-size argument or wait until first bd init call.

  We can probably drop the block size in BD_TRACE statements now, but
  that's a TODO item.

- Instead of one amalgamated regex, we use string searches to figure out
  the bd op and then smaller regexes to parse. Lesson learned here:
  Python's string search is very fast (compared to regex).

- We do _not_ support labels on blocks like we do in treemap.py/
  codemap.py. It's less useful here and would just be more hassle.

I also tried to reorganize main a bit to mirror the simple two-main
approach in dbgbmap.py and other ascii-rendering scripts, but it's a bit
difficult here since trace info is very stateful. Building up main
functions in the main main function seemed to work well enough:

  main -+-> main_ -> trace__ (main thread)
        '-> draw_ -> draw__ (daemon thread)

---

You may note some weirdness going on with flags. That's me trying to
avoid upcoming flag conflicts.

I think we want -n/--lines in more scripts, now that it's relatively
self-contained, but this conflicts with -n/--namespace-depth in
codemap[d3].py, and risks conflict with -N/--notes in csv.py which may
end up with namespace-related functionality in the future.

I ended up hijacking -_, but this conflicted with -_/--add-line-char in
plot.py, but that's ok because we also want a common "secondary char"
flag for wear in tracebd.py... Long story short I ended up moving a
bunch of flags around:

- added                   -n/--lines
- -n/--namespace-depth -> -_/--namespace-depth
- -N/--notes           -> -N/--notes
- -./--add-char        -> -./--add-char
- -_/--add-line-char   -> -,/--add-line-char
- added                   -,/--add-wear-char
- -C/--color           -> -C/--add-color
- added                -> -G/--add-wear-color

Worth it? Dunno.
This commit is contained in:
Christopher Haster
2025-04-10 02:35:47 -05:00
parent 33e2e5b1db
commit d5c0e142f0
7 changed files with 2512 additions and 897 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
scripts/codemap.py
View File

@@ -1382,7 +1382,7 @@ if __name__ == "__main__":
action=AppendPath, action=AppendPath,
help="Input *.json files.") help="Input *.json files.")
parser.add_argument( parser.add_argument(
'-n', '--namespace-depth', '-_', '--namespace-depth',
nargs='?', nargs='?',
type=lambda x: int(x, 0), type=lambda x: int(x, 0),
const=0, const=0,

2
scripts/codemapd3.py
View File

@@ -2090,7 +2090,7 @@ if __name__ == "__main__":
required=True, required=True,
help="Output *.svg file.") help="Output *.svg file.")
parser.add_argument( parser.add_argument(
'-n', '--namespace-depth', '-_', '--namespace-depth',
nargs='?', nargs='?',
type=lambda x: int(x, 0), type=lambda x: int(x, 0),
const=0, const=0,

136
scripts/dbgbmap.py
View File

@@ -3645,6 +3645,7 @@ else:
else: else:
self.add_watch(path, flags) self.add_watch(path, flags)
# TODO negative maxlen from terminal height? like -H nowadays?
class RingIO: class RingIO:
def __init__(self, maxlen=None, head=False): def __init__(self, maxlen=None, head=False):
self.maxlen = maxlen self.maxlen = maxlen
@@ -4392,23 +4393,25 @@ def main_(f, disk, mroots=None, *,
corrupted = not bool(lfs) corrupted = not bool(lfs)
# if we can't figure out the block_count, guess # if we can't figure out the block_count, guess
block_size_ = block_size
block_count_ = block_count
if block_count is None: if block_count is None:
if lfs.config.geometry is not None: if lfs.config.geometry is not None:
block_count = lfs.config.geometry.block_count block_count_ = lfs.config.geometry.block_count
else: else:
f_.seek(0, os.SEEK_END) f_.seek(0, os.SEEK_END)
block_count = mt.ceil(f_.tell() / block_size) block_count_ = mt.ceil(f_.tell() / block_size)
# flatten blocks, default to all blocks # flatten blocks, default to all blocks
blocks = list( blocks_ = list(
range(blocks.start or 0, blocks.stop or block_count) range(blocks.start or 0, blocks.stop or block_count_)
if isinstance(blocks, slice) if isinstance(blocks, slice)
else range(blocks, blocks+1) else range(blocks, blocks+1)
if blocks if blocks
else range(block_count)) else range(block_count_))
# traverse the filesystem and create a block map # traverse the filesystem and create a block map
bmap = {b: BmapBlock(b, 'unused') for b in blocks} bmap = {b: BmapBlock(b, 'unused') for b in blocks_}
mdir_count = 0 mdir_count = 0
btree_count = 0 btree_count = 0
data_count = 0 data_count = 0
@@ -4455,21 +4458,21 @@ def main_(f, disk, mroots=None, *,
else: else:
bmap[b] = BmapBlock(b, 'conflict', bmap[b] = BmapBlock(b, 'conflict',
[bmap[b].value, child], [bmap[b].value, child],
range(block_size)) range(block_size_))
corrupted = True corrupted = True
# corrupt metadata? # corrupt metadata?
elif (not no_ckmeta elif (not no_ckmeta
and isinstance(child, (Mdir, Rbyd)) and isinstance(child, (Mdir, Rbyd))
and not child): and not child):
bmap[b] = BmapBlock(b, 'corrupt', child, range(block_size)) bmap[b] = BmapBlock(b, 'corrupt', child, range(block_size_))
corrupted = True corrupted = True
# corrupt data? # corrupt data?
elif (not no_ckdata elif (not no_ckdata
and isinstance(child, Bptr) and isinstance(child, Bptr)
and not child): and not child):
bmap[b] = BmapBlock(b, 'corrupt', child, range(block_size)) bmap[b] = BmapBlock(b, 'corrupt', child, range(block_size_))
corrupted = True corrupted = True
# normal block # normal block
@@ -4509,44 +4512,47 @@ def main_(f, disk, mroots=None, *,
# if contiguous, compute the global curve # if contiguous, compute the global curve
if contiguous: if contiguous:
min__ = min(bmap.keys(), default=0) global_block = min(bmap.keys(), default=0)
curve__ = list(curve(canvas.width, canvas.height)) global_curve = list(curve(canvas.width, canvas.height))
# if blocky, figure out block sizes/locations # if blocky, figure out block sizes/locations
else: else:
# figure out block_cols/block_rows # figure out block_cols_/block_rows_
if block_cols is not None and block_rows is not None: if block_cols is not None and block_rows is not None:
pass block_cols_ = block_cols
block_rows_ = block_rows
elif block_rows is not None: elif block_rows is not None:
block_cols = mt.ceil(len(bmap) / block_rows) block_cols_ = mt.ceil(len(bmap) / block_rows)
block_rows_ = block_rows
elif block_cols is not None: elif block_cols is not None:
block_rows = mt.ceil(len(bmap) / block_cols) block_cols_ = block_cols
block_rows_ = mt.ceil(len(bmap) / block_cols)
else: else:
# divide by 2 until we hit our target ratio, this works # divide by 2 until we hit our target ratio, this works
# well for things that are often powers-of-two # well for things that are often powers-of-two
block_cols = 1 block_cols_ = 1
block_rows = len(bmap) block_rows_ = len(bmap)
while (abs(((canvas.width/(block_cols * 2)) while (abs(((canvas.width/(block_cols_ * 2))
/ (canvas.height/mt.ceil(block_rows / 2))) / (canvas.height/mt.ceil(block_rows_ / 2)))
- block_ratio) - block_ratio)
< abs(((canvas.width/block_cols) < abs(((canvas.width/block_cols_)
/ (canvas.height/block_rows))) / (canvas.height/block_rows_)))
- block_ratio): - block_ratio):
block_cols *= 2 block_cols_ *= 2
block_rows = mt.ceil(block_rows / 2) block_rows_ = mt.ceil(block_rows_ / 2)
block_width = canvas.width / block_cols block_width_ = canvas.width / block_cols_
block_height = canvas.height / block_rows block_height_ = canvas.height / block_rows_
# assign block locations based on block_rows/block_cols and the # assign block locations based on block_rows_/block_cols_ and
# requested space filling curve # the requested space filling curve
for (x, y), b in zip( for (x, y), b in zip(
curve(block_cols, block_rows), curve(block_cols_, block_rows_),
sorted(bmap.values())): sorted(bmap.values())):
b.x = x * block_width b.x = x * block_width_
b.y = y * block_height b.y = y * block_height_
b.width = block_width b.width = block_width_
b.height = block_height b.height = block_height_
# apply top padding # apply top padding
if x == 0: if x == 0:
@@ -4603,25 +4609,25 @@ def main_(f, disk, mroots=None, *,
# skip blocks with no usage # skip blocks with no usage
if not b.usage: if not b.usage:
continue continue
block__ = b.block - min__ block__ = b.block - global_block
usage__ = range( usage__ = range(
mt.floor(((block__*block_size + b.usage.start) mt.floor(((block__*block_size_ + b.usage.start)
/ (block_size * len(bmap))) / (block_size_ * len(bmap)))
* len(curve__)), * len(global_curve)),
mt.ceil(((block__*block_size + b.usage.stop) mt.ceil(((block__*block_size_ + b.usage.stop)
/ (block_size * len(bmap))) / (block_size_ * len(bmap)))
* len(curve__))) * len(global_curve)))
else: else:
block__ = b.block - min__ block__ = b.block - global_block
usage__ = range( usage__ = range(
mt.floor((block__/len(bmap)) * len(curve__)), mt.floor((block__/len(bmap)) * len(global_curve)),
mt.ceil((block__/len(bmap)) * len(curve__))) mt.ceil((block__/len(bmap)) * len(global_curve)))
# map to global curve # map to global curve
for i in usage__: for i in usage__:
if i >= len(curve__): if i >= len(global_curve):
continue continue
x__, y__ = curve__[i] x__, y__ = global_curve[i]
# flip y # flip y
y__ = canvas.height - (y__+1) y__ = canvas.height - (y__+1)
@@ -4647,9 +4653,9 @@ def main_(f, disk, mroots=None, *,
continue continue
# scale from bytes -> pixels # scale from bytes -> pixels
usage__ = range( usage__ = range(
mt.floor((b.usage.start/block_size) mt.floor((b.usage.start/block_size_)
* (width__*height__)), * (width__*height__)),
mt.ceil((b.usage.stop/block_size) mt.ceil((b.usage.stop/block_size_)
* (width__*height__))) * (width__*height__)))
# map to in-block curve # map to in-block curve
for i, (dx, dy) in enumerate(curve(width__, height__)): for i, (dx, dy) in enumerate(curve(width__, height__)):
@@ -4736,6 +4742,7 @@ def main_(f, disk, mroots=None, *,
def main(disk, mroots=None, *, def main(disk, mroots=None, *,
height=None, height=None,
keep_open=False, keep_open=False,
lines=None,
head=False, head=False,
cat=False, cat=False,
sleep=False, sleep=False,
@@ -4743,23 +4750,44 @@ def main(disk, mroots=None, *,
# keep-open? # keep-open?
if keep_open: if keep_open:
try: try:
# keep track of history if lines specified
if lines is not None:
ring = RingIO(lines+1
if not args.get('no_header') and lines > 0
else lines)
while True: while True:
# register inotify before running the command, this avoids # register inotify before running the command, this avoids
# modification race conditions # modification race conditions
if Inotify: if Inotify:
inotify = Inotify([disk]) inotify = Inotify([disk])
# TODO sync these comments
# cat? write directly to stdout
if cat: if cat:
main_(sys.stdout, disk, mroots, main_(sys.stdout, disk, mroots,
# make space for shell prompt # make space for shell prompt
height=height if height is not False else -1, height=height if height is not False else -1,
**args) **args)
# not cat? write to a bounded ring
else: else:
ring = RingIO(head=head) ring_ = RingIO(head=head)
main_(ring, disk, mroots, main_(ring_, disk, mroots,
height=height if height is not False else 0, height=height if height is not False else 0,
**args) **args)
ring.draw() # no history? draw immediately
if lines is None:
ring_.draw()
# history? merge with previous lines
else:
# write header separately?
if not args.get('no_header'):
if not ring.lines:
ring.lines.append('')
ring.lines.extend(it.islice(ring_.lines, 1, None))
ring.lines[0] = ring_.lines[0]
else:
ring.lines.extend(ring_.lines)
ring.draw()
# try to inotifywait # try to inotifywait
if Inotify: if Inotify:
@@ -4952,7 +4980,7 @@ if __name__ == "__main__":
'--title-littlefs', '--title-littlefs',
action='store_true', action='store_true',
help="Use the littlefs mount string as the title.") help="Use the littlefs mount string as the title.")
# TODO drop padding, no one is ever going to use this # TODO drop padding in ascii scripts, no one is ever going to use this
parser.add_argument( parser.add_argument(
'--padding', '--padding',
type=float, type=float,
@@ -4965,6 +4993,14 @@ if __name__ == "__main__":
'-k', '--keep-open', '-k', '--keep-open',
action='store_true', action='store_true',
help="Continue to open and redraw the CSV files in a loop.") help="Continue to open and redraw the CSV files in a loop.")
# TODO drop this?
parser.add_argument(
'-n', '--lines',
nargs='?',
type=lambda x: int(x, 0),
const=0,
help="Show this many lines of history. 0 uses the terminal "
"height. Defaults to 1.")
parser.add_argument( parser.add_argument(
'-^', '--head', '-^', '--head',
action='store_true', action='store_true',

65
scripts/dbgbmapd3.py
View File

@@ -4721,23 +4721,25 @@ def main(disk, output, mroots=None, *,
corrupted = not bool(lfs) corrupted = not bool(lfs)
# if we can't figure out the block_count, guess # if we can't figure out the block_count, guess
block_size_ = block_size
block_count_ = block_count
if block_count is None: if block_count is None:
if lfs.config.geometry is not None: if lfs.config.geometry is not None:
block_count = lfs.config.geometry.block_count block_count_ = lfs.config.geometry.block_count
else: else:
f.seek(0, os.SEEK_END) f.seek(0, os.SEEK_END)
block_count = mt.ceil(f.tell() / block_size) block_count_ = mt.ceil(f_.tell() / block_size)
# flatten blocks, default to all blocks # flatten blocks, default to all blocks
blocks = list( blocks_ = list(
range(blocks.start or 0, blocks.stop or block_count) range(blocks.start or 0, blocks.stop or block_count_)
if isinstance(blocks, slice) if isinstance(blocks, slice)
else range(blocks, blocks+1) else range(blocks, blocks+1)
if blocks if blocks
else range(block_count)) else range(block_count_))
# traverse the filesystem and create a block map # traverse the filesystem and create a block map
bmap = {b: BmapBlock(b, 'unused') for b in blocks} bmap = {b: BmapBlock(b, 'unused') for b in blocks_}
for child, path in lfs.traverse( for child, path in lfs.traverse(
mtree_only=mtree_only, mtree_only=mtree_only,
path=True): path=True):
@@ -4779,21 +4781,21 @@ def main(disk, output, mroots=None, *,
else: else:
bmap[b] = BmapBlock(b, 'conflict', bmap[b] = BmapBlock(b, 'conflict',
[bmap[b].value, child], [bmap[b].value, child],
range(block_size)) range(block_size_))
corrupted = True corrupted = True
# corrupt metadata? # corrupt metadata?
elif (not no_ckmeta elif (not no_ckmeta
and isinstance(child, (Mdir, Rbyd)) and isinstance(child, (Mdir, Rbyd))
and not child): and not child):
bmap[b] = BmapBlock(b, 'corrupt', child, range(block_size)) bmap[b] = BmapBlock(b, 'corrupt', child, range(block_size_))
corrupted = True corrupted = True
# corrupt data? # corrupt data?
elif (not no_ckdata elif (not no_ckdata
and isinstance(child, Bptr) and isinstance(child, Bptr)
and not child): and not child):
bmap[b] = BmapBlock(b, 'corrupt', child, range(block_size)) bmap[b] = BmapBlock(b, 'corrupt', child, range(block_size_))
corrupted = True corrupted = True
# normal block # normal block
@@ -4838,41 +4840,44 @@ def main(disk, output, mroots=None, *,
y__ += mt.ceil(FONT_SIZE * 1.3) y__ += mt.ceil(FONT_SIZE * 1.3)
height__ -= min(mt.ceil(FONT_SIZE * 1.3), height__) height__ -= min(mt.ceil(FONT_SIZE * 1.3), height__)
# figure out block_cols/block_rows # figure out block_cols_/block_rows_
if block_cols is not None and block_rows is not None: if block_cols is not None and block_rows is not None:
pass block_cols_ = block_cols
block_rows_ = block_rows
elif block_rows is not None: elif block_rows is not None:
block_cols = mt.ceil(len(bmap) / block_rows) block_cols_ = mt.ceil(len(bmap) / block_rows)
block_rows_ = block_rows
elif block_cols is not None: elif block_cols is not None:
block_rows = mt.ceil(len(bmap) / block_cols) block_cols_ = block_cols
block_rows_ = mt.ceil(len(bmap) / block_cols)
else: else:
# divide by 2 until we hit our target ratio, this works # divide by 2 until we hit our target ratio, this works
# well for things that are often powers-of-two # well for things that are often powers-of-two
block_cols = 1 block_cols_ = 1
block_rows = len(bmap) block_rows_ = len(bmap)
while (abs(((width__/(block_cols * 2)) while (abs(((width__/(block_cols_ * 2))
/ (height__/mt.ceil(block_rows / 2))) / (height__/mt.ceil(block_rows_ / 2)))
- block_ratio) - block_ratio)
< abs(((width__/block_cols) < abs(((width__/block_cols_)
/ (height__/block_rows))) / (height__/block_rows_)))
- block_ratio): - block_ratio):
block_cols *= 2 block_cols_ *= 2
block_rows = mt.ceil(block_rows / 2) block_rows_ = mt.ceil(block_rows_ / 2)
block_width = width__ / block_cols block_width_ = width__ / block_cols_
block_height = height__ / block_rows block_height_ = height__ / block_rows_
# assign block locations based on block_rows/block_cols and the # assign block locations based on block_rows_/block_cols_ and
# requested space filling curve # the requested space filling curve
for (x, y), b in zip( for (x, y), b in zip(
(hilbert_curve if hilbert (hilbert_curve if hilbert
else lebesgue_curve if lebesgue else lebesgue_curve if lebesgue
else naive_curve)(block_cols, block_rows), else naive_curve)(block_cols_, block_rows_),
sorted(bmap.values())): sorted(bmap.values())):
b.x = x__ + (x * block_width) b.x = x__ + (x * block_width_)
b.y = y__ + (y * block_height) b.y = y__ + (y * block_height_)
b.width = block_width b.width = block_width_
b.height = block_height b.height = block_height_
# apply top padding # apply top padding
if x == 0: if x == 0:

2
scripts/plot.py
View File

@@ -1855,7 +1855,7 @@ if __name__ == "__main__":
"specific group where a group is the comma-separated " "specific group where a group is the comma-separated "
"'by' fields. Accepts %% modifiers.") "'by' fields. Accepts %% modifiers.")
parser.add_argument( parser.add_argument(
'-_', '--add-line-char', '--line-chars', '-,', '--add-line-char', '--line-chars',
dest='line_chars', dest='line_chars',
action='append', action='append',
type=lambda x: ( type=lambda x: (

73
scripts/tailpipe.py
View File

@@ -112,17 +112,48 @@ class RingIO:
def main(path='-', *, def main(path='-', *,
lines=5, lines=5,
cat=False, cat=False,
coalesce=None,
sleep=None, sleep=None,
keep_open=False): keep_open=False):
lock = th.Lock()
event = th.Event()
# TODO adopt f -> ring name in all scripts?
def main_(ring):
try:
while True:
with openio(path) as f:
count = 0
for line in f:
with lock:
ring.write(line)
count += 1
# wait for coalesce number of lines
if count >= (coalesce or 1):
event.set()
count = 0
if not keep_open:
break
# don't just flood open calls
time.sleep(sleep or 2)
except FileNotFoundError as e:
print("error: file not found %r" % path,
file=sys.stderr)
sys.exit(-1)
except KeyboardInterrupt:
pass
# cat? let main_ write directly to stdout
if cat: if cat:
ring = sys.stdout main_(sys.stdout)
# not cat? print in a background thread
else: else:
ring = RingIO(lines) ring = RingIO(lines)
# if sleep print in background thread to avoid getting stuck in a read call
event = th.Event()
lock = th.Lock()
if not cat:
done = False done = False
def background(): def background():
while not done: while not done:
@@ -134,28 +165,13 @@ def main(path='-', *,
time.sleep(sleep or 0.01) time.sleep(sleep or 0.01)
th.Thread(target=background, daemon=True).start() th.Thread(target=background, daemon=True).start()
try: main_(ring)
while True:
with openio(path) as f:
for line in f:
with lock:
ring.write(line)
event.set()
if not keep_open:
break
# don't just flood open calls
time.sleep(sleep or 2)
except FileNotFoundError as e:
print("error: file not found %r" % path,
file=sys.stderr)
sys.exit(-1)
except KeyboardInterrupt:
pass
if not cat:
done = True done = True
lock.acquire() # avoids https://bugs.python.org/issue42717 lock.acquire() # avoids https://bugs.python.org/issue42717
# give ourselves one last draw, helps if background is
# never triggered
ring.draw()
sys.stdout.write('\n') sys.stdout.write('\n')
@@ -181,10 +197,15 @@ if __name__ == "__main__":
'-c', '--cat', '-c', '--cat',
action='store_true', action='store_true',
help="Pipe directly to stdout.") help="Pipe directly to stdout.")
parser.add_argument(
'-S', '--coalesce',
type=lambda x: int(x, 0),
help="Number of lines to coalesce together.")
parser.add_argument( parser.add_argument(
'-s', '--sleep', '-s', '--sleep',
type=float, type=float,
help="Seconds to sleep between reads.") help="Seconds to sleep between draws, coalescing lines in "
"between.")
parser.add_argument( parser.add_argument(
'-k', '--keep-open', '-k', '--keep-open',
action='store_true', action='store_true',

3129
scripts/tracebd.py
View File

File diff suppressed because it is too large Load Diff