scripts: Added dbgbmapd3.py for bmap -> svg rendering

Like codemapd3.py this include an interactive UI for viewing the
underlying filesystem graph, including:

- mode-tree - Shows all reachable blocks from a given block
- mode-branches - Shows immediate children of a given block
- mode-references - Shows parents of a given block
- mode-redund - Shows sibling blocks in redund groups (This is
  currently just mdir pairs, but the plan is to add more)

This is _not_ a full filesystem explorer, so we don't embed all block
data/metadata in the svg. That's probably a project for another time.
However we do include interesting bits such as trunk addresses,
checksums, etc.

An example:

  # create an filesystem image
  $ make test-runner -j
  $ ./scripts/test.py -B test_files_many -a -ddisk -O- \
          -DBLOCK_SIZE=1024 \
          -DCHUNK=10 \
          -DSIZE=2050 \
          -DN=128 \
          -DBLOCK_RECYCLES=1
  ... snip ...
  done: 2/2 passed, 0/2 failed, 164pls!, in 0.16s

  # generate bmap svg
  $ ./scripts/dbgbmapd3.py disk -b1024 -otest.svg \
          -W1400 -H750 -Z --dark
  updated test.svg, littlefs v0.0 1024x1024 0x{26e,26f}.d8 w64.128, cksu
  m 41ea791e

And open test.svg in a browser of your choice.

Here's what the current colors mean:

- yellow => mdirs
- blue   => btree nodes
- green  => data blocks
- red    => corrupt/conflict issue
- gray   => unused blocks

But like codemapd3.py the output is decently customizable. See -h/--help
for more info.

And, just like codemapd3.py, this is based on ideas from d3 and
brendangregg's flamegraphs:

- d3 - https://d3js.org
- brendangregg's flamegraphs - https://github.com/brendangregg/FlameGraph

Note we don't actually use d3... the name might be a bit confusing...

---

One interesting change from the previous dbgbmap.py is the addition of
"corrupt" (bad checksum) and "conflict" (multiple parents) blocks, which
can help find bugs.

You may find the "conflict" block reporting a bit strange. Yes it's
useful for finding block allocation failures, but won't naturally formed
dags in file btrees also be reported as "conflicts"?

Yes, but the long-term plan is to move away from dags and make littlefs
a pure tree (for block allocator and error correction reasons). This
hasn't been implemented yet, so for now dags will result in false
positives.

---

Implementation wise, this script was pretty straightforward given prior
dbglfs.py and codemapd3.py work.

However there was an interesting case of https://xkcd.com/1425:

- Traverse the filesystem and build a graph - easy
- Tile a rectangle with n nice looking rectangles - uhhh

I toyed around with an analytical approach (something like block width =
sqrt(canvas_width*canvas_height/n) * block_aspect_ratio), but ended up
settling on an algorithm that divides the number of columns by 2 until
we hit our target aspect ratio.

This algorithm seems to work quite well, runs in only O(log n), and
perfectly tiles the grid for powers-of-two. Honestly the result is
better than I was expecting.
This commit is contained in:
Christopher Haster
2025-04-05 02:16:49 -05:00
parent 27370dec66
commit 5f06558cbe
20 changed files with 6303 additions and 200 deletions

View File

@@ -298,37 +298,54 @@ def fold(results, by=None, x=None, y=None, defines=[]):
# a representation of optionally key-mapped attrs
class Attr:
def __init__(self, attrs, *,
defaults=None):
# include defaults?
if (defaults is not None
and not any(
not isinstance(attr, tuple)
or attr[0] in {None, (), ('*',)}
for attr in (attrs or []))):
attrs = list(defaults) + (attrs or [])
def __init__(self, attrs, defaults=None):
if attrs is None:
attrs = []
if isinstance(attrs, dict):
attrs = attrs.items()
# normalize
self.attrs = []
self.keyed = co.OrderedDict()
for attr in (attrs or []):
if not isinstance(attr, tuple):
for attr in attrs:
if (not isinstance(attr, tuple)
or attr[0] in {None, (), (None,), ('*',)}):
attr = ((), attr)
elif attr[0] in {None, (), ('*',)}:
attr = ((), attr[1])
if not isinstance(attr[0], tuple):
attr = ((attr[0],), attr[1])
self.attrs.append(attr)
if attr[0] not in self.keyed:
self.keyed[attr[0]] = []
self.keyed[attr[0]].append(attr[1])
# create attrs object for defaults
if isinstance(defaults, Attr):
self.defaults = defaults
elif defaults is not None:
self.defaults = Attr(defaults)
else:
self.defaults = None
def __repr__(self):
return 'Attr(%r)' % [
(','.join(attr[0]), attr[1])
for attr in self.attrs]
if self.defaults is None:
return 'Attr(%r)' % (
[(','.join(attr[0]), attr[1])
for attr in self.attrs])
else:
return 'Attr(%r, %r)' % (
[(','.join(attr[0]), attr[1])
for attr in self.attrs],
[(','.join(attr[0]), attr[1])
for attr in self.defaults.attrs])
def __iter__(self):
return it.cycle(self.keyed[()])
if () in self.keyed:
return it.cycle(self.keyed[()])
elif self.defaults is not None:
return iter(self.defaults)
else:
return iter(())
def __bool__(self):
return bool(self.attrs)
@@ -342,6 +359,9 @@ class Attr:
else:
i, key = key, ()
if not isinstance(key, tuple):
key = (key,)
# try to lookup by key
best = None
for ks, vs in self.keyed.items():
@@ -361,6 +381,10 @@ class Attr:
# cycle based on index
return best[1][i % len(best[1])]
# fallback to defaults?
if self.defaults is not None:
return self.defaults[i, key]
return None
def __contains__(self, key):
@@ -368,11 +392,8 @@ class Attr:
# a key function for sorting by key order
def key(self, key):
# allow key to be a tuple to make sorting dicts easier
if (isinstance(key, tuple)
and len(key) >= 1
and isinstance(key[0], tuple)):
key = key[0]
if not isinstance(key, tuple):
key = (key,)
best = None
for i, ks in enumerate(self.keyed.keys()):
@@ -391,6 +412,10 @@ class Attr:
if best is not None:
return best[1]
# fallback to defaults?
if self.defaults is not None:
return len(self.keyed) + self.defaults.key(key)
return len(self.keyed)
# parse %-escaped strings
@@ -882,7 +907,7 @@ def main(csv_paths, output, *,
# order by labels
datasets_ = co.OrderedDict(sorted(
datasets_.items(),
key=labels_.key))
key=lambda kv: labels_.key(kv[0])))
# and merge dataattrs
mergedattrs_ = {k: v
@@ -969,7 +994,7 @@ def main(csv_paths, output, *,
# order by labels
subdatasets = co.OrderedDict(sorted(
subdatasets.items(),
key=labels_.key))
key=lambda kv: labels_.key(kv[0])))
# filter by subplot x/y
subdatasets = co.OrderedDict([(name, dataset)