MagnitudeOS/littlefs
1
0
Fork 0
forked from Imagelibrary/littlefs
Code Pull Requests Activity

Adopted odd-parity-zero rbyd perturb scheme

Browse Source 
I've been scratching my head over our rbyd perturb scheme. It's gotten
rather clunky with needing to xor valid bits and whatnot.

But it's tricky with needing erased-state to be included in parity bits,
while at the same time excluded from our canonical checksum. If only
there was some way to flip the checksums parity without changing its
value...

Enter the crc32c odd-parity zero: 0xfca42daf!

This bends the definition of zero a bit, but it is one of two numbers in
our crc32c-ring with a very interesting property:

  crc32c(m) == crc32c(m xor 0xfca42daf) xor 0xfca42daf  // odd-p zero
  crc32c(m) == crc32c(m xor 0x00000000) xor 0x00000000  // even-p zero

Recall that crc32c's polynomial, 0x11edc6f41, is composed of two
polynomials: 0x3, the parity polynomial, and 0xf5b4253f, a maximally
sized irreducible polynomial. Because our polynomial breaks down into
two smaller polynomials, our crc32c space turns out to not be a field,
but rather a ring containing two smaller sub-fields. Because these
sub-fields are defined by their polynomials, one is the 31-bit crc
defined by the polynomial 0xf5b4253f, while the other is the current
parity.

We can move in the parity sub-field without changing our position in the
31-bit crc sub-field by xoring with a number that is one in the parity
sub-field, but zero in the 31-bit crc sub-field.

This number happens to be 0xf5b4253f (0xfca42daf bit-reversed)!

(crcs being bit-reversed will never not be annoying)

So long story short, xoring any crc32c with 0xfca42daf will change its
parity but not its value.

---

An that's basically our new perturb scheme. If we need to perturb, xor
with 0xfca42daf to change the parity, and after calculating/validating
the checksum, xor with 0xfca42daf to get our canonical checksum.

Isn't that neat!

There was one small hiccup: At first I assumed you could continue
including the valid bits in the checksum, which would have been nice for
bulk checksumming. But this doesn't work because while valid bits cancel
out so the parity doesn't change, changing valid bits _does_ change the
underlying 31-bit crc, poisoning our checksum and making everything a
mess.

So we still need to mask out valid bits, which is a bit annoying.

But then I stumbled on the funny realization that by masking our valid
bits, we accidentally end up with a fully functional parity scheme.
Because valid bits _don't_ include the previous valid bit, we can figure
out the parity for not only the entire commit, but also each individual
tag:

  80 03 00 08 6c 69 74 74 6c 65 66 73 80
  ^'----------------.---------------' ^
  |                 |                 |
  v       +       parity      =       v'

Or more simply:

  80 03 00 08 6c 69 74 74 6c 65 66 73 80
  '----------------.----------------' ^
                   |                  |
                 parity       =       v'

Double neat!

Some other notes:

- By keeping the commit checksum perturbed, but not the canonical
  checksum, the perturb state is self-validating. We no longer need to
  explicitly check the previous-perturb-bit (q) to avoid the perturb
  hole we ran into previously.

  I'm still keeping the previous-perturb-bit (q) around, since it's
  useful for debugging. We still need to know the perturb state
  internally at all times in order to xor out the canonical checksum
  correctly anyways.

- Thanks to all of our perturb iterations, we now know how to remove the
  valid bits from the checksum easily:

    cksum ^= 0x00000080 & (tag >> 8)

  This makes the whole omitting-valid-bits thing less of a pain point.

- It wasn't actually worth it to perturb the checksum when building
  commits, vs manually flipping each valid bit, as this would have made
  our internal appendattr API really weird.

  At least the perturbed checksum made fetch a bit simpler.

Not sure exactly how to draw this with our perturb scheme diagrams,
maybe something like this?

  .---+---+---+---. \   \   \   \
  |v|    tag      | |   |   |   |
  +---+---+---+---+ |   |   |   |
  |     commit    | |   |   |   |
  |               | +-. |   |   |
  +---+---+---+---+ / | |   |   |
  |v|qp-------------->p>p-->p   .
  +---+---+---+---+   | .   .   .
  |     cksum     |   | .   .   .
  +---+---+---+---+   | .   .   .
  |    padding    |   | .   .   .
  |               |   | .   .   .
  +---+---+---+---+   | |   |   |
  |v------------------' |   |   |
  +---+---+---+---+     |   |   |
  |     commit    |     +-. |   +- rbyd
  |               |     | | |   |  cksum
  +---+---+---+---+     / | +-. /
  |v----------------------' | |
  +-------+---+---+         / |
  |     cksum ----------------'
  +---+---+---+---+
  |    padding    |
  |               |
  +---+---+---+---+
  |     erased    |
  |               |
  .               .
  .               .

---

Code changes were minimal, saving a tiny bit of code:

           code          stack
  before: 36368           2664
  after:  36352 (-0.0%)   2672 (+0.3%)

There was a stack bump in lfsr_bd_readtag, but as far as I can tell it's
just compiler noise? I poked around a bit but couldn't figure out why it
changed...
This commit is contained in:
Christopher Haster
2024-08-02 01:15:32 -05:00
parent dd339e3090
commit 1044c9d2b7
7 changed files with 103 additions and 125 deletions
Download Patch File Download Diff File Expand all files Collapse all files

16
scripts/dbglfs.py
View File

@@ -368,14 +368,11 @@ class Rbyd:
weight_ = 0
weight__ = 0
while j_ < len(data) and (not trunk or eoff <= trunk):
# perturb?
if perturb:
cksum__ ^= 0x00000080
# read next tag
v, tag, w, size, d = fromtag(data[j_:])
if v != parity(cksum__):
break
cksum__ ^= 0x00000080 if v else 0
cksum__ = crc32c(data[j_:j_+d], cksum__)
j_ += d
if not tag & TAG_ALT and j_ + size > len(data):
@@ -387,9 +384,6 @@ class Rbyd:
cksum__ = crc32c(data[j_:j_+size], cksum__)
# found a cksum?
else:
# check perturb bit
if perturb != bool(tag & TAG_Q):
break
# check cksum
cksum___ = fromle32(data[j_:j_+4])
if cksum__ != cksum___:
@@ -401,8 +395,8 @@ class Rbyd:
weight = weight_
# update perturb bit
perturb = tag & TAG_P
# revert to data cksum
cksum__ = cksum_
# revert to data cksum and perturb
cksum__ = cksum_ ^ (0xfca42daf if perturb else 0)
# evaluate trunks
if (tag & 0xf000) != TAG_CKSUM and (
@@ -417,8 +411,8 @@ class Rbyd:
# end of trunk?
if not tag & TAG_ALT:
# update canonical checksum
cksum_ = cksum__
# update canonical checksum, xoring out any perturb state
cksum_ = cksum__ ^ (0xfca42daf if perturb else 0)
# update trunk/weight unless we found a shrub or an
# explicit trunk (which may be a shrub) is requested
if not tag & TAG_SHRUB or trunk___ == trunk: