forked from Imagelibrary/littlefs
scripts: Fixed rounding-towards-zero issue in si/si2 prefixes
This should be floor (rounds towards -inf), not int (rounds towards zero), otherwise sub-integer results get funky: - floor si(0.00001) => 10u - int si(0.00001) => 0.01m - floor si(0.000001) => 1u - int si(0.000001) => m (???)
This commit is contained in:
@@ -101,7 +101,7 @@ def si(x, w=4):
|
||||
#
|
||||
# note we adjust this so that 100K = .1M, which has more info
|
||||
# per character
|
||||
p = 3*int(mt.log(abs(x)*10, 10**3))
|
||||
p = 3*mt.floor(mt.log(abs(x)*10, 10**3))
|
||||
p = min(18, max(-18, p))
|
||||
# format with enough digits
|
||||
s = '%.*f' % (w, abs(x) / (10.0**p))
|
||||
@@ -120,7 +120,7 @@ def si2(x, w=5):
|
||||
#
|
||||
# note we adjust this so that 128Ki = .1Mi, which has more info
|
||||
# per character
|
||||
p = 10*int(mt.log(abs(x)*10, 2**10))
|
||||
p = 10*mt.floor(mt.log(abs(x)*10, 2**10))
|
||||
p = min(30, max(-30, p))
|
||||
# format with enough digits
|
||||
s = '%.*f' % (w, abs(x) / (2.0**p))
|
||||
|
||||
Reference in New Issue
Block a user