[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[sc-users] Problems with "real" numbers



Hi list,

Among us, sure some know that  the "real" result of some operations can be slightly different from what's posted (because of the "translation" between hex. and dec.)

Example :

(1.68 / 3).dump

Float 0.5599999999999999   3FE1EB85 1EB851EB
0.56

0.55999.. is the "real" number
0.56 is what appears when you just enter 1.68 / 3 (without dump) 

What I call the "real" number is e.g. the number really used in booleans :
Still with : x = 1.68 / 3
(x == 0.56) -> false ! Because it's the "real" version that is taken in account here.

Even if in this case .round(x) would work, it's not always the case, e.g. :
(0.01 * 35).round(0.01).dump (hoping to get a rounded "real" number because the "real" number corresponding to 0.01 * 35 is not 0.35 but 0.35000000...

Float 0.35000000000000003   3FD66666 66666667
0.35

Where even the "rounded" value is not "really" rounded ! I didn't figure it out, but of course it's logical, after all "round" is an operation as the others with hexadecimal numbers !

So I found a way to use 0.56 instead of its long version, that is to convert it to a string and re-convert it to a float this way :
(x.asString.asFloat == 0.56) -> true

My question is simply : how would any of you do to get the same result (true) if not using asString.asFloat, without using "round" or "trunc" etc. which are uncertain methods ? How do you manage with this kind of problem ?

Cheers (and best wishes to all !)

T