Larger/Smaller Bug?

Replies to This Discussion

Permalink Reply by Danny Boyes on April 12, 2014 at 1:00pm

What you are seeing here is a floating point error.

The Number you have is probably 3.200000000001

When you put this into a [Panel] it converts it to a String for its significant figures only.

This is the age old question: What is 0.1 + 0.2?

Answer is (according to your computer) 0.30000000000000004.

Why?

Because 0.1 in binary is 0.000110011001100110011001100110011001100110011...

And 0.2 is 0.0011001100110011001100110011011001100110011001100110011...

At some point the recurring pattern is ignored and a Floating Point Error is introduced.

You can check what the number really is by using an [expression] component with Format("{0:R}",x) where x is your input

Permalink Reply by doraET on April 12, 2014 at 2:24pm

Thank you!  you are right.

but how could I round off to the specific decimal place?

Permalink Reply by Danny Boyes on April 12, 2014 at 5:11pm

Using the same expression except with {0:0.000} for 3 decimal places or you could do {0:0.0##} which will do 1 dp if there are no other significant figures and 3dp if there is.

But be warned this is still a string and not a number.

If you want to have it work without converting to text then i would use an <Expression> i/o modifier set to x + 0.0000001 on the comparison input.