Format function returning unexpected number value

I've simplified down a problem that I have in a different model. Simplified, I created a series and then output via a format function. For some reason the format is changing the value of the input and I don't see any reason why it should. Is there an explanation for this or is it a bug?

Attachments:

Format output anomaly.gh, 9 KB

Replies to This Discussion

Permalink Reply by Pieter Segeren on June 14, 2015 at 12:31pm

Hi Norm, you could use {0:0.###} as formatting rule to fix that.
(Or use {0:0.00#} to have at least 2 digits, and 3 digits if the third digit is relevant (not 0).
To have all values use 3 digits you could also use {0:N3}

Permalink Reply by Pieter Segeren on June 14, 2015 at 2:42pm

Sorry, I meant decimals/decimal places instead of digits ...

Permalink Reply by Norm Sash on June 14, 2015 at 6:03pm

Hey Pieter... Thanks for the suggestions. It kinda works, but the specific usage that I have for this requires that the result not be rounded. It's important to know if the result is 3.46 or if it is 3.4599999999994.

It really seems to be an anomaly with the Format function... also interesting is that the last digit starts a repeating pattern as the number of results increases.

But your ideas got me to experimenting and I think that the format of:

{0:0.############} (12 #'s)

will work as it seems to give the correct results in the list.

If I use {0:0.#############} (13 #'s) the output fails again.

Permalink Reply by Pieter Segeren on June 15, 2015 at 2:32am

Well, the "correct results" in the original series are already rounded: Try {0:R} on the series output to see 'all' decimals (depending on your GH Display settings). I'm not sure how come the noise is introduced. I think it's a floating point precision issue. (Try a search for that).

But since your series increments in steps of 0.0010 I figured three (or four) decimals would suffice in you case (plus it kills the noise).

RSS