 |
 | Home | Contact Us | FAQ | Search & Site Map | Link to Us |
 | Sign In | Join | Other 45 Sites in Network |
Mac Forum / Country Specific / Australian Mac Group / May 2005calculator error?
When I use Calculator (v3.2.1) to do the following calculation: 9000 - 8387.7 - 557.3 I get the result: 54.999999999999 What gives?  SignaturePeter McCallum Mackay Qld AUSTRALIA > When I use Calculator (v3.2.1) to do the following calculation: > 9000 - 8387.7 - 557.3 > > I get the result: > 54.999999999999 Using v. 4.0 I get the same result. But changing the precision from 12 to 16 gives a more pleasing result. And so does changing the precision to 6. A bit odd. >> When I use Calculator (v3.2.1) to do the following calculation: >> 9000 - 8387.7 - 557.3 [quoted text clipped - 5 lines] > to 16 gives a more pleasing result. And so does changing the precision > to 6. A bit odd. I noticed that about a year ago when I first purchased OSX. (Reminded me of using Fortan in College.) Anyway, changing the precision to 10 worked for me. Thanks for the tip! GW > When I use Calculator (v3.2.1) to do the following calculation: > 9000 - 8387.7 - 557.3 [quoted text clipped - 3 lines] > > What gives? Most computers nowadays converts all floating point numbers into binary form, and then performs operations on those. Thus, round-off errors might occur. You can get better precision in Terminal, by searching for calculator programs: man -k calculator One such program is "bc". You can learn about it by typing "man bc". Then type: echo "9000 - 8387.7 - 557.3" | bc 55.0 Here, the answer is merely a round-off of a higher precision result. SignatureHans Aberg > When I use Calculator (v3.2.1) to do the following calculation: > 9000 - 8387.7 - 557.3 [quoted text clipped - 3 lines] > > What gives? http://docs.sun.com/source/806-3568/ncg_goldberg.html SignatureRandy Howard (2reply remove FOOBAR) "I don't really care about being right you know, I just care about success." --Steve Jobs |
Reply to this Message |
 Sign In
 Join
 My Latest Posts My Monitored Threads My Blog My Photo Gallery My Profile My Homepage Start New Thread Enable EMail Alerts Rate this Thread
|
©2008 Advenet LLC Privacy Policy - Terms of Use
This website includes both content owned or controlled by Advenet as well as content owned or controlled by third parties.