How to deal with Number precision in Actionscript?

Question

I have BigDecimal objects serialized with BlazeDS to Actionscript. Once they hit Actionscript as Number objects, they have values like:

140475.32 turns into 140475.31999999999998

How do I deal with this? The problem is that if I use a NumberFormatter with precision of 2, then the value is truncated to 140475.31. Any ideas?

Answer 1

This is my generic solution for the problem (I have blogged about this here):

var toFixed:Function = function(number:Number, factor:int) {
  return Math.round(number * factor)/factor;
}

For example:

trace(toFixed(0.12345678, 10)); //0.1

• Multiply 0.12345678 by 10; that gives us 1.2345678.
• When we round 1.2345678, we get 1.0,
• and finally, 1.0 divided by 10 equals 0.1.

Another example:

trace(toFixed(1.7302394309234435, 10000)); //1.7302

• Multiply 1.7302394309234435 by 10000; that gives us 17302.394309234435.
• When we round 17302.394309234435 we get 17302,
• and finally, 17302 divided by 10000 equals 1.7302.

---

Edit

Based on the anonymous answer below, there is a nice simplification for the parameter on the method that makes the precision much more intuitive. e.g:

var setPrecision:Function = function(number:Number, precision:int) {
 precision = Math.pow(10, precision);
 return Math.round(number * precision)/precision;
}

var number:Number = 10.98813311;
trace(setPrecision(number,1)); //Result is 10.9
trace(setPrecision(number,2)); //Result is 10.98
trace(setPrecision(number,3)); //Result is 10.988 and so on

N.B. I added this here just in case anyone sees this as the answer and doesn't scroll down...