MATLAB division... should 29/128 return 0?

jakev picture jakev · Sep 11, 2010 · Viewed 19.7k times · Source

I really don't think this is a precision problem, the answer should be about 0.226. Here's the exact code:

val = I(i,j)
bucketSize    
pos = val / bucketSize

I is just a matrix I'm taking values from. Here is the output from MATLAB:

val =

   29

bucketSize =

   128

pos =

   0

What am I missing?

Answer

gnovice picture gnovice · Sep 11, 2010

My guess would be that your matrix I is pixel data loaded from an image file, which will have values that are typically unsigned 8-bit integers. As already mentioned, converting both integer values to a double precision value will ensure that MATLAB performs floating point division instead of integer division (which will round off the result).

Converting one value to double precision is insufficient:

For all binary operations in which one operand is an array of integer data type (except 64-bit integers) and the other is a scalar double, MATLAB computes the operation using elementwise double-precision arithmetic, and then converts the result back to the original integer data type.

If you'd like to find out more about the different numeric data types in MATLAB, you can check out this documentation.