Sqlalchemy: Latitude and Longitude Float Precision?

Question

I'm using Sqlalchemy to define my tables and such and here is some code I came up with:

locations = Table('locations', Base.metadata,
Column("lat", Float(Precision=64), primary_key=True),
Column("lng", Float(Precision=64), primary_key=True),
)

I read somewhere that latitude and longitude require better precision than floats, usually double precision. So I set the precision manually to 64, is this sufficient? Overkill? Would this even help for my situation?

Answer 1

Nobody else here provided concrete numbers with proof for the worst-case accuracy of a floating point lat/long. I needed to know this for something I was working on, so here is my analysis in case it helps someone else.

A single-precision floating point offers 24-bits of precision in the significand (the binary exponential notation of a number). As the whole part of the number gets larger, the number of bits after the decimal goes down. Therefore, the worst-case accuracy for a latitude or longitude is when the magnitude is as far away from 0 as is possible. Assuming you bound your latitudes to [-90, 90] and longitudes from (-180, 180], the worst-case will be at the equator for longitude 180.

In binary, 180 requires 8-bits of the 24-bits available, leaving 16 bits after the decimal point. Therefore, the distance between consecutively representable values at this longitude would be 2^-16 deg (approximately 1.526E-5). Multiplying that number (in radians) by the WGS-84 radius of the Earth at the equator (6,378,137 m) yields a worst-case precision of:

2^-16 deg * 6,378,137 m * PI rad / 180 deg = 1.6986 m (5.5728 ft).

The same analysis against lat/longs stored in radians yields the following:

2^-22 rad * 6,378,137 m = 1.5207 m (4.9891 ft)

And finally, if you normalize the latitudes to the range [-1, 1] and the longitudes to the range (-1, 1], then you can achieve the following worst-case precision:

2^-24 * PI rad * 6,378,137 m = 1.1943 m (3.9184 ft)

So storing lat/long in radians buys you around 7 inches of additional accuracy, and storing them in normalized form buys you around 1'8" of additional accuracy, both in the worst-case scenario.

If, when converting between double-precision and single-precision you rounded (instead of truncating), the single-precision value will be within half of the distance between two consecutive values computed above.