How can I find the largest circle that can fit inside a concave polygon?
A brute force algorithm is OK as long as it can handle polygons with ~50 vertices in real-time.
The key to solving this problem is first making an observation: the center of the largest circle that will fit inside an arbitrary polygon is the point that is:
Why? Because every point on the edge of a circle is equidistant from that center. By definition, the largest circle will have the largest radius and will touch the polygon on at least two points so if you find the point inside furthest from the polygon you've found the center of the circle.
This problem appears in geography and is solved iteratively to any arbitrary precision. Its called the Poles of Inaccessibility problem. See Poles of Inaccessibility: A Calculation Algorithm for the Remotest Places on Earth.
The basic algorithm works like this:
One note, How to test if a point is inside the polygon or not: The simplest solution to this part of the problem is to cast a ray to the right of the point. If it crosses an odd number of edges, it's within the polygon. If it's an even number, it's outside.
Also, as far as testing the distance to any edge there are two cases you need to consider:
(2) is easy. The distance to the edge is the minimum of the distances to the two vertices. For (1), the closest point on that edge will be the point that intersects the edge at a 90 degree angle starting from the point you're testing. See Distance of a Point to a Ray or Segment.