Does anyone have any recommendations for resources or methods on how to create a bot in java which can play a flash game?
I am thinking of using the Robot class to watch the screen and make actions, but I need ways of finding images in images, etc. I am sure this has been done before but google searches return alot of nonsense..
Answer
I have just built a casual card playing bot (not poker) in about 3 days and it was overwhelmingly easy. The method below will work for most button based games. Of course for 3D games and fast moving 2D graphics you will need a different approach.
It's strange that there is so little useful information out there. All the articles on writing bots will link to Windows stuff with DLLs, hooks, memory snooping and all kinds of other tricks. And of course, you dont need any of this nonsense. All you need is Java and awt.Robot.
Emitting clicks is straightforward enough, the game logic is something you have to figure out on your own, so I will focus on getting the screen Input.
You start by getting a fullscreen capture. Then you find a distinct image that is present in every frame, and use that image as an anchor to calculate the offset of the top left corner of the game interface. The anchor image could be anything, even a two or three pixel wide border between the game and the browser background, just make sure to make it long enough. Then you simply search the screen capture to find where this image is located, this is a very simple algorithm that compares images one pixel at the time and changes the offset until the match is found. Once you have the UI offset, you find the relative offset to all the buttons on the screen, and take a little snapshot for each button you want to interact with. You use these offsets to detect active buttons or other indicators. As a bonus you can also use these offsets when clicking on buttons. You should make a collection of png (not jpg) images of all the buttons you want to detect. Then you run simple image comparison algorithms against the content of the screen capture.
Flash does vector font rendering and does not produce pixel-perfect consistency between different instances of the game. Also image scaling will not be consistent. Here are the ways to fight this:
- Filtering. You can utilize many filtering methods, the simplest one is a threshold filter. Simply make all the pixels 100% black if the intensity is less that your threshold, or 100% white otherwise. Then run pixel-by-pixel comparison on the filtered images and find the number of distinct pixels. Assume a match for low values of distinct pixels. You may also have to shift the images by + or - 1 pixel in each direction, for a total of 3*3 = 9 comparisons. Just choose the lowest distinct pixel value as a result.
- Average color. If you are simply looking if an indicator (for example a green glowing circle) is present on the screen or not, just find the average color in the rectangle where the indicator is and see if it's close enough to the average color of the stored image.
Font recognition can be tricky. If you can separate the letters (usually with monochrome fonts), then you can simply use the Filtering method above to recognize each letter individually.
If you hit a wall, you can try a variety of more complicated filtering methods (blur + intensity threshold is a good combo).
The final tips are:
- Keep your image recognition and logic completely separate.
- Invest in writing good debug tools that dump the information on the image detection process so that you can tweak the filters and threshold values until they work. This debug-dump should include the filtered images being compared, so you get a good feel on how the filters work, what went wrong, and how to fix it. You will not succeed without these tools, so if you are planning to skip them, then don't bother at all.
- Have a working prototype early. Even if it only detects one button.
- Do not optimize any image algorithms until you have a working bot.