Extracting fields from forms with varying structures

Question

I am trying to extract certain fields from a balance sheet. For example, I would like to be able to tell that the value of 'Inventory' is 1,277,838 for the following balance sheet:

Currently, I am using Tesseract to convert images to text. However, this conversion results in a stream of text, so it is difficult to associate fields with their values (since these values are not always right next to the text of their corresponding fields).

After some searching, I read the Tesseract can use uzn files to read from zones of an image. However, the specific zones of the balance sheet values may shift from form to form, so I am interested in any solutions that can determine that 'Inventory' and 1,277,838 are on the same line. Ideally, I would like a grid structure output of text (so that I can tell the spatially that which chunks of text are in the same rows/columns).

Could anyone help explain how I can achieve this result?

Answer 1

I have been performing a similar task using Tesseract and Python (pytesseract library). I have been able to use Tesseract's .hocr output files (https://en.wikipedia.org/wiki/HOCR) to find the location of my search term (e.g 'Inventory') on the page and then rerun Tesseract on a small section of the page which gives it higher accuracy for that area. Here's the code I use to parse the HOCR output from Tesseract:

def parse_hocr(search_terms=None, hocr_file=None, regex=None):
    """Parse the hocr file and find a reasonable bounding box for each of the strings
    in search_terms.  Return a dictionary with values as the bounding box to be used for 
    extracting the appropriate text.

    inputs:
        search_terms = Tuple, A tuple of search terms to look for in the HOCR file.

    outputs:
        box_dict = Dictionary, A dictionary whose keys are the elements of search_terms and values
        are the bounding boxes where those terms are located in the document.
    """
    # Make sure the search terms provided are a tuple.
    if not isinstance(search_terms,tuple):
        raise ValueError('The search_terms parameter must be a tuple')

    # Make sure we got a HOCR file handle when called.
    if not hocr_file:
        raise ValueError('The parser must be provided with an HOCR file handle.')

    # Open the hocr file, read it into BeautifulSoup and extract all the ocr words.
    hocr = open(hocr_file,'r').read()
    soup = bs.BeautifulSoup(hocr,'html.parser')
    words = soup.find_all('span',class_='ocrx_word')

    result = dict()

    # Loop through all the words and look for our search terms.        
    for word in words:

        w = word.get_text().lower()

        for s in search_terms:

            # If the word is in our search terms, find the bounding box
            if len(w) > 1 and difflib.SequenceMatcher(None, s, w).ratio() > .5:
                bbox = word['title'].split(';')
                bbox = bbox[0].split(' ')
                bbox = tuple([int(x) for x in bbox[1:]])

                # Update the result dictionary or raise an error if the search term is in there twice.
                if s not in result.keys():
                    result.update({s:bbox})

            else:
                pass

    return result 

This allows me to search an HOCR file for the appropriate terms and return the bounding box of that particular word. I can then expand the bounding box slightly to run Tesseract on a very small subset of the page. This allows for MUCH greater accuracy than just OCRing the whole page. Obviously, some of this code is particular to my use, but it should give you a place to start.

This page is very helpful for finding the appropriate arguments to give to Tesseract. I found the Page Segmentation Modes to be VERY important for obtaining accurate results for small sections of an image.