I'm trying to extract each page of a PDF as a string:
import pyPdf
pages = []
pdf = pyPdf.PdfFileReader(file('g-reg-101.pdf', 'rb'))
for i in range(0, pdf.getNumPages()):
this_page = pdf.getPage(i).extractText() + "\n"
this_page = " ".join(this_page.replace(u"\xa0", " ").strip().split())
pages.append(this_page.encode("ascii", "xmlcharrefreplace"))
for page in pages:
print '*' * 80
print page
But this script ignore newline characters, leaving me with messy strings like information concerning an individual which, because of name, identifyingnumber, mark or description
(i.e, this should read identifying number
, not identifyingumber
).
Here's an example of the type of PDF I'm trying to parse.
I don't know much about PDF encoding, but I think you can solve your particular problem by modifying pdf.py
. In the PageObject.extractText
method, you see what's going on:
def extractText(self):
[...]
for operands,operator in content.operations:
if operator == "Tj":
_text = operands[0]
if isinstance(_text, TextStringObject):
text += _text
elif operator == "T*":
text += "\n"
elif operator == "'":
text += "\n"
_text = operands[0]
if isinstance(_text, TextStringObject):
text += operands[0]
elif operator == '"':
_text = operands[2]
if isinstance(_text, TextStringObject):
text += "\n"
text += _text
elif operator == "TJ":
for i in operands[0]:
if isinstance(i, TextStringObject):
text += i
If the operator is Tj
or TJ
(it's Tj in your example PDF) then the text is simply appended and no newline is added. Now you wouldn't necessarily want to add a newline, at least if I'm reading the PDF reference right: Tj/TJ
are simply the single and multiple show-string operators, and the existence of a separator of some kind isn't mandatory.
Anyway, if you modify this code to be something like
def extractText(self, Tj_sep="", TJ_sep=""):
[...]
if operator == "Tj":
_text = operands[0]
if isinstance(_text, TextStringObject):
text += Tj_sep
text += _text
[...]
elif operator == "TJ":
for i in operands[0]:
if isinstance(i, TextStringObject):
text += TJ_sep
text += i
then the default behaviour should be the same:
In [1]: pdf.getPage(1).extractText()[1120:1250]
Out[1]: u'ing an individual which, because of name, identifyingnumber, mark or description can be readily associated with a particular indiv'
but you can change it when you want to:
In [2]: pdf.getPage(1).extractText(Tj_sep=" ")[1120:1250]
Out[2]: u'ta" means any information concerning an individual which, because of name, identifying number, mark or description can be readily '
or
In [3]: pdf.getPage(1).extractText(Tj_sep="\n")[1120:1250]
Out[3]: u'ta" means any information concerning an individual which, because of name, identifying\nnumber, mark or description can be readily '
Alternatively, you could simply add the separators yourself by modifying the operands themselves in-place, but that could break something else (methods like get_original_bytes
make me nervous).
Finally, you don't have to edit pdf.py
itself if you don't want to: you could simply pull out this method into a function.