I have a multi-line string defined like this:
foo = """
this is
a multi-line string.
"""
This string we used as test-input for a parser I am writing. The parser-function receives a file
-object as input and iterates over it. It does also call the next()
method directly to skip lines, so I really need an iterator as input, not an iterable.
I need an iterator that iterates over the individual lines of that string like a file
-object would over the lines of a text-file. I could of course do it like this:
lineiterator = iter(foo.splitlines())
Is there a more direct way of doing this? In this scenario the string has to traversed once for the splitting, and then again by the parser. It doesn't matter in my test-case, since the string is very short there, I am just asking out of curiosity. Python has so many useful and efficient built-ins for such stuff, but I could find nothing that suits this need.
Here are three possibilities:
foo = """
this is
a multi-line string.
"""
def f1(foo=foo): return iter(foo.splitlines())
def f2(foo=foo):
retval = ''
for char in foo:
retval += char if not char == '\n' else ''
if char == '\n':
yield retval
retval = ''
if retval:
yield retval
def f3(foo=foo):
prevnl = -1
while True:
nextnl = foo.find('\n', prevnl + 1)
if nextnl < 0: break
yield foo[prevnl + 1:nextnl]
prevnl = nextnl
if __name__ == '__main__':
for f in f1, f2, f3:
print list(f())
Running this as the main script confirms the three functions are equivalent. With timeit
(and a * 100
for foo
to get substantial strings for more precise measurement):
$ python -mtimeit -s'import asp' 'list(asp.f3())'
1000 loops, best of 3: 370 usec per loop
$ python -mtimeit -s'import asp' 'list(asp.f2())'
1000 loops, best of 3: 1.36 msec per loop
$ python -mtimeit -s'import asp' 'list(asp.f1())'
10000 loops, best of 3: 61.5 usec per loop
Note we need the list()
call to ensure the iterators are traversed, not just built.
IOW, the naive implementation is so much faster it isn't even funny: 6 times faster than my attempt with find
calls, which in turn is 4 times faster than a lower-level approach.
Lessons to retain: measurement is always a good thing (but must be accurate); string methods like splitlines
are implemented in very fast ways; putting strings together by programming at a very low level (esp. by loops of +=
of very small pieces) can be quite slow.
Edit: added @Jacob's proposal, slightly modified to give the same results as the others (trailing blanks on a line are kept), i.e.:
from cStringIO import StringIO
def f4(foo=foo):
stri = StringIO(foo)
while True:
nl = stri.readline()
if nl != '':
yield nl.strip('\n')
else:
raise StopIteration
Measuring gives:
$ python -mtimeit -s'import asp' 'list(asp.f4())'
1000 loops, best of 3: 406 usec per loop
not quite as good as the .find
based approach -- still, worth keeping in mind because it might be less prone to small off-by-one bugs (any loop where you see occurrences of +1 and -1, like my f3
above, should automatically trigger off-by-one suspicions -- and so should many loops which lack such tweaks and should have them -- though I believe my code is also right since I was able to check its output with other functions').
But the split-based approach still rules.
An aside: possibly better style for f4
would be:
from cStringIO import StringIO
def f4(foo=foo):
stri = StringIO(foo)
while True:
nl = stri.readline()
if nl == '': break
yield nl.strip('\n')
at least, it's a bit less verbose. The need to strip trailing \n
s unfortunately prohibits the clearer and faster replacement of the while
loop with return iter(stri)
(the iter
part whereof is redundant in modern versions of Python, I believe since 2.3 or 2.4, but it's also innocuous). Maybe worth trying, also:
return itertools.imap(lambda s: s.strip('\n'), stri)
or variations thereof -- but I'm stopping here since it's pretty much a theoretical exercise wrt the strip
based, simplest and fastest, one.