How does this lambda/yield/generator comprehension work?

Question

I was looking through my codebase today and found this:

def optionsToArgs(options, separator='='):
    kvs = [
        (
            "%(option)s%(separator)s%(value)s" %  
            {'option' : str(k), 'separator' : separator, 'value' : str(v)}
        ) for k, v in options.items()
    ]
    return list(
        reversed(
            list(
                    (lambda l, t: 
                        (lambda f: 
                            (f((yield x)) for x in l)
                        )(lambda _: t)
                    )(kvs, '-o')
                )
            )
        )

It seems to take a dict of parameters and turn them into a list of parameters for a shell command. It looks like it's using yield inside a generator comprehension, which I thought would be impossible...?

>>> optionsToArgs({"x":1,"y":2,"z":3})
['-o', 'z=3', '-o', 'x=1', '-o', 'y=2']

How does it work?

Answer 1

Since Python 2.5, yield <value> is an expression, not a statement. See PEP 342.

The code is hideously and unnecessarily ugly, but it's legal. Its central trick is using f((yield x)) inside the generator expression. Here's a simpler example of how this works:

>>> def f(val):
...     return "Hi"
>>> x = [1, 2, 3]
>>> list(f((yield a)) for a in x)
[1, 'Hi', 2, 'Hi', 3, 'Hi']

Basically, using yield in the generator expression causes it to produce two values for every value in the source iterable. As the generator expression iterates over the list of strings, on each iteration, the yield x first yields a string from the list. The target expression of the genexp is f((yield x)), so for every value in the list, the "result" of the generator expression is the value of f((yield x)). But f just ignores its argument and always returns the option string "-o". So on every step through the generator, it yields first the key-value string (e.g., "x=1"), then "-o". The outer list(reversed(list(...))) just makes a list out of this generator and then reverses it so that the "-o"s will come before each option instead of after.

However, there is no reason to do it this way. There are a number of much more readable alternatives. Perhaps the most explicit is simply:

kvs = [...] # same list comprehension can be used for this part
result = []
for keyval in kvs:
   result.append("-o")
   result.append(keyval)
return result

Even if you like terse, "clever" code, you could still just do

return sum([["-o", keyval] for keyval in kvs], [])

The kvs list comprehension itself is a bizarre mix of attempted readability and unreadability. It is more simply written:

kvs = [str(optName) + separator + str(optValue) for optName, optValue in options.items()]

You should consider arranging an "intervention" for whoever put this in your codebase.