Malformed String ValueError ast.literal_eval() with String representation of Tuple

Question

I'm trying to read in a string representation of a Tuple from a file, and add the tuple to a list. Here's the relevant code.

raw_data = userfile.read().split('\n')
for a in raw_data : 
    print a
    btc_history.append(ast.literal_eval(a))

Here is the output:

(Decimal('11.66985'), Decimal('0E-8'))
Traceback (most recent call last):


File "./goxnotify.py", line 74, in <module>
    main()
  File "./goxnotify.py", line 68, in main
    local.load_user_file(username,btc_history)
  File "/home/unix-dude/Code/GoxNotify/local_functions.py", line 53, in load_user_file
    btc_history.append(ast.literal_eval(a))
  File "/usr/lib/python2.7/ast.py", line 80, in literal_eval
    return _convert(node_or_string)

  `File "/usr/lib/python2.7/ast.py", line 58, in _convert
   return tuple(map(_convert, node.elts))
  File "/usr/lib/python2.7/ast.py", line 79, in _convert
   raise ValueError('malformed string')
   ValueError: malformed string

Answer 1

ast.literal_eval (located in ast.py) parses the tree with ast.parse first, then it evaluates the code with quite an ugly recursive function, interpreting the parse tree elements and replacing them with their literal equivalents. Unfortunately the code is not at all expandable, so to add Decimal to the code you need to copy all the code and start over.

For a slightly easier approach, you can use ast.parse module to parse the expression, and then the ast.NodeVisitor or ast.NodeTransformer to ensure that there is no unwanted syntax or unwanted variable accesses. Then compile with compile and eval to get the result.

The code is a bit different from literal_eval in that this code actually uses eval, but in my opinion is simpler to understand and one does not need to dig too deep into AST trees. It specifically only allows some syntax, explicitly forbidding for example lambdas, attribute accesses (foo.__dict__ is very evil), or accesses to any names that are not deemed safe. It parses your expression fine, and as an extra I also added Num (float and integer), list and dictionary literals.

Also, works the same on 2.7 and 3.3

import ast
import decimal

source = "(Decimal('11.66985'), Decimal('1e-8'),"\
    "(1,), (1,2,3), 1.2, [1,2,3], {1:2})"

tree = ast.parse(source, mode='eval')

# using the NodeTransformer, you can also modify the nodes in the tree,
# however in this example NodeVisitor could do as we are raising exceptions
# only.
class Transformer(ast.NodeTransformer):
    ALLOWED_NAMES = set(['Decimal', 'None', 'False', 'True'])
    ALLOWED_NODE_TYPES = set([
        'Expression', # a top node for an expression
        'Tuple',      # makes a tuple
        'Call',       # a function call (hint, Decimal())
        'Name',       # an identifier...
        'Load',       # loads a value of a variable with given identifier
        'Str',        # a string literal

        'Num',        # allow numbers too
        'List',       # and list literals
        'Dict',       # and dicts...
    ])

    def visit_Name(self, node):
        if not node.id in self.ALLOWED_NAMES:
            raise RuntimeError("Name access to %s is not allowed" % node.id)

        # traverse to child nodes
        return self.generic_visit(node)

    def generic_visit(self, node):
        nodetype = type(node).__name__
        if nodetype not in self.ALLOWED_NODE_TYPES:
            raise RuntimeError("Invalid expression: %s not allowed" % nodetype)

        return ast.NodeTransformer.generic_visit(self, node)


transformer = Transformer()

# raises RuntimeError on invalid code
transformer.visit(tree)

# compile the ast into a code object
clause = compile(tree, '<AST>', 'eval')

# make the globals contain only the Decimal class,
# and eval the compiled object
result = eval(clause, dict(Decimal=decimal.Decimal))

print(result)