HTML Parse tree using Python 2.7
I was trying to get configure one parse tree for the below HTML table,but couldn't form it.I want to see how the tree structure looks like!can anyone help me here?
# <html>
# <head>
# <title>
# The Dormouse's story
# </title>
# </head>
# <body>
# <p class="title">
# <b>
# The Dormouse's story
# </b>
# </p>
# <p class="story">
# Once upon a time there were three little sisters; and their names were
# <a class="sister" href="http://example.com/elsie" id="link1">
# Elsie
# </a>
# ,
# <a class="sister" href="http://example.com/lacie" id="link2">
# Lacie
# </a>
# and
# <a class="sister" href="http://example.com/tillie" id="link2">
# Tillie
# </a>
# ; and they lived at the bottom of a well.
# </p>
# <p class="story">
# ...
# </p>
# </body>
# </html>
EDIT
Microsoft Windows [Version 6.1.7600]
Copyright (c) 2009 Microsoft Corporation. All rights reserved.
C:\Users\matt>easy_install ete2
Searching for ete2
Reading http://pypi.python.org/simple/ete2/
Reading http://ete.cgenomics.org
Reading http://ete.cgenomics.org/releases/ete2/
Reading http://ete.cgenomics.org/releases/ete2
Best match: ete2 2.1rev539
Downloading http://ete.cgenomics.org/releases/ete2/ete2-2.1rev539.tar.gz
Processing ete2-2.1rev539.tar.gz
Running ete2-2.1rev539\setup.py -q bdist_egg --dist-dir c:\users\arupra~1\appdat
a\local\temp\easy_install-sypg3x\ete2-2.1rev539\egg-dist-tmp-zemohm
Installing ETE (A python Environment for Tree Exploration).
Checking dependencies...
numpy cannot be found in your python installation.
Numpy is required for the ArrayTable and ClusterTree classes.
MySQLdb cannot be found in your python installation.
MySQLdb is required for the PhylomeDB access API.
PyQt4 cannot be found in your python installation.
PyQt4 is required for tree visualization and image rendering.
lxml cannot be found in your python installation.
lxml is required from Nexml and Phyloxml support.
However, you can still install ETE without such functionality.
Do you want to continue with the installation anyway? [y,n]y
Your installation ID is: d33ba3b425728e95c47cdd98acda202f
warning: no files found matching '*' under directory '.'
warning: no files found matching '*.*' under directory '.'
warning: manifest_maker: MANIFEST.in, line 4: path 'doc/ete_guide/' cannot end w
ith '/'
warning: manifest_maker: MANIFEST.in, line 5: path 'doc/' cannot end with '/'
warning: no previously-included files matching '*.pyc' found under directory '.'
zip_safe flag not set; analyzing archive contents...
Adding ete2 2.1rev539 to easy-install.pth file
Installing ete2 script to C:\Python27\Scripts
Installed c:\python27\lib\site-packages\ete2-2.1rev539-py2.7.egg
Processing dependencies for ete2
Finished processing dependencies for ete2
Answer
This answer comes a bit late, but still I'd like to share it:
I used networkx and lxml (which I found to allow much more elegant traversal of the DOM-tree). However, the tree-layout depends on graphviz and pygraphviz installed. networkx itself would just distribute the nodes somehow on the canvas. The code actually is longer than required cause I draw the labels myself to have them boxed (networkx provides for drawing the labels but it doesn't pass on the bbox keyword to matplotlib).
import networkx as nx
from lxml import html
import matplotlib.pyplot as plt
from networkx.drawing.nx_agraph import graphviz_layout
raw = "...your raw html"
def traverse(parent, graph, labels):
labels[parent] = parent.tag
for node in parent.getchildren():
graph.add_edge(parent, node)
traverse(node, graph, labels)
G = nx.DiGraph()
labels = {} # needed to map from node to tag
html_tag = html.document_fromstring(raw)
traverse(html_tag, G, labels)
pos = graphviz_layout(G, prog='dot')
label_props = {'size': 16,
'color': 'black',
'weight': 'bold',
'horizontalalignment': 'center',
'verticalalignment': 'center',
'clip_on': True}
bbox_props = {'boxstyle': "round, pad=0.2",
'fc': "grey",
'ec': "b",
'lw': 1.5}
nx.draw_networkx_edges(G, pos, arrows=True)
ax = plt.gca()
for node, label in labels.items():
x, y = pos[node]
ax.text(x, y, label,
bbox=bbox_props,
**label_props)
ax.xaxis.set_visible(False)
ax.yaxis.set_visible(False)
plt.show()
Changes to the code if you prefer (or have) to use BeautifulSoup:
I'm no expert... just looked at BS4 for the first time,... but it works:
#from lxml import html
from bs4 import BeautifulSoup
from bs4.element import NavigableString
...
def traverse(parent, graph, labels):
labels[hash(parent)] = parent.name
for node in parent.children:
if isinstance(node, NavigableString):
continue
graph.add_edge(hash(parent), hash(node))
traverse(node, graph, labels)
...
#html_tag = html.document_fromstring(raw)
soup = BeautifulSoup(raw)
html_tag = next(soup.children)
...