I'm trying to get the content "My home address" using the following but got the AttributeError:
address = soup.find(text="Address:")
print address.nextSibling
This is my HTML:
<td><b>Address:</b></td>
<td>My home address</td>
What is a good way to navigate down td tag and pull the content?
Answer
The problem is that you have found a NavigableString, not the <td>. Also nextSibling will find the next NavigableString or Tag so even if you had the <td> it wouldn't work the way you expect.
This is what you want:
address = soup.find(text="Address:")
b_tag = address.parent
td_tag = b_tag.parent
next_td_tag = td_tag.findNext('td')
print next_td_tag.contents[0]
Or more concise:
print soup.find(text="Address:").parent.parent.findNext('td').contents[0]
Actually you could just do
print soup.find(text="Address:").findNext('td').contents[0]
Since findNext just calls next over and over again, and next finds the next element as parsed repeatedly until it matches.