Headless browser for C# (.NET)?

c# .net browser automation web-scraping
Bo Milanovich picture Bo Milanovich · Apr 15, 2012 · Viewed 46.5k times · Source

I am (was) a Python developer who is building a GUI web scraping application. Recently I've decided to migrate to .NET framework and write the same application in C# (this decision wasn't mine).

In Python, I've used the Mechanize library. However, I can't seem to find anything similar in .NET. What I need is a browser that will run in a headless mode, which has the ability to fill out forms, submit them, etc. JavaScript parser is not a must, but it would be quite useful.

Answer

Yahia picture Yahia · Apr 15, 2012

There are some options:

  • WebKit.Net (free)

  • Awesomium
    It is based on Chrome/WebKit and works like a charm. There is a free license available but also a commercial one and if need be you can buy the source code :-)

  • HTML Agility Pack (free) (An HTML Parser library, NOT a headless browser)
    This helps with extracting information from HTML etc. and might be useful in your case (possibly in combination with HttpWebRequest)

Related questions

.NET C#: WebBrowser control Navigate() does not load targeted URL

I'm trying to programmatically load a web page via the WebBrowser control with the intent of testing the page & it's JavaScript functions. Basically, I want to compare the HTML & JavaScript run through this control against a known output …

Read More
How to clear System.Windows.Forms.WebBrowser session data?

How can I clear current session data (cookies, cached data, auth sessions, etc) without restarting the application? Update: I'm talking about WebBrowser control in Windows.Forms, not the ASP.Net session.

Read More
Setting a cookie in a WebBrowser control

I am loading a website using a WebBrowser's Navigate function, and I want the browser to load the page with a cookie I've given it. The following code doesn't work: wb.Navigate(url, null, null, "Cookie: " + cookie + "\n"); What am …

Read More