Headless, scriptable Firefox/Webkit on linux?

firefox webkit screen-scraping headless-browser
Parand picture Parand · Jan 15, 2010 · Viewed 21.8k times · Source

I'm looking to automate some web interactions, namely periodic download of files from a secure website. This basically involves entering my username/password and navigating to the appropriate URL.

I tried simple scripting in Python, followed by more sophisticated scripting, only to discover this particular website is using some obnoxious javascript and flash based mechanism for login, rendering my methods useless.

I then tried HTMLUnit, but that doesn't seem to want to work either. I suspect use of Flash is the issue.

I don't really want to think about it any more, so I'm leaning towards scripting an actual browser to log in and grab the file I need.

Requirements are:

  • Run on linux server (ie. no X running). If I really need to have X I can make that happen, but I won't be happy.
  • Be reliable. I want to start this thing and never think about it again.
  • Be scriptable. Nothing too sophisticated, but I should be able to tell the browser the various steps to take and pages to visit.

Are there any good toolkits for a headless, X-less scriptable browser? Have you tried something like this and if so do you have any words of wisdom?

Answer

Phil picture Phil · Feb 24, 2011

What about phantomjs?

Related questions

Custom CSS Scrollbar for Firefox

I want to customize a scrollbar with CSS. I use this WebKit CSS code, which works well for Safari and Chrome: ::-webkit-scrollbar { width: 15px; height: 15px; } ::-webkit-scrollbar-track-piece { background-color: #C2D2E4; } ::-webkit-scrollbar-thumb:vertical { height: 30px; background-color: #0A4C95; } How can …

Read More
What is the correct "-moz-appearance" value to hide dropdown arrow of a <select> element

I'm trying to style the dropdown arrow of a <select> element with CSS only , it works perfectly in Chrome/Safari: select { -webkit-appearance: button; -webkit-border-radius: 2px; -webkit-box-shadow: 0px 1px 3px rgba(0, 0, 0, 0.1); -webkit-padding-end: 20px; -webkit-padding-start: 2px; -webkit-user-select: none; background-image: url(…

Read More
Image scaling by CSS: is there a webkit alternative for -moz-crisp-edges?

I have an image that is 100x100 in pixels. I want to show it twice the size, so 200x200 and I want to do it by CSS and (explicitly) not by the server. Since a few years, images get anti-aliased …

Read More