What is the difference between NightwatchJS and WebdriverIO?

Question

As the title states, what is the difference between Nightwatch.js and Webdriver.io?

It seems like they have kind of the same syntax and do almost the same thing. How are they different?

I need to choose between them.

Answer 1

I've written a test suite using each of these tools a few times.

Webdriver.io allows you to write your test cases "from scratch" and have great control over reporting, by say, integrating with slack using slack npm, and other packages. You would need to know or quickly learn node.js. In addition to working very well with desktop browsers, it integrates well with Appium, Android Studio, and Xcode so you can run your automated tests on Android emulators and iOS simulators locally. You will need to install those things and write some code to tell Appium which drivers to use, and select capabilities, etc.

Nightwatch is a fairly extensive solution that uses an iterator to automatically re-try tests up to three times when they fail. Nightwatch has decent support for integration with VM tools like SauceLabs so that you can theoretically run your test cases against 700+ different platform/browser/version combinations without writing code to manage each driver. Nightwatch handles starting and shutting down selenium for you. While this last sounds very impressive, in reality it's quite a lot of work to attain & maintain that level of test coverage. Nightwatch also has fairly built-in separation of concerns, allowing you to define custom commands and require them in your base test case or individual tests. You could modularize some portions of tests and import them so that you don't have to constantly re-write say, the login test for use in multiple cases. Additionally, you could use custom commands to import selectors as key value pairs.

Having used each I would summarize it this way:

webdriver.io: If you're looking for more control, a very custom solution and you don't need an iterator, and you are confident that you know enough to write code for selecting your browser driver, setting capabilities and you want custom control of your reporting.

Nightwatch: if you want to just start writing tests quickly, to know that it will be relatively easy to run them against specific platforms/browsers/versions and will still allow you significant flexibility to extend your tests by writing custom commands.

Another option out there right now is Dalek.js, which has the easy script creation of Nightwatch but without all the bells and whistles.

Prior to running nightwatch, you can configure browsers in the Magellan.json file, and then when you run your test you call the browsers, or a set of browsers (a "profile") as command line arguments, thusly:

For local browsers:

./node_modules/.bin/magellan --serial --browsers=chrome,firefox

Assuming you have setup a saucelabs account and added you username and access key, you could call a profile of browsers like this:

./node_modules/.bin/magellan --serial --profile=myBrowsers

This assumes you have setup a profile called myBrowsers in the Magellan.json file like this:

{
    "profiles": {
        "myBrowsers": [
          { "browser": "chrome_46_OS_X_10_10_Desktop" },
          { "browser": "firefox_42_Windows_2012_R2_Desktop" },
          { "browser": "safari_8_OS_X_10_10_Desktop" },
          { "browser": "safari_7_OS_X_10_9_Desktop" }, 
          { "browser": "safari_9_OS_X_10_11_Desktop" }, 
          { "browser": "IE_10_Windows_2012_Desktop" }, 
          { "browser": "IE_11_Windows_2012_R2_Desktop" },
          { "browser": "chrome_45_OS_X_10_8_Desktop" },
          { "browser": "chrome_45_OS_X_10_9_Desktop" },
          { "browser": "chrome_45_OS_X_10_10_Desktop" },
          { "browser": "chrome_45_OS_X_10_11_Desktop" },
          { "browser": "chrome_46_OS_X_10_10_Desktop" }, 
          { "browser": "chrome_45_Windows_10_Desktop" },
          { "browser": "chrome_45_Windows_2003_Desktop" },
          { "browser": "chrome_45_Windows_2008_Desktop" },
          { "browser": "chrome_45_Windows_2012_Desktop" },
          { "browser": "chrome_45_Windows_2012_R2_Desktop" },
          { "browser": "chrome_46_Windows_10_Desktop" },
          { "browser": "chrome_46_Windows_2003_Desktop" },
          { "browser": "chrome_46_Windows_2008_Desktop" },
          { "browser": "chrome_46_Windows_2012_Desktop" },
          { "browser": "chrome_46_Windows_2012_R2_Desktop" }, 
          { "browser": "firefox_42_OS_X_10_9_Desktop" }, 
          { "browser": "firefox_42_Windows_2012_R2_Desktop" },
          { "browser": "android_4_4_Linux_Samsung_Galaxy_S4_Emulator", "orientation": "portrait" },
          { "browser": "ipad_8_4_iOS_iPad_Simulator", "orientation": "landscape"},
          { "browser": "ipad_8_4_iOS_iPad_Simulator", "orientation": "landscape"},
          { "browser": "ipad_9_0_iOS_iPad_Simulator", "orientation": "landscape"},
          { "browser": "ipad_9_0_iOS_iPad_Simulator", "orientation": "portrait"},
          { "browser": "ipad_9_1_iOS_iPad_Simulator", "orientation": "landscape"},
          { "browser": "ipad_9_1_iOS_iPad_Simulator", "orientation": "portrait"},
          { "browser": "iphone_9_1_iOS_iPhone_Simulator", "orientation": "portrait"},
          { "browser": "iphone_9_1_iOS_iPhone_Simulator", "orientation": "landscape"}

        ]
}

}

SOME OF THE MORE USEFUL (optional) COMMAND LINE ARGS:

toggling the --serial argument results in the test execution being serialized and with more verbose test experience where you can review the errors that have been returned during a run. It also take much longer to run as it wait for tests to complete.

adding the --sauce argument once your test cases work for browsers that exist on your local machine, you can tap into the (currently) 760 browsers supported by Sauce Labs. Go ahead an paste this into the terminal & hit return:

./node_modules/.bin/magellan --serial --list_browsers

For each device/browser you want to test you just add the listing in the Copy-Paste Command-Line Option column as comma separated values after --browser= when executing the script. NOTE: when running without --sauce you can just use --browser=chrome or --browser=chrome,firefox

BREAKING IT DOWN:

Using nightwatch without the --sauce but with --serial is a great way to get started. Work on your script until you have validated the things you want to check and when you are confident that all tests that should pass, do, run it with sauce labs and the major browsers you want to test. Once you are confident that the major browsers are covered you can run it without --serial to reduce run time (useful on Sauce Labs, which will cost money).

But enough proselytizing, you can find out what you need about Saucelabs here: https://wiki.saucelabs.com/display/DOCS/The+Sauce+Labs+Cookbook+Home

And for a boilerplate example of Nightwatch to do the canonical hello world: try this boilerplater

UPDATES: A few points that people are bringing up and that have occurred to me since posting this.

Webdriver.io: Since there is no iterator, there is less ability to recover from failures during a test execution, this means failures are more definitive. Because this is purely async, you may have headaches tracing the exact origin of the failure. You may also end up having to create separate tear-down scripts for any data you create to avoid data collisions during execution.

Nightwatch.js: Since the iterator allows you retries, you will often be able to find where your script is failing. This may allow you to more quickly find the defect instead of focusing on why your script is failing. It's also easier to toggle off your individual scripts.

UPDATE 2:

With Nightwatch shorter tests are useful/encouraged. Because the iterator reads the test files into memory each and every iteration immediately before execution, you can very literally edit the tests between iteration execution. Let me say that a different way: Your Nightwatch suite:

test_1 starts
test_1 FAIL    // because you made a trivial error in your test case
test-2 starts  // while it is running, you make the change, save it
test-2 PASS
test_1 starts  // the iteration starts * with your change! *
test_1 PASS
============= Suite Complete =============

     Status: PASSED
    Runtime: 2m 48.3s
Total tests: 2
 Successful: 2 / 2
1 test(s) have retried: 1 time(s)

On the other hand, setting up Slack webhooks with node/webdriver.io is easy. What this means is that you can set up your node/webdriver.io tests to report to Slack when they complete. Clients appreciate this because after a build has completed they soon see the results of the automation, like:

✅ Automated testing of [client/product name here] Sprint ##.#.# passed on [server URL or IP address] with OS X Firefox 59.0.2

❌ Automated testing of [client/product name here] Sprint ##.#.# failed on [server URL or IP address] with OS X Firefox 59.0.2

UPDATE 3 (August 6, 2017)

Having spent another year and half working with both on a daily basis, I want to add the following points.

There are similar numbers of NPM packages out there that integrate with each, but you'll note that there are far more Stackoverflow questions about Nightwatch (4x). I believe this is because Webdriver.io is more of a roll-your-own approach to automated testing [that's my opinion, and I welcome feedback/pushback]. Those who use it won't have questions about how to use it, they'll have specific questions about techniques.

Nightwatch would be a better entry point for someone with extensive Selenium IDE and solid javascript experience. It has a lot of useful solutions out of the box. What little experience I have with Dalek suggests that would similarly be a good option.

Someone with more javascript and perhaps some Object Oriented Programming and unix experience would likely find Webdriver.io to be better. It's just a great option to build your own custom framework, as I am currently doing. If you can envision how you would like your initialization, flow control and reporting to work, and are willing to put in the sweat equity, it's apt.

I was asked below which I prefer, and I prefer Webdriver.io for extensive e2e tests by far. While I often use a personalized Nightwatch repo for most client work built on top of our platform, that may change in the near future as I build my own Webdriver.io solution.

UPDATE 4 (May 2, 2018)

Updated for clarity about controlling Selenium and browser drivers as well as adding some details about using Appium and Xcode/Android Studio.