How to calculate code coverage of selenium tests with respect to web application code

Question

I have a requirement to capture the code coverage of my selenium tests to the amount of source code in the server code (web application source code) covered .

For example the tests for login functionality should capture the amount of code covered in web application for the login function.

Else need to know which package or class it has touched in web application code for a scenario done . Eg a login

I couldn't find a suitable solution , although i came to know about Jacoco code coverage tool and tried some samples with the use of Jacoco Jenkins plugin , but there is no success .

I am not aware of the possiblity. Please provide me a suggestion on how to achieve this , Thanks in advance

Answer 1

Make sure you do this. Note sure if you are using Gradle, Maven or ANT. But the following concept is very similar to any build system.

1. You must be having a .war/.ear app artifact that you need to run behind Tomcat / similar.
   • You also need to make sure that you compiled your main code in debug mode, or jacoco will not be happy.
   • For ex: -g option in Java and similar debug option (if you are using groovy).

tasks.withType(Compile) {
    options.debug = true
    options.compilerArgs = ["-g"]
}

2. Lets say you have the .war and using Tomcat. Then while starting Tomcat.
   • In Tomcat start script, make sure you are telling Tomcat's JVM where's jacocoagent.jar file and passing other parameters. THIS is the MAIN missing point sometime we see (aka not attaching jacoco to target JVM's session and trying to get code coverage).

For ex: I start my Tomcat script with the following parameter passed to Tomcat (-Dxxxx=value way)

PROJ_EXTRA_JVM_OPTS=-javaagent:tomcat/jacocoagent.jar=destfile=build/jacoco/ST/jacocoST.exec,append=false

Basically, Tomcat start script would have -Dparameter=value, you can pass the above parameter (Linux/Unix export the variable) to Tomcat/Target JVM's scope.

The above parameter line when sent to Tomcat, will attach JACOCO agent .jar file to the "TARGET" (aka Tomcat JVM). Here you are telling Tomcat that go look for jacocoagent.jar file from a director called "tomcat" under your workspace. It's gonna create a jacoco .exec file named "jacocoST.exec" (aka jacoco exec file for Selenium Test) under build/jacoco/ST folder (I'm using Gradle so Gradle creates "build" folder anytime you run a build/compile/test/integrationTest/customSeleniumTaskThatYouMightHaveCreated).

NOTE: This means, that you DON'T have to specify jacoco section in the test task (as that'll run in your BUILD systems' JVM either Gradle or Maven or ANT whatever you have).

//We don't need jacoco for non-unit tests type of tasks as Jacoco won't be able to find any coverage if done this way. Jacoco agent file needs to be attached/visible to the TARGET's JVM (where you run your application via a .war / .ear etc).

jacoco { 
  //  ... As Gradle runs Unit tests (while doing build), they run free, in the same JVM where Gradle runs the build so Unit test have visibility to the main classes in the same JVM (which Gradle is using to run the build). Thus, you can use jacoco section in Gradle for running unit tests. BUT,
  // ... Don't use this section for running Integration, Acceptance, Selenium tests which run on a target JVM. Instead attach jacocoagent.jar and specify jacoco parameters to the target JVM.
}

3. Once you have your Tomcat up and running, now you run your Selenium tests. NOTE: -- I'm using Jenkins on Linux/Unix machine and "xvfb" plugin is very handy i.e. now I can run Selenium GUI tests in HEADLESS mode and I won't bug any user on a machine where the tests are running by popping up the tests pages while the GUI tests are running.

   -- if you end up using "xvfb" plugin in Jenkins, you FIRST need to start "Xvfb" service on the server (Linux/Unix) where you are running the tests.

   -- If you are running your non-units tests (aka Integration/Selenium etc) on a Windows machine, then you can see the GUI tests pop up when you run your tests. If you don't want to see the popup windows, then your Jenkins instance can run the slave (your windows machine) process as a service ("Install as a Service"). If you create your windows machine as a slave, when you run the JLNP installation on your machine, You'll see a popup that Jenkins has successfully started a slave process, clicking File > Install as a service will run your slave on a windows machine as "HEADLESS".

4. While your tests are running, you'll notice that this time, jacoco will create a folder structure/exec file as per your defined value for destfile parameter but it'll still be 0 or some small size.

5. Once your Selenium/non-unit tests are complete, you have to "STOP" Tomcat / target JVM. This will FLUSH all jacoco coverage info to this jacocoST.exec file (custom file that you wanted jacoco to create). -- Note: If you want jacocoST.exec file to be flushed on the fly (without requiring the Tomcat JVM/session to stop, then you can look into jacoco documentation how to do that, there is one topic there which tells about this, this way your application can continue to run and you dont have to stop your application/webservice).

6. Run jacocoTestReport task and you'll see jacoco code coverage.

   • Make sure you specify where are your sources/classes for main code.

for ex:

  jacocoTestReport {
      group = "Reporting"
      description = "Generate Jacoco coverage reports after running tests."
      ignoreFailures = true


      //UT=is for Unit tests, IT=integrationTest, AT=acceptanceTest, ST=Selenium GUI tests.
      //executionData = files('build/jacoco/UT/jacocoUT.exec')
      //executionData = files('build/jacoco/IT/jacocoIT.exec')
      //executionData = files('build/jacoco/UT/jacocoUT.exec', 'build/jacoco/IT/jacocoIT.exec')

      //executionData = files(['build/jacoco/UT/jacocoUT.exec', 'build/jacoco/IT/jacocoIT.exec'])
      //OR use the following way for all.
      executionData = fileTree(dir: 'build/jacoco', include: '**/*.exec')

      reports {
             xml{
                 enabled true
                 //Following value is a file
                 destination "${buildDir}/reports/jacoco/xml/jacoco.xml"
             }
             csv.enabled false
             html{
                 enabled true
                 //Following value is a folder
                 destination "${buildDir}/reports/jacoco/html"
             }
      }

      //sourceDirectories = files(sourceSets.main.allJava.srcDirs)
      sourceDirectories = files('src/java')
      //sourceDirectories = files(['src/java','src/groovy'])
      classDirectories =  files('build/classes/main')

      //------------------------------------------
      //additionalSourceDirs = files(['test/java','test/groovy','src/java-test', 'src/groovy-test'])
      //additionalSourceDirs += files('src/java-test')
}

Feel free to ping me if you still see any issues. You can also see few of my posts here on stackoverflow on how I achieved this and also publishing the same coverage to SonarQube.