Maven failsafe plugin doesn't run parallel Test

java maven testing maven-failsafe-plugin parallel-testing
SuperMan picture SuperMan · Oct 31, 2013 · Viewed 8k times · Source

I have a Maven POM file and when I provide parallel execution options, I dont see any signs of parallel execution in the logs. And XML debugging is driving me nuts. Any thoughts what is wrong here?

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
   <modelVersion>4.0.0</modelVersion>
   <parent>
      <groupId>com.xxx.xxxx.testing.ranger</groupId>
      <artifactId>ranger-parent</artifactId>
      <relativePath>../parent/pom.xml</relativePath>
      <version>1.0.0</version>
   </parent>
   <artifactId>ranger-api-tests</artifactId>
   <name>Ranger API Tests</name>
   <description>Ranger API integration tests. Depends on the Ranger test framework project.</description>
   <properties>
      <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
      <test.groups />
      <surefire.and.failsafe.report.dir>C:/Users/xxx/Documents/Projects/LearnCubeAPIIT/tests/target</surefire.and.failsafe.report.dir>
   </properties>
   <dependencies>
      <!-- Granite Testing framework -->
      <dependency>
         <groupId>com.xxx.xxxx.testing.ranger</groupId>
         <artifactId>ranger-api-it-framework</artifactId>
         <version>1.0.0</version>
      </dependency>
      <dependency>
         <groupId>commons-lang</groupId>
         <artifactId>commons-lang</artifactId>
         <version>2.4</version>
      </dependency>
      <dependency>
         <groupId>commons-httpclient</groupId>
         <artifactId>commons-httpclient</artifactId>
         <version>3.1</version>
      </dependency>
      <dependency>
         <groupId>org.apache.maven.surefire</groupId>
         <artifactId>surefire-junit47</artifactId>
         <version>2.16</version>
      </dependency>
      <dependency>
         <groupId>junit</groupId>
         <artifactId>junit</artifactId>
         <version>4.11</version>
      </dependency>
   </dependencies>
   <build>
      <plugins>
         <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-failsafe-plugin</artifactId>
            <version>2.16</version>
            <configuration>
               <parallel>classes</parallel>
               <threadCount>4</threadCount>
               <reportsDirectory>${project.build.directory}/surefire-reports</reportsDirectory>
               <groups>${test.groups}</groups>
            </configuration>
            <executions>
               <execution>
                  <goals>
                     <goal>integration-test</goal>
                     <goal>verify</goal>
                  </goals>
               </execution>
            </executions>
         </plugin>
      </plugins>
   </build>
   <reporting>
      <plugins>
         <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-surefire-report-plugin</artifactId>
            <version>2.16</version>
            <configuration>
               <reportsDirectory>${project.build.directory}/surefire-reports</reportsDirectory>
            </configuration>
            <reportSets>
               <reportSet>
                  <id>integration-tests</id>
                  <reports>
                     <report>failsafe-report-only</report>
                     <report>report-only</report>
                  </reports>
               </reportSet>
            </reportSets>
         </plugin>
      </plugins>
   </reporting>
</project>

Logs from run - http://pastebin.com/WkCDwYzJ

Question - When parallel execution options are passed to failsafe plugin, how are the logs supposed to look like to confirm that parallel execution of tests is taking place?

UPDATE

I noticed that when I set debug option for maven to execute, I noticed that parallelExecution is being set to false

[DEBUG] Populating class realm plugin>org.apache.maven.plugins:maven-surefire-plugin:2.12.4--1057774753
[DEBUG]   Included: org.apache.maven.plugins:maven-surefire-plugin:jar:2.12.4
[DEBUG]   Included: org.apache.maven.surefire:surefire-booter:jar:2.12.4
[DEBUG]   Included: org.apache.maven.surefire:surefire-api:jar:2.12.4
[DEBUG]   Included: org.apache.maven.surefire:maven-surefire-common:jar:2.12.4
[DEBUG]   Included: org.apache.commons:commons-lang3:jar:3.1
[DEBUG]   Included: org.apache.maven.shared:maven-common-artifact-filters:jar:1.3
[DEBUG]   Included: org.codehaus.plexus:plexus-utils:jar:3.0.8
[DEBUG]   Included: org.apache.maven.reporting:maven-reporting-api:jar:2.0.9
[DEBUG]   Included: org.apache.maven.plugin-tools:maven-plugin-annotations:jar:3.1
[DEBUG]   Excluded: org.apache.maven:maven-plugin-api:jar:2.0.9
[DEBUG]   Excluded: org.apache.maven:maven-artifact:jar:2.0.9
[DEBUG]   Excluded: org.apache.maven:maven-project:jar:2.0.9
[DEBUG]   Excluded: org.apache.maven:maven-settings:jar:2.0.9
[DEBUG]   Excluded: org.apache.maven:maven-profile:jar:2.0.9
[DEBUG]   Excluded: org.apache.maven:maven-model:jar:2.0.9
[DEBUG]   Excluded: org.apache.maven:maven-artifact-manager:jar:2.0.9
[DEBUG]   Excluded: org.apache.maven:maven-plugin-registry:jar:2.0.9
[DEBUG]   Excluded: org.codehaus.plexus:plexus-container-default:jar:1.0-alpha-9-stable-1
[DEBUG]   Excluded: junit:junit:jar:3.8.1
[DEBUG]   Excluded: org.apache.maven:maven-core:jar:2.0.9
[DEBUG]   Excluded: org.apache.maven:maven-plugin-parameter-documenter:jar:2.0.9
[DEBUG]   Excluded: org.apache.maven:maven-repository-metadata:jar:2.0.9
[DEBUG]   Excluded: org.apache.maven:maven-error-diagnostics:jar:2.0.9
[DEBUG]   Excluded: org.apache.maven:maven-plugin-descriptor:jar:2.0.9
[DEBUG]   Excluded: org.apache.maven:maven-monitor:jar:2.0.9
[DEBUG]   Excluded: classworlds:classworlds:jar:1.1
[DEBUG]   Excluded: org.apache.maven:maven-toolchain:jar:2.0.9
[DEBUG] Configuring mojo org.apache.maven.plugins:maven-surefire-plugin:2.12.4:test from plugin realm ClassRealm[plugin>org.apache.maven.plugins:maven-surefire-plugin:2.12.4--1057774753, parent: sun.misc.Launcher$AppClassLoader@7987aeca]
[DEBUG] Configuring mojo 'org.apache.maven.plugins:maven-surefire-plugin:2.12.4:test' with basic configurator -->
[DEBUG]   (s) basedir = /Users/nallagun/Git/testing/Ranger/tests
[DEBUG]   (s) childDelegation = false
[DEBUG]   (s) classesDirectory = /Users/nallagun/Git/testing/Ranger/tests/target/classes
[DEBUG]   (s) disableXmlReport = false
[DEBUG]   (s) enableAssertions = true
[DEBUG]   (s) forkMode = once
[DEBUG]   (s) junitArtifactName = junit:junit
[DEBUG]   (s) localRepository =        id: local
      url: file:///Users/nallagun/.m2/repository/
   layout: none

[DEBUG]   (s) parallel = classes
[DEBUG]   (f) parallelMavenExecution = false
[DEBUG]   (s) perCoreThreadCount = true

Any Idea why this might be happening?

Thanks

Answer

Boj picture Boj · Nov 1, 2013

Firstly, make sure you're forcing a specific jUnit provider, since older versions (<4.7) are known to have issues with concurrent execution.

<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-failsafe-plugin</artifactId>
  <version>2.16</version>
  <dependencies>
    <dependency>
      <groupId>org.apache.maven.surefire</groupId>
      <artifactId>surefire-junit47</artifactId>
      <version>2.16</version>
    </dependency>
  </dependencies>
</plugin>

You can use the forkCount for concurrent execution option instead of parallel. I find it more flexible and since each test run in its own JVM, you would have less concurrency issues that might make your test results non-deterministic. These configurations apply to maven-surefire-plugin and maven-failsafe-plugin.

<forkCount>8</forkCount>
<reuseForks>true</reuseForks>

You can combine forkCount and parallel, but the different possible combinations of forkCount, reuseForks, etc make things more complicated. Note this from the docs:

When using reuseForks=true and a forkCount value larger than one, test classes are handed over to the forked process one-by-one. Thus, parallel=classes would not change anything. However, you can use parallel=methods: classes are executed in forkCount concurrent processes, each of the processes can then use threadCount threads to execute the methods of one class in parallel.

I'm not sure the reports would tell you if tests ran in concurrent mode or not, but the way I know they are running is by using top or htop (on Linux or OS X). Here's a screenshot of one my test runs. You can see multiple JVMs running tests and consuming CPU. Also my tests run much faster in this fashion.

htop

Related questions

How to get a test resource file?

In a unit test I need to import a csv file. This is located in the resources folder, i.e. src/test/resources

Read More
Running a single test in maven -> No tests were executed!

When I run a single test in Maven with this command: mvn test -Dtest=InitiateTest I'm getting the following result: No tests were executed! It worked a couple of minutes ago, but now it stopped working for some reason. I …

Read More
Getting "java.lang.NoClassDefFoundError: org/junit/platform/engine/DiscoverySelector" trying to run Serenity JBheave

Created a new generic Serenity BDD JBehave framework with Maven ( via command line). Imported the project into Intelli J but when i try to run the "AcceptanceTestSuite" I get an error saying "No Tests are Found" and the following errorException …

Read More