Jenkins building a product consisting of many Maven projects? (with Jenkins Pipeline plugin?)

Question

We have a product that consists of many Maven projects that depend on each other. All of these Maven projects come together in a single project which delivers the end product.

The Maven projects share the same life cycle. In other words, they are not managed by separate teams of people with explicit <dependency> changes to pick up newer versions of other projects. Rather, when someone changes something in one of the projects, then the result should go directly into the end product without additional changes.

We use Jenkins as our Continuous Integration tool.

The main wishes we have are as follows:

• No need to copy all the inter-project dependencies to Jenkins configuration: these should be in a single place, ideally the pom.xml files.
• Avoid unnecessary builds: on an SCM change, only build the projects that are potentially affected.
• In case of diamond dependencies (C depends on both B1 and B2, which both depend on A), if the lowest (A) is changed, then the end product (C) should always use the version of A that was also used to build/test B1 and B2.

Question: What is the best approach to do this with Jenkins?

We are currently thinking to use a single job using the Jenkins Pipeline plugin, which analyzes the Maven dependencies and the SCM changes, decides what needs to be built and in which order, and then actually build the projects.

Answer 1

In order to achieve your requirements you could rely on many of the default features of a maven multi-module build and an additional configuration explained below.

From your question:

No need to copy all the inter-project dependencies to Jenkins configuration: these should be in a single place, ideally the pom.xml files.

Using an aggregator/multi-module pom you can have a single entry point (a pom.xml file) for a maven build which would then build all of the defined modules:

A project with modules is known as a multimodule, or aggregator project. Modules are projects that this POM lists, and are executed as a group. An pom packaged project may aggregate the build of a set of projects by listing them as modules, which are relative directories to those projects.

That is, a pom.xml file like the following:

<project>
    <modelVersion>4.0.0</modelVersion>
    <groupId>com.sample</groupId>
    <artifactId>project</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <packaging>pom</packaging>

    <modules>
        <module>module-1</module>
        <module>module-2</module>
        <module>module-n</module>
    </modules>
</project>

Is a minimal example: define a list of modules (other maven projects) which will be built starting from this maven project (an empty project which only purpose is to build other maven projects). Note the pom packaging required to have modules, it's telling maven that this project only provides a pom (no further artifacts).

You could hence have a root Maven aggregator project which would define the other maven projects as its modules and have a single Jenkins job building this entry point.

---

Additionally, from your question:

Avoid unnecessary builds: on an SCM change, only build the projects that are potentially affected.

To meet this requirement, you could use the incremental-build-plugin:

Because it looks for modification on modules of an unique project, the Maven Incremental Plugin takes it sens only on multi modules projects. If modification on a module is detected, the output directory is removed.

This plugin will verify whether any of the pom file, resources, sources, test sources, test resources would change in a module and if the case remove its output directory. As such, saving build time for a concerned module and in chain for the whole multi-module project (our own build, in this case).

To enable this mechanism, you can configure in the aggregator pom above, the following:

<project>
    <modelVersion>4.0.0</modelVersion>
    <groupId>com.sample</groupId>
    <artifactId>project</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <packaging>pom</packaging>
    <modules>
        <module>module-1</module>
        <module>module-2</module>
        <module>module-n</module>
    </modules>

    <properties>
        <skip.incremental>true</skip.incremental>
    </properties>

    <build>
        <plugins>
            <plugin>
                <groupId>net.java.maven-incremental-build</groupId>
                <artifactId>incremental-build-plugin</artifactId>
                <version>1.6</version>
                <executions>
                    <execution>
                        <goals>
                            <goal>incremental-build</goal>
                        </goals>
                        <configuration>
                            <noIncrementalBuild>${skip.incremental}</noIncrementalBuild>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>
</project>

Above, we have simply added the incremental-build-plugin to the aggregator build and a property, skip.incremental, which will skip the plugin execution for the first build, the empty aggregator one, while enabling it into the modules as following:

<project>
    <modelVersion>4.0.0</modelVersion>
    <parent>
        <groupId>com.sample</groupId>
        <artifactId>project</artifactId>
        <version>0.0.1-SNAPSHOT</version>
    </parent>
    <artifactId>simple-module</artifactId>

    <properties>
        <skip.incremental>false</skip.incremental>
    </properties>
</project>

Note: in the pom above of a sample module we are pointing at the aggregator pom file as a parent, hence using maven inheritance in combination with aggregation (a classic usage) and as such having a multi-module build plus a common build governance provided by the common parent for all the declared modules (in this case, the common build governance provides the additional incremental-build-plugin configuration). Moreover, each module re-configures the skip.incremental property to not skip the aforementioned plugin. That's a trick: now the plugin will only be executed in modules, not in its root (which would not make sense and in this case would actually throw an error otherwise).

Obviously, in the related Jenkins job, in its Source Code Management section we don't have to configure a fresh new check-out as part of each build, otherwise no changes would be detectable (and everytime it would start from zero, which is a good practice for release builds actually).

Moreover, your maven command should not invoke the clean lifecycle, which would also remove the target at each build and as such make no change detectable either.

---

Furthermore, from your question:

In case of 'diamond dependencies' (C depends on both B1 and B2, which both depend on A), if the lowest (A) is changed, then the end product (C) should always use the version of A that was also used to build/test B1 and B2.

Maven will take care of this requirement as part of a multi-module build and its reactor mechanism:

The mechanism in Maven that handles multi-module projects is referred to as the reactor. This part of the Maven core does the following:

• Collects all the available modules to build
• Sorts the projects into the correct build order
• Builds the selected projects in order

By default, the reactor will also hence create a build order and always build a module before its dependent/consumer module. That also means that the last built module will actually be the module responsible of building the final artifact, the deliverable product depending on part or all of the other modules (for instance, a module responsible of delivering a war file will probably be the last one to build in a multi-module webapp project).

---

Further related reading concerning Maven and skip actions when something is unchanged:

• The maven-jar-plugin provides the forceCreation which by default is already enabled

  Require the jar plugin to build a new JAR even if none of the contents appear to have changed. By default, this plugin looks to see if the output jar exists and inputs have not changed. If these conditions are true, the plugin skips creation of the jar.

• The maven-compiler-plugin provides the useIncrementalCompilation, although not properly working at the moment.
• The maven-war-plugin provides the recompressZippedFiles option which could also be used to speed up builds and avoid re-doing something (to switch to false in this case):

  Indicates if zip archives (jar,zip etc) being added to the war should be compressed again. Compressing again can result in smaller archive size, but gives noticeably longer execution time.
  Default: true

---

Update

The Maven projects share the same life cycle.

This requirement would also enforce the usage of an aggregator/multi-module project, however may also apply to different projects linked through dependencies indeed.

they are not managed by separate teams of people with explicit changes to pick up newer versions of other projects.

This point also enforces the usage of a multi-module project. In a multi-module project you may have different versions among modules, however the common practice and guideline is to share the same version, defined by the parent project (the aggregator) and cascaded across its modules. As such, its centralization and governance will avoid misalignments and mistakes.

when someone changes something in one of the projects, then the result should go directly into the end product without additional changes.

Again, this would be handled automatically by a multi-module project. The same would happen with different projects each using SNAPSHOT versions, then no need to change the dependency version in its consumer project (the one responsible of building the final product). However, while SNAPSHOT versions are really helpful (and recommended) during development, they should definitely not be used when delivering the final product since build reproducibility would be in danger (that is, you may not be able to re-build the same version later on since it was relying on SNAPSHOT versions, hence not frozen versions). Hence, SNAPSHOT is not a silver bullet solution and should be used only during certain phases of the SDLC of the product, not as a finalized and frozen solution.

---

Update 2
Worth to also look at the new Maven Incremental Module Builder for Maven 3.3.1+ and Java 7.

More details on:

• Its Github project
• Its author blog post (one the main Apache Maven contributors)