Invalid or corrupt JAR File built by Maven shade plugin

Question 1

Invalid or corrupt JAR File built by Maven shade plugin

java maven jfreechart maven-shade-plugin

d0x · Oct 23, 2012 · Viewed 24k times · Source

Answer

Answer

On my end if I build your project using the pom.xml you've showed us, with apache-poi declared after jfreechart, then as you've mentioned I get a corrupt JAR. Swapping the order of these two dependencies indeed gives me a correct JAR.

I've some previous experience with the maven-shade-plugin and when I used it I had trouble with the created JAR's META-INF directory, so I've checked that for defects.

I've tried to copy (using Total Commander) the META-INF directory somewhere on my local filesystem and what I've noticed is that there were errors while copying the files in META-INF/licences/. If I've tried to copy those somewhere individually and it worked but I couldn't copy the whole thing. My conclusion was that the JAR/ZIP archive is corrupt.

What I did was I entered that JAR in Total Commander (Ctrl+PgDown over the JAR file) and I renamed thirdpartylicenses.txt to thirdpartylicenses.txt.wtf. Doing this Total Commander offers to save this and it repackages the JAR (I've the Total7zip Total Commander packer plugin installed - if anyone tries this and it doesn't work try it with this installed).

After this. It works.

(I've also tried to repack the whole thing without renaming anything using Cygwin's unzip/zip commands, but that didn't work the new archive was still corrupt. Total Commander or the plugin I've mentioned does some magic.)

I guess the maven-shade-plugin merely creates a corrupt or invalid ZIP/JAR archive. I'm not sure why and maybe what I've described won't work for anyone else but I thought I'd mention it so maybe I could help.

I couldn't just leave this alone, so I've dug deeper and I think I've found the answer.

The bad JAR contains 65608 entries. The good JAR contains 65450 entries.

Guess what's the upper limit of the number of entries for a plain ZIP file? Yeah. The Wiki article speaks about a ZIP64 format which overcomes this limitation.

The good JAR has less entries because the actual dependencies change because of the position of the dependency declarations in your pom.xml. (As described by this answer.)

I've counted the entries like this.

Collections.list(new JarFile("...").entries()).size();

I was using Java 7 which seems to support the new ZIP64 format, so maybe if someone tries to count the entries in the bad JAR using Java 5 or 6 will receive an error (I'm not sure though).

I also tried to run the JAR exploded. I've unpacked the whole JAR into a directory and run the whole thing like this.

java -cp <dir/ com.mycompany.test.Start

Worked like a charm.

Bottom line. Don't overuse the maven-shade-plugin.

I have a project at work where I build my project like this.

I copy my project's dependencies using the maven-dependency-plugin. Checkout the copy-dependencies goal. This copies your dependencies into target/dependency if I remember correctly.
Using the maven-jar-plugin I add these dependencies to my final JAR's MANIFEST.MF as Class-Path entries using these options in the plugins configuration.
```
<classpathPrefix>dependency/</classpathPrefix>
<addClasspath>true</addClasspath>
```
So I'll have Class-Path entries like dependency/<artifactId>-<version>.jar, etc.
After this I'm using the maven-assembly-plugin to create a distribution ZIP which contains my final JAR and the whole dependency/ folder.
When I deploy my application I can run it like java -jar final.jar.

Primarily I've opted to use this solution, because in my project I use some Bouncy Castle JARs which JARs have some extravagant this and that inside their META-INF directory. When I used the maven-shade-plugin to create my final runnable JAR all hell broke loose and I got nasty that method couldn't be found and this signature is not quite right errors.

You should be doing something like this also. This Maven shade business is too shady to be useful (pun intended).

Here's a blog post on the whole process that I've tried to describe just above (thanks to baba), maybe it'll help someone in the future.

Question 2

After adding the Maven jFree dependency to my existing application, I'm not able to execute the created jar.

The only error message I get is the following:

java -jar target/com.company.product-1.0.0-SNAPSHOT.jar 
Error: Invalid or corrupt jarfile target/com. company.product-1.0.0-SNAPSHOT.jar

The complete pom.xml looks like this:

<?xml version="1.0" encoding="UTF-8"?>

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion
<groupId>com.mycompany</groupId>
<artifactId>com.mycompany.test</artifactId>
<name>${project.artifactId}</name>
<version>1.0.0-SNAPSHOT</version>

<properties>
    <java-version>1.7</java-version>
    <org.springframework-version>3.1.1.RELEASE</org.springframework-version>
    <org.springframework.data-version>1.0.3.RELEASE</org.springframework.data-version>
    <org.springframework.ws-version>2.0.4.RELEASE</org.springframework.ws-version>
    <org.springframework.ws.oxm-version>1.5.10</org.springframework.ws.oxm-version>
    <org.aspectj-version>1.6.12</org.aspectj-version>
    <org.slf4j-version>1.5.10</org.slf4j-version>
    <selenium-java-version>2.25.0</selenium-java-version>
    <browser-mob-version>2.0-beta-6</browser-mob-version>
</properties>

<dependencies>

    <!-- Hint A: If we delete this dependency it works -->
    <dependency>
        <groupId>org.jfree</groupId>
        <artifactId>jfreechart</artifactId>
        <version>1.0.14</version>
                    <exclusions>
            <exclusion>
                <artifactId>itext</artifactId>
                <groupId>com.lowagie</groupId>
            </exclusion>
        </exclusions>
    </dependency>

    <dependency>
        <groupId>com.itextpdf</groupId>
        <artifactId>itextpdf</artifactId>
        <version>5.3.2</version>
    </dependency>

    <dependency>
        <groupId>de.schlichtherle.io</groupId>
        <artifactId>truezip</artifactId>
        <version>6.6</version>
    </dependency>

    <dependency>
        <groupId>log4j</groupId>
        <artifactId>apache-log4j-extras</artifactId>
        <version>1.1</version>
    </dependency>

    <!-- Caching with ehcache -->
    <dependency>
        <groupId>net.sf.ehcache</groupId>
        <artifactId>ehcache</artifactId>
        <version>2.5.2</version>
        <type>pom</type>
    </dependency>

    <!-- Spring -->
    <dependency>
        <groupId>org.springframework</groupId>
        <artifactId>spring-context</artifactId>
        <version>${org.springframework-version}</version>
        <exclusions>
            <!-- Exclude Commons Logging in favor of SLF4j -->
            <exclusion>
                <groupId>commons-logging</groupId>
                <artifactId>commons-logging</artifactId>
            </exclusion>
        </exclusions>
    </dependency>
    <dependency>
        <groupId>org.springframework</groupId>
        <artifactId>spring-webmvc</artifactId>
        <version>${org.springframework-version}</version>
    </dependency>
    <dependency>
        <groupId>org.springframework</groupId>
        <artifactId>spring-core</artifactId>
        <version>${org.springframework-version}</version>
    </dependency>
    <dependency>
        <groupId>org.springframework</groupId>
        <artifactId>spring-beans</artifactId>
        <version>${org.springframework-version}</version>
    </dependency>
    <dependency>
        <groupId>org.springframework</groupId>
        <artifactId>spring-web</artifactId>
        <version>${org.springframework-version}</version>
    </dependency>
    <dependency>
        <groupId>org.springframework</groupId>
        <artifactId>spring-aop</artifactId>
        <version>${org.springframework-version}</version>
    </dependency>
    <dependency>
        <groupId>org.springframework</groupId>
        <artifactId>spring-context-support</artifactId>
        <version>${org.springframework-version}</version>
    </dependency>

    <dependency>
        <groupId>org.springframework</groupId>
        <artifactId>spring-tx</artifactId>
        <version>${org.springframework-version}</version>
    </dependency>

    <dependency>
        <groupId>org.springframework</groupId>
        <artifactId>spring-orm</artifactId>
        <version>${org.springframework-version}</version>
    </dependency>

    <dependency>
        <groupId>org.springframework.security</groupId>
        <artifactId>spring-security-core</artifactId>
        <version>3.0.3.RELEASE</version>
    </dependency>

    <dependency>
        <groupId>org.springframework.security</groupId>
        <artifactId>spring-security-web</artifactId>
        <version>3.0.3.RELEASE</version>
    </dependency>

    <dependency>
        <groupId>org.springframework.security</groupId>
        <artifactId>spring-security-config</artifactId>
        <version>3.0.3.RELEASE</version>
    </dependency>

    <dependency>
        <groupId>org.springframework</groupId>
        <artifactId>spring-test</artifactId>
        <version>${org.springframework-version}</version>
        <scope>test</scope>
    </dependency>
    <!-- Hibernate -->
    <dependency>
        <groupId>org.hibernate.java-persistence</groupId>
        <artifactId>jpa-api</artifactId>
        <version>2.0-cr-1</version>
    </dependency>

    <dependency>
        <groupId>org.hibernate</groupId>
        <artifactId>hibernate-entitymanager</artifactId>
        <version>3.5.1-Final</version>
    </dependency>

    <!-- Database Connectors (HSQL should be removed later) -->
    <dependency>
        <groupId>hsqldb</groupId>
        <artifactId>hsqldb</artifactId>
        <version>1.8.0.7</version>
    </dependency>

    <dependency>
        <groupId>mysql</groupId>
        <artifactId>mysql-connector-java</artifactId>
        <version>5.1.16</version>
    </dependency>

    <!-- AspectJ -->
    <dependency>
        <groupId>org.aspectj</groupId>
        <artifactId>aspectjrt</artifactId>
        <version>${org.aspectj-version}</version>
    </dependency>

    <!-- Logging -->
    <dependency>
        <groupId>org.slf4j</groupId>
        <artifactId>slf4j-api</artifactId>
        <version>${org.slf4j-version}</version>
    </dependency>

    <dependency>
        <groupId>log4j</groupId>
        <artifactId>log4j</artifactId>
        <version>1.2.16</version>
    </dependency>

    <!-- @Inject -->
    <dependency>
        <groupId>javax.inject</groupId>
        <artifactId>javax.inject</artifactId>
        <version>1</version>
    </dependency>

    <dependency>
        <groupId>javax.servlet</groupId>
        <artifactId>jstl</artifactId>
        <version>1.2</version>
    </dependency>

    <!-- Spring Data JPA dependencies -->
    <dependency>
        <groupId>org.springframework.data</groupId>
        <artifactId>spring-data-jpa</artifactId>
        <version>${org.springframework.data-version}</version>
    </dependency>
    <dependency>
        <groupId>javax.ws.rs</groupId>
        <artifactId>jsr311-api</artifactId>
        <version>1.1.1</version>
    </dependency>

    <dependency>
        <groupId>org.springframework.ws</groupId>
        <artifactId>spring-oxm</artifactId>
        <version>${org.springframework.ws.oxm-version}</version>
    </dependency>
    <dependency>
        <groupId>org.springframework.ws</groupId>
        <artifactId>spring-xml</artifactId>
        <version>${org.springframework.ws-version}</version>
    </dependency>
    <dependency>
        <groupId>org.springframework.ws</groupId>
        <artifactId>spring-ws-core</artifactId>
        <version>${org.springframework.ws-version}</version>
    </dependency>
    <dependency>
        <groupId>javax.xml</groupId>
        <artifactId>jaxb-impl</artifactId>
        <version>2.1</version>
    </dependency>
    <dependency>
        <groupId>javax.xml</groupId>
        <artifactId>jaxb-api</artifactId>
        <version>2.1</version>
    </dependency>

    <!-- Test -->
    <dependency>
        <groupId>junit</groupId>
        <artifactId>junit</artifactId>
        <version>4.7</version>
        <scope>test</scope>
    </dependency>

    <dependency>
        <groupId>org.mockito</groupId>
        <artifactId>mockito-all</artifactId>
        <version>1.9.0</version>
        <scope>test</scope>
    </dependency>

    <!-- Common Utils -->
    <dependency>
        <groupId>commons-lang</groupId>
        <artifactId>commons-lang</artifactId>
        <version>2.6</version>
    </dependency>

    <dependency>
        <groupId>commons-collections</groupId>
        <artifactId>commons-collections</artifactId>
        <version>3.2.1</version>
    </dependency>

    <dependency>
        <groupId>commons-cli</groupId>
        <artifactId>commons-cli</artifactId>
        <version>1.2</version>
    </dependency>       

    <!-- Selenium -->
    <dependency>
        <groupId>org.seleniumhq.selenium</groupId>
        <artifactId>selenium-java</artifactId>
        <version>${selenium-java-version}</version>
    </dependency>

    <dependency>
        <groupId>org.seleniumhq.selenium</groupId>
        <artifactId>selenium-firefox-driver</artifactId>
        <version>${selenium-java-version}</version>
    </dependency>

    <dependency>
        <groupId>org.seleniumhq.selenium</groupId>
        <artifactId>selenium-chrome-driver</artifactId>
        <version>${selenium-java-version}</version>
    </dependency>

    <!-- CSV Lib for Keyword Checker -->
    <dependency>
        <groupId>net.sf.opencsv</groupId>
        <artifactId>opencsv</artifactId>
        <version>2.0</version>
    </dependency>

    <!-- Google Places API -->
    <dependency>
        <groupId>com.google.api-client</groupId>
        <artifactId>google-api-client</artifactId>
        <version>1.10.3-beta</version>
        <exclusions>
            <exclusion>
                <artifactId>jackson-core-asl</artifactId>
                <groupId>org.codehaus.jackson</groupId>
            </exclusion>
        </exclusions>
    </dependency>
    <dependency>
        <groupId>com.google.api-client</groupId>
        <artifactId>google-api-client-appengine</artifactId>
        <version>1.10.3-beta</version>
    </dependency>

    <dependency>
        <groupId>org.codehaus.jackson</groupId>
        <artifactId>jackson-mapper-asl</artifactId>
        <version>1.9.9</version>
    </dependency>

    <!-- Google Geocode -->
    <dependency>
        <groupId>com.google.code.geocoder-java</groupId>
        <artifactId>geocoder-java</artifactId>
        <version>0.9</version>
    </dependency>

    <dependency>
        <groupId>com.googlecode.json-simple</groupId>
        <artifactId>json-simple</artifactId>
        <version>1.1</version>
    </dependency>

    <dependency>
        <groupId>com.googlecode.json-simple</groupId>
        <artifactId>json-simple</artifactId>
        <version>1.1</version>
    </dependency>

    <dependency>
        <groupId>net.sf.jgrapht</groupId>
        <artifactId>jgrapht</artifactId>
        <version>0.8.3</version>
    </dependency>

    <dependency>
        <groupId>jgraph</groupId>
        <artifactId>jgraph</artifactId>
        <version>5.13.0.0</version>
    </dependency>

    <!-- Apache Http Client -->
    <dependency>
        <groupId>org.apache.httpcomponents</groupId>
        <artifactId>httpclient</artifactId>
        <version>4.2.1</version>
    </dependency>

    <!-- Amazon web services client -->
    <dependency>
        <groupId>com.amazonaws</groupId>
        <artifactId>aws-java-sdk</artifactId>
        <version>1.2.15</version>
        <exclusions>
            <exclusion>
                <artifactId>jackson-core-asl</artifactId>
                <groupId>org.codehaus.jackson</groupId>
            </exclusion>
        </exclusions>
    </dependency>

    <!-- Docx4j - reading excel files  -->
    <dependency>
        <groupId>org.docx4j</groupId>
        <artifactId>docx4j</artifactId>
        <version>2.8.0</version>
    </dependency>

    <!-- Browser Mob Proxy -->

    <dependency>
        <groupId>biz.neustar</groupId>
        <artifactId>browsermob-proxy</artifactId>
        <version>${browser-mob-version}</version>
        <exclusions>
            <exclusion>
                <groupId>org.seleniumhq.selenium</groupId>
                <artifactId>selenium-api</artifactId>
            </exclusion>
            <exclusion>
                <artifactId>icu4j</artifactId>
                <groupId>com.ibm.icu</groupId>
            </exclusion>
            <exclusion>
                <artifactId>jackson-mapper-asl</artifactId>
                <groupId>org.codehaus.jackson</groupId>
            </exclusion>
            <exclusion>
                <artifactId>jackson-core-asl</artifactId>
                <groupId>org.codehaus.jackson</groupId>
            </exclusion>
            <exclusion>
                <groupId>org.slf4j</groupId>
                <artifactId>slf4j-jdk14</artifactId>
            </exclusion>
        </exclusions>
    </dependency>

    <!-- Hint B: If we copy this Apache POI dependencies to the top, it works -->
    <!-- Apache POI - for reading xlsx files -->
    <dependency>
        <groupId>org.apache.poi</groupId>
        <artifactId>poi</artifactId>
        <version>3.8</version>
    </dependency>

    <dependency>
        <groupId>org.apache.poi</groupId>
        <artifactId>poi-ooxml</artifactId>
        <version>3.8</version>
    </dependency>

    <dependency>
        <groupId>org.apache.xmlbeans</groupId>
        <artifactId>xmlbeans</artifactId>
        <version>2.5.0</version>
    </dependency>

    <dependency>
        <groupId>org.apache.poi</groupId>
        <artifactId>ooxml-schemas</artifactId>
        <version>1.1</version>
    </dependency>

</dependencies>

<build>
    <plugins>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-compiler-plugin</artifactId>
            <configuration>
                <source>${java-version}</source>
                <target>${java-version}</target>
            </configuration>
        </plugin>

        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-shade-plugin</artifactId>
            <version>1.7</version>
            <executions>
                <execution>
                    <phase>package</phase>
                    <goals>
                        <goal>shade</goal>
                    </goals>
                    <configuration>
                        <transformers>

                            <!-- must be SURE to do this with both spring.handlers and spring.schemas. 
                                otherwise you won't be able to use them in the spring config files. -->
                            <transformer implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
                                <resource>META-INF/spring.handlers</resource>
                            </transformer>

                            <transformer implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
                                <resource>META-INF/spring.schemas</resource>
                            </transformer>

                            <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                                <mainClass>com.mycompany.test.Start</mainClass>
                            </transformer>
                        </transformers>

                        <filters>
                          <filter>
                            <artifact>bouncycastle:bcprov-jdk15</artifact>
                            <excludes>
                              <exclude>META-INF/BCKEY.DSA</exclude>
                              <exclude>META-INF/BCKEY.SF</exclude>
                              <exclude>META-INF/MANIFEST.MF</exclude>
                            </excludes>
                          </filter>
                        </filters>                          

                    </configuration>
                </execution>
            </executions>
        </plugin>

    </plugins>
</build>

I don't get any other debug information if I execute the jar.

Is there any option to validate the jarfile? Or having something like java -verbose...?

Steps to reproduce:

create a new folder
Copy the Pom.xml from above into the folder.
Save the small `public static void main(...). into src/main/java/com/mycompany/test/Start.java
execute mvn clean package
execute java -jar target/com.mycompany.test-1.0.0-SNAPSHOT.jar

Attachment 1:

package com.mycompany.test;


public class Start
{
    public static void main(final String[] args) 
    {
        System.out.println("If you are able to get this printed with java -jar you made it. Thanks a lot! :)");
    }
}

Edit 1:

I started to delete some dependencies to identify the problem. But I haven't got a clear understanding what goes wrong.

Now I figured out (See hint B in the pom.xml) that moving the Apache POI dependencies to the top will solve the problem. But still I don't know how to why and what is the problem.

Invalid or corrupt JAR File built by Maven shade plugin

Answer

Related questions