Maven build gets connection reset when downloading artifacts

java maven jenkins amazon-ec2 maven-central
Bubbad picture Bubbad · Feb 3, 2017 · Viewed 12.3k times · Source

We have a maven build that runs in jenkins in amazon ec2 instances. The total build takes 20-30min and roughly every other run it fails because it cant download artifacts from maven central. This is the failing part of the log:

[DEBUG] Lifecycle default -> [validate, initialize, generate-sources, process-sources, generate-resources, process-resources, compile, process-classes, generate-test-sources, process-test-sources, generate-test-resources, process-test-resources, test-compile, process-test-classes, test, prepare-package, package, pre-integration-test, integration-test, post-integration-test, verify, install, deploy]
[DEBUG] Lifecycle clean -> [pre-clean, clean, post-clean]
[DEBUG] Lifecycle site -> [pre-site, site, post-site, site-deploy]
[DEBUG] Using transporter WagonTransporter with priority -1.0 for https://repo.maven.apache.org/maven2
[DEBUG] Using connector BasicRepositoryConnector with priority 0.0 for https://repo.maven.apache.org/maven2
Downloading: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-failsafe-plugin/2.18.1/maven-failsafe-plugin-2.18.1.pom
Feb 03, 2017 3:39:00 PM org.apache.http.impl.execchain.RetryExec execute
INFO: I/O exception (java.net.SocketException) caught when processing request to {s}->https://repo.maven.apache.org:443: Connection reset
Feb 03, 2017 3:39:00 PM org.apache.http.impl.execchain.RetryExec execute
INFO: Retrying request to {s}->https://repo.maven.apache.org:443
Feb 03, 2017 3:39:00 PM org.apache.http.impl.execchain.RetryExec execute
INFO: I/O exception (java.net.SocketException) caught when processing request to {s}->https://repo.maven.apache.org:443: Connection reset
Feb 03, 2017 3:39:00 PM org.apache.http.impl.execchain.RetryExec execute
INFO: Retrying request to {s}->https://repo.maven.apache.org:443
Feb 03, 2017 3:39:00 PM org.apache.http.impl.execchain.RetryExec execute
INFO: I/O exception (java.net.SocketException) caught when processing request to {s}->https://repo.maven.apache.org:443: Connection reset
Feb 03, 2017 3:39:00 PM org.apache.http.impl.execchain.RetryExec execute
INFO: Retrying request to {s}->https://repo.maven.apache.org:443

The interesting thing is that it runs fine for 20min downloading from maven central and our own repos, but then fails on the exact same download every other time. This makes me think it's not due to connection problems, as then it would fail on other downloads randomly.

After googling I found that it could be because of Ipv4 vs Ipv6 issues, so I tried setting java.net.prefetIPv4stack=true in the build, but that didn't help.

The command to run mvn is:

mvn -X -e clean deploy -U -Djava.net.preferIPv4Stack=true

I've also tried to increase -Xmx in both MAVEN_OPTS and JAVA_OPTS (I read somewhere that maven needs to be able to keep all dependencies in RAM, not sure if true tho), but without success.

Do you have any other ideas that I can try?

Edit: Added stacktrace

[ERROR] Plugin org.apache.maven.plugins:maven-failsafe-plugin:2.18.1 or one of its dependencies could not be resolved: Failed to read artifact descriptor for org.apache.maven.plugins:maven-failsafe-plugin:jar:2.18.1: Could not transfer artifact org.apache.maven.plugins:maven-failsafe-plugin:pom:2.18.1 from/to central (https://repo.maven.apache.org/maven2): Connection reset -> [Help 1]
org.apache.maven.plugin.PluginResolutionException: Plugin org.apache.maven.plugins:maven-failsafe-plugin:2.18.1 or one of its dependencies could not be resolved: Failed to read artifact descriptor for org.apache.maven.plugins:maven-failsafe-plugin:jar:2.18.1
    at org.apache.maven.plugin.internal.DefaultPluginDependenciesResolver.resolve(DefaultPluginDependenciesResolver.java:117)
    at org.apache.maven.plugin.internal.DefaultMavenPluginManager.getPluginDescriptor(DefaultMavenPluginManager.java:179)
    at org.apache.maven.plugin.internal.DefaultMavenPluginManager.getMojoDescriptor(DefaultMavenPluginManager.java:284)
    at org.apache.maven.plugin.DefaultBuildPluginManager.getMojoDescriptor(DefaultBuildPluginManager.java:241)
    at org.apache.maven.lifecycle.internal.DefaultLifecycleMappingDelegate.calculateLifecycleMappings(DefaultLifecycleMappingDelegate.java:110)
    at org.apache.maven.lifecycle.internal.DefaultLifecycleExecutionPlanCalculator.calculateLifecycleMappings(DefaultLifecycleExecutionPlanCalculator.java:266)
    at org.apache.maven.lifecycle.internal.DefaultLifecycleExecutionPlanCalculator.calculateMojoExecutions(DefaultLifecycleExecutionPlanCalculator.java:217)
    at org.apache.maven.lifecycle.internal.DefaultLifecycleExecutionPlanCalculator.calculateExecutionPlan(DefaultLifecycleExecutionPlanCalculator.java:127)
    at org.apache.maven.lifecycle.internal.DefaultLifecycleExecutionPlanCalculator.calculateExecutionPlan(DefaultLifecycleExecutionPlanCalculator.java:145)
    at org.apache.maven.lifecycle.internal.builder.BuilderCommon.resolveBuildPlan(BuilderCommon.java:96)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:109)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
    at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)
    at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
    at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
    at org.apache.maven.cli.MavenCli.execute(MavenCli.java:863)
    at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288)
    at org.apache.maven.cli.MavenCli.main(MavenCli.java:199)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
Caused by: org.eclipse.aether.resolution.ArtifactDescriptorException: Failed to read artifact descriptor for org.apache.maven.plugins:maven-failsafe-plugin:jar:2.18.1
    at org.apache.maven.repository.internal.DefaultArtifactDescriptorReader.loadPom(DefaultArtifactDescriptorReader.java:282)
    at org.apache.maven.repository.internal.DefaultArtifactDescriptorReader.readArtifactDescriptor(DefaultArtifactDescriptorReader.java:198)
    at org.eclipse.aether.internal.impl.DefaultRepositorySystem.readArtifactDescriptor(DefaultRepositorySystem.java:287)
    at org.apache.maven.plugin.internal.DefaultPluginDependenciesResolver.resolve(DefaultPluginDependenciesResolver.java:103)
    ... 27 more
Caused by: org.eclipse.aether.resolution.ArtifactResolutionException: Could not transfer artifact org.apache.maven.plugins:maven-failsafe-plugin:pom:2.18.1 from/to central (https://repo.maven.apache.org/maven2): Connection reset
    at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:444)
    at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifacts(DefaultArtifactResolver.java:246)
    at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifact(DefaultArtifactResolver.java:223)
    at org.apache.maven.repository.internal.DefaultArtifactDescriptorReader.loadPom(DefaultArtifactDescriptorReader.java:267)
    ... 30 more
Caused by: org.eclipse.aether.transfer.ArtifactTransferException: Could not transfer artifact org.apache.maven.plugins:maven-failsafe-plugin:pom:2.18.1 from/to central (https://repo.maven.apache.org/maven2): Connection reset
    at org.eclipse.aether.connector.basic.ArtifactTransportListener.transferFailed(ArtifactTransportListener.java:43)
    at org.eclipse.aether.connector.basic.BasicRepositoryConnector$TaskRunner.run(BasicRepositoryConnector.java:355)
    at org.eclipse.aether.util.concurrency.RunnableErrorForwarder$1.run(RunnableErrorForwarder.java:67)
    at org.eclipse.aether.connector.basic.BasicRepositoryConnector$DirectExecutor.execute(BasicRepositoryConnector.java:581)
    at org.eclipse.aether.connector.basic.BasicRepositoryConnector.get(BasicRepositoryConnector.java:249)
    at org.eclipse.aether.internal.impl.DefaultArtifactResolver.performDownloads(DefaultArtifactResolver.java:520)
    at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:421)
    ... 33 more
Caused by: org.apache.maven.wagon.TransferFailedException: Connection reset
    at org.apache.maven.wagon.providers.http.AbstractHttpClientWagon.fillInputData(AbstractHttpClientWagon.java:1066)
    at org.apache.maven.wagon.providers.http.AbstractHttpClientWagon.fillInputData(AbstractHttpClientWagon.java:960)
    at org.apache.maven.wagon.StreamWagon.getInputStream(StreamWagon.java:116)
    at org.apache.maven.wagon.StreamWagon.getIfNewer(StreamWagon.java:88)
    at org.apache.maven.wagon.StreamWagon.get(StreamWagon.java:61)
    at org.eclipse.aether.transport.wagon.WagonTransporter$GetTaskRunner.run(WagonTransporter.java:560)
    at org.eclipse.aether.transport.wagon.WagonTransporter.execute(WagonTransporter.java:427)
    at org.eclipse.aether.transport.wagon.WagonTransporter.get(WagonTransporter.java:404)
    at org.eclipse.aether.connector.basic.BasicRepositoryConnector$GetTaskRunner.runTask(BasicRepositoryConnector.java:447)
    at org.eclipse.aether.connector.basic.BasicRepositoryConnector$TaskRunner.run(BasicRepositoryConnector.java:350)
    ... 38 more
Caused by: java.net.SocketException: Connection reset
    at java.net.SocketInputStream.read(SocketInputStream.java:209)
    at java.net.SocketInputStream.read(SocketInputStream.java:141)
    at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
    at sun.security.ssl.InputRecord.read(InputRecord.java:503)
    at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
    at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:930)
    at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
    at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:136)
    at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:152)
    at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:270)
    at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
    at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
    at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:260)
    at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:161)
    at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:153)
    at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:271)
    at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123)
    at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:254)
    at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:195)
    at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:86)
    at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:108)
    at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
    at org.apache.maven.wagon.providers.http.AbstractHttpClientWagon.execute(AbstractHttpClientWagon.java:832)
    at org.apache.maven.wagon.providers.http.AbstractHttpClientWagon.fillInputData(AbstractHttpClientWagon.java:983)
    ... 47 more
[ERROR] 
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginResolutionException

Answer

Barchetta picture Barchetta · Jun 26, 2019

TL;DR

Try upgrading to Maven 3.6.1. It has a new version of Wagon HTTP with improved retry behavior. This fixed the problem for us.

Background

Our builds ran fine for months (in ec2). Then one day they started sporadically failing downloading Maven artifacts (from Maven central) with Connection reset. We never got to the bottom of it. Maybe something in ec2 changed. Maybe something in our build changed. Dunno.

The bottom line is that networks are unreliable, and Maven (or more specifically Wagon) was not very resilient to connection failures. Thankfully it looks like they've improved that.

Maven 3.6.1 includes a new version of Wagon that supports configureable retry behavior. By default it appears to have a retry count of 3 -- which was good enough for us. Some details:

Related questions

Maven dependencies are failing with a 501 error

Recently Maven build jobs running in Jenkins are failing with the below exception saying that they couldn't pull dependencies from Maven Central and should use HTTPS. I'm not sure how to change the requests from HTTP to HTTPS. Could someone …

Read More
Sonarqube: Missing blame information for the following files

I am getting the warning Missing blame information for the following files during analysis by SonarQube. [INFO] [22:19:57.714] Sensor SCM Sensor [INFO] [22:19:57.715] SCM provider for this project is: git [INFO] [22:19:57.715] 48 files to be analyzed [INFO] [22:19:58.448] 0/48 files analyzed [WARN] [22:19:58.448] Missing blame information …

Read More
Jenkins auto deploy tomcat 7

We're trying to deploy a war file with Jenkins, but nothing seems to happen. The project is built successfully, and we're using Jenkins deploy plugin. It is configured with the following options: Post steps are set to "run regardless of …

Read More