Cannot get Jedis connection; Could not get a resource from the pool
I am running a batch job for every 5 minutes and I don't wanna other nodes to run the same job hence I am using Jedis lock to lock an object for 5 minutes. So that other node won't get the lock if they try to run the same job. Job started after acquiring the lock and when I try to read it from Redis I am getting the following exception saying
'Caused by: redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool
at redis.clients.util.Pool.getResource(Pool.java:53)
at redis.clients.jedis.JedisPool.getResource(JedisPool.java:226)
at redis.clients.jedis.JedisPool.getResource(JedisPool.java:16)
at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.fetchJedisConnector(JedisConnectionFactory.java:194)
... 40 more
Caused by: redis.clients.jedis.exceptions.JedisConnectionException: java.net.SocketTimeoutException: Read timed out
at redis.clients.jedis.Connection.disconnect(Connection.java:224)
at redis.clients.jedis.BinaryClient.disconnect(BinaryClient.java:941)
at redis.clients.jedis.Connection.close(Connection.java:214)
at redis.clients.jedis.BinaryClient.close(BinaryClient.java:947)
at redis.clients.jedis.Jedis.close(Jedis.java:3412)
at redis.clients.jedis.JedisFactory.makeObject(JedisFactory.java:117)
at org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:836)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:434)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:361)
at redis.clients.util.Pool.getResource(Pool.java:49)'
This is what the code I have
@Bean
public Jedis getJedis()
{
Jedis jedis = new Jedis(this.redisHost, nteger.valueOf(this.redisPort));
jedis.auth(this.redisPassword);
return jedis;
}
spring-boot Application.properties file
# DATA REDIS
spring.data.redis.repositories.enabled=true
# REDIS (RedisProperties)
spring.redis.host=10.160.49.22
spring.redis.password=qweqewqw
spring.redis.ssl=true
#spring.redis.pool.max-active=10
#spring.redis.pool.max-idle=10
# spring.redis.pool.max-wait=30000
spring.redis.port=6379
Job executes the following code at the start to get the lock
JedisLock jedisLock = new JedisLock(jedis, getLockName(), getTimeInMillis());
jedisLoc.acquire()
After that redis repository class is trying to read values for a particular pattern..
List<String> hotelCodes = redisTemplate.execute(new RedisCallback<List<String>>() {
/**
* Gets called by {@link RedisTemplate} with an active Redis connection. Does not need to care about activating or
* closing the connection or handling exceptions.
*
* @param connection active Redis connection
* @return a result object or {@code null} if none
* @throws DataAccessException
*/
@Override
public List<String> doInRedis(RedisConnection connection) throws DataAccessException {
ScanOptions options = ScanOptions.scanOptions().match(pattern).count(1).build();
Cursor<Map.Entry<byte[], byte[]>> entries = connection.hScan(HASH_KEY.getBytes(), options);
List<String> result = new ArrayList<>();
if (entries != null)
while (entries.hasNext()) {
Map.Entry<byte[], byte[]> entry = entries.next();
byte[] actualValue = entry.getValue();
result.add(new String(actualValue));
}
return result;
}
});
return hotelCodes;
Then this is the complete exception I see in log.
org.springframework.data.redis.RedisConnectionFailureException: Cannot get Jedis connection; nested exception is redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool
at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.fetchJedisConnector(JedisConnectionFactory.java:204)
at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.getConnection(JedisConnectionFactory.java:348)
at org.springframework.data.redis.core.RedisConnectionUtils.doGetConnection(RedisConnectionUtils.java:129)
at org.springframework.data.redis.core.RedisConnectionUtils.getConnection(RedisConnectionUtils.java:92)
at org.springframework.data.redis.core.RedisConnectionUtils.getConnection(RedisConnectionUtils.java:79)
at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:194)
at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:169)
at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:157)
at com.hyatt.pms.jobs.dao.impl.HeartbeatRepositoryImpl.findAll(HeartbeatRepositoryImpl.java:62)
at com.hyatt.pms.jobs.dao.impl.HeartbeatRepositoryImpl$$FastClassBySpringCGLIB$$e3fe6169.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:136)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673)
at com.hyatt.pms.jobs.dao.impl.HeartbeatRepositoryImpl$$EnhancerBySpringCGLIB$$68719252.findAll(<generated>)
at com.hyatt.pms.jobs.processors.HeartbeatTestProcessor.processCluster(HeartbeatTestProcessor.java:54)
at com.hyatt.pms.jobs.processors.TestProcessor.process(TestProcessor.java:61)
at com.hyatt.pms.jobs.processors.HeartbeatTestProcessor.process(HeartbeatTestProcessor.java:39)
at com.hyatt.pms.jobs.processors.HeartbeatTestProcessor$$FastClassBySpringCGLIB$$99fdfbdc.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:669)
at com.hyatt.pms.jobs.processors.HeartbeatTestProcessor$$EnhancerBySpringCGLIB$$860eb7e4.process(<generated>)
at com.hyatt.pms.jobs.domain.jobs.HeartbeatJob.runHealthCheck(HeartbeatJob.java:34)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:65)
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
at org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:81)
at java.util.concurrent.Executors$RunnableAdapter.call$$$capture(Executors.java:511)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java)
at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
at java.util.concurrent.FutureTask.run(FutureTask.java)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool
at redis.clients.util.Pool.getResource(Pool.java:53)
at redis.clients.jedis.JedisPool.getResource(JedisPool.java:226)
at redis.clients.jedis.JedisPool.getResource(JedisPool.java:16)
at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.fetchJedisConnector(JedisConnectionFactory.java:194)
... 40 more
Caused by: redis.clients.jedis.exceptions.JedisConnectionException: java.net.SocketTimeoutException: Read timed out
at redis.clients.jedis.Connection.disconnect(Connection.java:224)
at redis.clients.jedis.BinaryClient.disconnect(BinaryClient.java:941)
at redis.clients.jedis.Connection.close(Connection.java:214)
at redis.clients.jedis.BinaryClient.close(BinaryClient.java:947)
at redis.clients.jedis.Jedis.close(Jedis.java:3412)
at redis.clients.jedis.JedisFactory.makeObject(JedisFactory.java:117)
at org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:836)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:434)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:361)
at redis.clients.util.Pool.getResource(Pool.java:49)
... 43 more
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
at sun.security.ssl.InputRecord.read(InputRecord.java:503)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983)
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1385)
at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:757)
at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:123)
at redis.clients.util.RedisOutputStream.flushBuffer(RedisOutputStream.java:52)
at redis.clients.util.RedisOutputStream.flush(RedisOutputStream.java:216)
at redis.clients.jedis.Connection.disconnect(Connection.java:220)
... 52 more
Anyone knows why I get this exception and how to fix it?
Answer
1.Try to use Jedis connection pool to get Jedis instance instead of using new Jedis(), for that you will have to configure JedisPool in config
Since you are using it as bean, you should use JedisPool bean and then get Jedis from it wherever you are supposed to perform an operation and close it after finishing.
2.Otherwise use spring's redisTemplate with config as below
@Bean
public JedisConnectionFactory redisConnectionFactory() {
JedisConnectionFactory factory = new JedisConnectionFactory();
factory.setHostName(redisHostName);
factory.setPort(redisPort);
factory.setUsePool(true);
return factory;
}
@Bean
public RedisTemplate<Object, Object> redisTemplate(RedisConnectionFactory cf) {
RedisTemplate<Object, Object> redisTemplate = new RedisTemplate<Object, Object>();
redisTemplate.setDefaultSerializer(new StringRedisSerializer());
redisTemplate.setConnectionFactory(cf);
return redisTemplate;
}