Programmatically changing Hystrix properties

Question

I have a circuit breaker set up that I would like to change parameters for runtime. Things like threads and timeout needs to be tuned at customer site.

I create a HystrixCommandProperties.Setter like this:

HystrixCommandProperties.Setter hystrixProps = 
    HystrixCommandProperties.defaultSetter()
        .withCircuitBreakerSleepWindowInMilliseconds(myconf.sleepWindow);
HystrixThreadPoolProperties.Setter threadPoolSettings = 
    HystrixThreadPoolProperties.Setter()
        .withCoreSize(myconf.threadPoolSize);

new MyCommand(HystrixCommand.Setter.withGroupKey("mygroup")
    .andCommandPropertiesDefaults(hystrixProps)
    .andThreadPoolPropertiesDefaults(threadPoolSettings));

MyCommand implements standard HystrixCommand and calls super(hystrixProps).

This works the first time, but when I try to change the properties at runtime (same group name) nothing happens. Is there another way to programmatically change this?

I don't want to go through the property files or specifying an URL to Archaius.

There are also answers that tells me to go through Archaius with ConfigurationManager.getConfigInstance().setProperty("...") . But surely there has to be a way that is similar to the original setters I create? Doing it completely different because it's the second time around just feels awkward.

Answer 1

Late answer, but today i struggled with the same thing and found a way.

The way the default property manager is implemented is that it uses a cache of HystrixCommandProperties based on the name of the command you run. On the first use of the command, it caches what it gets out of the HystrixCommandProperties.Setter passed to the Command's constructor and that's it.

However, using the custom HystrixPropertiesStrategy as a Plugin you can override the cache key (and hence force Hystrix to re-evaluate the Setter passed to the new Command instance, because the cache key is new, so it thinks it's a new Command).

The code would then look similar to this:

public HystrixCommandFactory(....) {
    HystrixPlugins.getInstance().registerPropertiesStrategy(new HystrixPropertiesStrategyWithReloadableCache());
    updateHystrixSettings();        
}

//configurable attributes
private volatile int commandTimeoutMillis;
private volatile long lastSettingsUpdatedTimestamp;
private volatile HystrixCommand.Setter setterForNewCommands;

private void updateHystrixSettings() {
    lastSettingsUpdatedTimestamp = LocalDateTime.now().toDateTime().getMillis();
    HystrixCommandProperties.Setter propertiesSetter = HystrixCommandProperties.Setter()
        .withExecutionTimeoutInMilliseconds(commandTimeoutMillis)
        .withExecutionTimeoutEnabled(true);

    this.setterForNewCommands = HystrixCommand.Setter
        .withGroupKey(HystrixCommandGroupKey.Factory.asKey(GROUP_NAME))
        .andCommandPropertiesDefaults(propertiesSetter);

}

public void setCommandTimeoutMillis(int commandTimeoutMillis) {     
    this.commandTimeoutMillis = commandTimeoutMillis;
    updateHystrixSettings();        
}

private class HystrixPropertiesStrategyWithReloadableCache extends HystrixPropertiesStrategy {

    @Override
    public String getCommandPropertiesCacheKey(HystrixCommandKey commandKey, HystrixCommandProperties.Setter builder) {
        return String.format("%s-%d", commandKey.name(), lastSettingsUpdatedTimestamp);
    }
}

Alternatively, you could always return null from the getCommandPropertiesCacheKey method (that completely turns off caching), but then you have the overhead of Hystrix having to reconstruct the HystrixCommandProperties each time a Command is called

PS: Be sure to use proper thread synchronization for reading and updating these properties, because those will be called from different threads. I omitted that in this sample for simplicity, but I actually use a ReentrantReadWriteLock in my code to guard accesses to these variables