Speeding up sklearn logistic regression
I have a model I'm trying to build using LogisticRegression in sklearn that has a couple thousand features and approximately 60,000 samples. I'm trying to fit the model and it's been running for about 10 mins now. The machine I'm running it on has gigabytes of RAM and several cores at its disposal and I was wondering if there is any way to speed the process up
EDIT The machine has 24 cores and here is the output of top to give an idea of memory
Processes: 94 total, 8 running, 3 stuck, 83 sleeping, 583 threads 20:10:19
Load Avg: 1.49, 1.25, 1.19 CPU usage: 4.34% user, 0.68% sys, 94.96% idle
SharedLibs: 1552K resident, 0B data, 0B linkedit.
MemRegions: 51959 total, 53G resident, 46M private, 676M shared.
PhysMem: 3804M wired, 57G active, 1042M inactive, 62G used, 34G free.
VM: 350G vsize, 1092M framework vsize, 52556024(0) pageins, 85585722(0) pageouts
Networks: packets: 172806918/25G in, 27748484/7668M out.
Disks: 14763149/306G read, 26390627/1017G written.
I'm trying to train the model with the following
classifier = LogisticRegression(C=1.0, class_weight = 'auto')
classifier.fit(train, response)
train has rows that are approximately 3000 long (all floating point) and each row in response is either 0 or 1. I have approximately 50,000 observations
Answer
UPDATE - 2017:
In current version of scikit-learn, LogisticRegression() now has n_jobs parameter to utilize multiple cores.
However, the actual text of the user guide suggests that multiple cores are still only being utilized during the second half of the computation. As of this update, the revised user guide for LogisticRegression now says that njobs chooses the "Number of CPU cores used during the cross-validation loop" whereas the other two items cited in the original response, RandomForestClassifier() and RandomForestRegressor(), both state that njobs specifies "The number of jobs to run in parallel for both fit and predict". In other words, the deliberate contrast in phrasing here seems to be pointing out that the njobs parameter in LogisticRegression(), while now implemented, is not really implemented as completely, or in the same way, as in the other two cases.
Thus, while it may now be possible to speed up LogisticRegression() somewhat by using multiple cores, my guess is that it probably won't be very linear in proportion to the number of cores used, as it sounds like the initial "fit" step (the first half of the algorithm) may not lend itself well to parallelization.
Original Answer:
To my eye, it looks like the major issue here isn't memory, it's that you are only using one core. According to top, you are loading the system at 4.34%. If your logistic regression process is monopolizing 1 core out of 24, then that comes out to 100/24 = 4.167%. Presumably the remaining 0.17% accounts for whatever other processes you are also running on the machine, and they are allowed to take up an extra 0.17% because they are being scheduled by the system to run in parallel on a 2nd, different core.
If you follow the links below and look at the scikit-learn API, you'll see that some of the ensemble methods such as RandomForestClassifier() or RandomForestRegressor() have an input parameter called n_jobs which directly controls the number of cores on which the package will attempt to run in parallel. The class that you are using, LogisticRegression() doesn't define this input. The designers of scikit-learn seem to have created an interface which is generally pretty consistent between classes, so if a particular input parameter is not defined for a given class, it probably means that the developers simply could not figure out a way to implement the option in a meaningful way for that class. It may be the case that the logistic regression algorithm simply doesn't lend itself well to parallelization; i.e., the potential speedup that could have been achieved just wasn't good enough to have justified implementing it with a parallel architecture.
Assuming that this is the case, then no, there's not much you can do to make your code go faster. 24 cores doesn't help you if the underlying library functions simply weren't designed to be able to take advantage of them.