Puppetserver service fails to start

Question

I have a Vagrant CentOS VM running with ps.memory = 2048 RAM allocated.

When I try to start the puppetserver service:

$ puppet --version
4.4.0
$ sudo puppet resource service puppetserver ensure=running
Error: Could not start Service[puppetserver]: Execution of '/bin/systemctl start puppetserver' returned 1: Job for puppetserver.service failed. See 'systemctl status puppetserver.service' and 'journalctl -xn' for details.
Error: /Service[puppetserver]/ensure: change from stopped to running failed: Could not start Service[puppetserver]: Execution of '/bin/systemctl start puppetserver' returned 1: Job for puppetserver.service failed. See 'systemctl status puppetserver.service' and 'journalctl -xn' for details.
service { 'puppetserver':
  ensure => 'stopped',
}
$ journalctl -xn
No journal files were found.
$ systemctl status puppetserver.service
puppetserver.service - puppetserver Service
   Loaded: loaded (/usr/lib/systemd/system/puppetserver.service; disabled)
  Process: 4708 ExecStartPre=/usr/bin/install --directory --owner=puppet --group=puppet --mode=775 /var/run/puppetlabs/puppetserver (code=exited, status=0/SUCCESS)
 Main PID: 4709 (java);         : 4710 (bash)
   CGroup: /system.slice/puppetserver.service
           ├─4709 /usr/bin/java -Xms1g -Xmx1g -XX:MaxPermSize=1g -XX:OnOutOfMemoryError=kill -9 %p -Djava.security.egd=/...
           └─control
             ├─4710 /bin/bash /opt/puppetlabs/server/apps/puppetserver/ezbake-functions.sh wait_for_app
             └─4755 sleep 1

My JAVA_ARGS from /etc/sysconfig/puppetserver:

JAVA_ARGS="-Xms1g -Xmx1g -XX:MaxPermSize=1g"

As requested, the puppetserver.service file:

$ cat /usr/lib/systemd/system/puppetserver.service
[Unit]
Description=puppetserver Service
After=syslog.target network.target

[Service]
Type=simple
EnvironmentFile=/etc/sysconfig/puppetserver
User=puppet
TimeoutStartSec=120
TimeoutStopSec=60
Restart=on-failure
StartLimitBurst=5

PermissionsStartOnly=true
ExecStartPre=/usr/bin/install --directory --owner=puppet --group=puppet --mode=775 /var/run/puppetlabs/puppetserver

ExecStart=/usr/bin/java $JAVA_ARGS \
          '-XX:OnOutOfMemoryError=kill -9 %%p' \
          -Djava.security.egd=/dev/urandom \
          -cp "${INSTALL_DIR}/puppet-server-release.jar" clojure.main \
          -m puppetlabs.trapperkeeper.main \
          --config "${CONFIG}" \
          -b "${BOOTSTRAP_CONFIG}" $@

KillMode=process

ExecStartPost=/bin/bash "${INSTALL_DIR}/ezbake-functions.sh" wait_for_app

SuccessExitStatus=143

StandardOutput=syslog

[Install]
WantedBy=multi-user.target

An attempt at running the ExecStartPost command by hand:

$ /usr/bin/java -Xms1g -Xmx1g -XX:MaxPermSize=1g -XX:OnOutOfMemoryError='kill -9 %%p' -Djava.security.egd=/dev/urandom -cp /opt/puppetlabs/server/apps/puppetserver/puppet-server-release.jar clojure.main -m puppetlabs.trapperkeeper.main --config /etc/puppetlabs/puppetserver/conf.d -b /etc/puppetlabs/puppetserver/bootstrap.cfg

RuntimeError: Got 2 failure(s) while initializing: File[/var/log/puppetlabs/puppetserver]: change from 0700 to 0750 failed: failed to set mode 0700 on /var/log/puppetlabs/puppetserver: Operation not permitted - No message available; File[/var/run/puppetlabs/puppetserver]: change from 0775 to 0755 failed: failed to set mode 0775 on /var/run/puppetlabs/puppetserver: Operation not permitted - No message available

So I tried again, but this time I changed some directory permissions, but still similar error (which doesn't make sense given I just changed the mode?):

$ sudo chown -R vagrant:vagrant /var/run/puppetlabs/
$ sudo chown -R vagrant:vagrant /var/log/puppetlabs/
$ sudo chmod -R 0755 /var/run/puppetlabs/

$ /usr/bin/java -Xms1g -Xmx1g -XX:MaxPermSize=1g -XX:OnOutOfMemoryError='kill -9 %%p' -Djava.security.egd=/dev/urandom -cp /opt/puppetlabs/server/apps/puppetserver/puppet-server-release.jar clojure.main -m puppetlabs.trapperkeeper.main --config /etc/puppetlabs/puppetserver/conf.d -b /etc/puppetlabs/puppetserver/bootstrap.cfg

OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=1g; support was removed in 8.0
RuntimeError: Got 1 failure(s) while initializing: File[/var/run/puppetlabs/puppetserver]: change from 0775 to 0755 failed: failed to set mode 0775 on /var/run/puppetlabs/puppetserver: Operation not permitted - No message available
                use at /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/settings.rb:1007

What could be the issue?

Answer 1

Are you certain that it’s an OnOutOfMemory error? I ask because I found that the latest PuppetServer includes a newer version of logback, as shown by this message in /var/log/messages:

Mar 18 01:56:21 puppetserver java: Exception in thread "main" java.lang.AbstractMethodError: ch.qos.logback.core.net.SyslogAppenderBase.createOutputStream()Lch/qos/logback/core/net/SyslogOutputStream;
Mar 18 01:56:21 puppetserver java: at ch.qos.logback.core.net.SyslogAppenderBase.start(SyslogAppenderBase.java:62)
Mar 18 01:56:21 puppetserver java: at ch.qos.logback.classic.net.SyslogAppender.start(SyslogAppender.java:48)

If you see, this replace “classic.net.Syslog” in logback.xml with "core.net.Syslog"

sed -i_old -e 's/classic.net.Syslog/core.net.Syslog/' /etc/puppetlabs/puppetserver/logback.xml

If that's not the problem, please post your logfiles.