"Connection closed by [HOST IP]" using dsa key authentication

Question

I have a shared /home setup using Perceus Cluster Software (http://perceus.org) for our Cluster. Nodes are using CentOS 6.1 x86_64. /home is shared from the head to the nodes by nfs (NFSv4).

root@head~]$ cat /etc/exports 
/var/lib/perceus/ 10.10.10.0/255.255.255.0(ro,no_root_squash,async)
/home/ 10.10.10.0/255.255.255.0(rw,no_root_squash,no_all_squash,async)

Here is the /etc/fstab on each node (all the same).

...
10.10.10.2:/var/lib/perceus/ /var/lib/perceus/ nfs ro,soft,bg 0 0
10.10.10.2:/home/ /home nfs rw,soft,bg 0 0

/etc/fstab on nodes is a copy of the head/master with identical UID:GID.

I have created key pairs using the following method:

$ cd ~
$ rm -rf .ssh
$ mkdir .ssh
$ chmod 700 .ssh
$ ssh-keygen -t dsa -P ""
Generating public/private dsa key pair.
Enter file in which to save the key (/home/user/.ssh/id_dsa):
Your identification has been saved in /home/user/.ssh/id_dsa.
Your public key has been saved in /home/user/.ssh/id_dsa.pub.
The key fingerprint is:
[SNIPPED] user@head
The key's randomart image is:
+--[ DSA 1024]----+
[SNIPPED]
$ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
$ chmod 400 ~/.ssh/authorized_keys

Here is the problem. When I try to ssh into each node, I am getting a "Connection Closed" error. Here is the debugging output.

$ ssh node01
Connection closed by 10.10.10.101

$ ssh node01 -vvv
OpenSSH_5.3p1, OpenSSL 1.0.0-fips 29 Mar 2010
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Connecting to node01 [10.10.10.101] port 22.
debug1: Connection established.
debug1: identity file /home/user/.ssh/identity type -1
debug1: identity file /home/user/.ssh/id_rsa type -1
debug3: Not a RSA1 key file /home/user/.ssh/id_dsa.
debug2: key_type_from_name: unknown key type '-----BEGIN'
debug3: key_read: missing keytype
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug2: key_type_from_name: unknown key type '-----END'
debug3: key_read: missing keytype
debug1: identity file /home/user/.ssh/id_dsa type 2
debug1: Remote protocol version 2.0, remote software version OpenSSH_5.3
debug1: match: OpenSSH_5.3 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_5.3
.... SNIPPED ...
debug2: dh_gen_key: priv key bits set: 139/256
debug2: bits set: 482/1024
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug3: Wrote 144 bytes for a total of 981
debug3: check_host_in_hostfile: filename /home/user/.ssh/known_hosts
debug3: check_host_in_hostfile: match line 1
debug3: check_host_in_hostfile: filename /home/user/.ssh/known_hosts
debug3: check_host_in_hostfile: match line 1
debug1: Host 'node01' is known and matches the RSA host key.
debug1: Found key in /home/user/.ssh/known_hosts:1
debug2: bits set: 501/1024
debug1: ssh_rsa_verify: signature correct
debug2: kex_derive_keys
debug2: set_newkeys: mode 1
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: Wrote 16 bytes for a total of 997
debug2: set_newkeys: mode 0
debug1: SSH2_MSG_NEWKEYS received
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug3: Wrote 48 bytes for a total of 1045
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug2: key: /home/user/.ssh/identity ((nil))
debug2: key: /home/user/.ssh/id_rsa ((nil))
debug2: key: /home/user/.ssh/id_dsa (0x7f79b940f650)
debug3: Wrote 64 bytes for a total of 1109
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug3: start over, passed a different list publickey,gssapi-keyex,gssapi-with-mic,password
debug3: preferred gssapi-keyex,gssapi-with-mic,publickey,keyboard-interactive,password
... [SNIPPED]...
debug1: Next authentication method: publickey
debug1: Trying private key: /home/user/.ssh/identity
debug3: no such identity: /home/user/.ssh/identity
debug1: Trying private key: /home/user/.ssh/id_rsa
debug3: no such identity: /home/user/.ssh/id_rsa
debug1: Offering public key: /home/user/.ssh/id_dsa
debug3: send_pubkey_test
debug2: we sent a publickey packet, wait for reply
debug3: Wrote 528 bytes for a total of 1637
debug1: Server accepts key: pkalg ssh-dss blen 434
debug2: input_userauth_pk_ok: SHA1 fp 46:a2:c3:86...........
debug3: sign_and_send_pubkey
debug1: read PEM private key done: type DSA
debug3: Wrote 592 bytes for a total of 2229
Connection closed by 10.10.10.101

I have making sure that /etc/ssh/sshd_config allows key based authentication (PubkeyAuthentication yes). I have made sure the the permissions on /home (once mounted on the nodes) is correct. Users are properly authenticated. I have tried nfs mounting with and without "no_all_squash" restarting nfs, rpcidmap, rpcbind and nfslock.

I have had this working with CentOS5 installed on the nodes with a different master/head node. CentOS6 just seems to be giving me extra problems with this.

If I don't create the key, of course I'm prompted for a password.

My hosts.allow/deny are empty on both clients and server.

The root user is able to connect. Perceus handles the key generation for the root user since it is part of the virtual file system. I am guessing that something is wrong with the generation of my key but I can't figure out what the problem is.

Answer 1

The correct solutions is to fix the problem, not disable the pam usage, as you might be hiding a security problem.

ssh is failing because PAM is denying the user login by failing some check. Verify the /etc/pam.d/sshd for what rules you have and what might be failing.

most common problem is a user without password (compare the /etc/passwd with /etc/shadow, or check your /etc/nsswitch and /etc/pam.d/* to see where the users and auth is coming from), but also no home directory, missing some extra auth configuration, UID too low or too high, etc.

If its the missing password, at least make sure you this in the /etc/ssh/sshd_config

PermitEmptyPasswords no

This blocks ssh to allow login on users without password (but does nothing to other protocols, like telnet, ftp, http and login).