Reliable example of how to use SFTP using public private key authentication with Java

Question

Recently a client of our unexpectedly shifted some important files we collect from an ftp to sftp server. Initially I was under the impression that it would be simple to write or find a java utility that can handle sftp, this has definitely not proven to be the case. What has also compounded this problem is that we are trying to connect to the sftp server from a windows platform (so the definition of where SSH_HOME is on the client gets very confused).

I have been using the apache-commons-vfs library and have managed to get a solution that reliably works for username/password authentication, but as of yet nothing that can reliably handle private/public key authentication.

The following example works for username/password authentication, but I want to adjust it for private/public key authentication.

public static void sftpGetFile(String server, String userName,String password, 
        String remoteDir, String localDir, String fileNameRegex)
    {

       File localDirFile  = new File(localDir);
       FileSystemManager fsManager = null;

       if (!localDirFile.exists()) {
           localDirFile.mkdirs();
       }

       try {
           fsManager = VFS.getManager();
       } catch (FileSystemException ex) {
           LOGGER.error("Failed to get fsManager from VFS",ex);
           throw new RuntimeException("Failed to get fsManager from VFS", ex);
       }

       UserAuthenticator auth = new StaticUserAuthenticator(null, userName,password);

       FileSystemOptions opts = new FileSystemOptions();

       try {
           DefaultFileSystemConfigBuilder.getInstance().setUserAuthenticator(opts,
                   auth);
       } catch (FileSystemException ex) {
           LOGGER.error("setUserAuthenticator failed", ex);
           throw new RuntimeException("setUserAuthenticator failed", ex);
       }
       Pattern filePattern = Pattern.compile(fileNameRegex);
       String startPath = "sftp://" + server + remoteDir;
       FileObject[] children;

       // Set starting path on remote SFTP server.
       FileObject sftpFile;
       try {
           sftpFile = fsManager.resolveFile(startPath, opts);

           LOGGER.info("SFTP connection successfully established to " +
                   startPath);
       } catch (FileSystemException ex) {
           LOGGER.error("SFTP error parsing path " +
                   remoteDir,
                   ex);

           throw new RuntimeException("SFTP error parsing path " +
                   remoteDir,
                   ex);
       }

       // Get a directory listing
       try {
           children = sftpFile.getChildren();
       } catch (FileSystemException ex) {
           throw new RuntimeException("Error collecting directory listing of " +
                   startPath, ex);
       }

       search:
       for (FileObject f : children) {
           try {
               String relativePath =
                       File.separatorChar + f.getName().getBaseName();

               if (f.getType() == FileType.FILE) {
                   System.out.println("Examining remote file " + f.getName());

                   if (!filePattern.matcher(f.getName().getPath()).matches()) {
                       LOGGER.info("  Filename does not match, skipping file ." +
                               relativePath);
                       continue search;
                   }

                   String localUrl = "file://" + localDir + relativePath;
                   String standardPath = localDir + relativePath;
                   System.out.println("  Standard local path is " + standardPath);
                   LocalFile localFile =
                           (LocalFile) fsManager.resolveFile(localUrl);
                   System.out.println("    Resolved local file name: " +
                           localFile.getName());

                   if (!localFile.getParent().exists()) {
                       localFile.getParent().createFolder();
                   }

                   System.out.println("  ### Retrieving file ###");
                   localFile.copyFrom(f,
                           new AllFileSelector());
               } else {
                   System.out.println("Ignoring non-file " + f.getName());
               }
           } catch (FileSystemException ex) {
               throw new RuntimeException("Error getting file type for " +
                       f.getName(), ex);
           }

       }

       FileSystem fs = null;
       if (children.length > 0) {
           fs = children[0].getFileSystem(); // This works even if the src is closed.
           fsManager.closeFileSystem(fs);
       }
    }

I've got my private key stored in a known location and my public key has been distrubuted to the server (we have tested that these keys work succesfully when connecting using other tools)

I have played around with adding the following line

SftpFileSystemConfigBuilder.getInstance().setIdentities(this.opts, new File[]{new File("c:/Users/bobtbuilder/.ssh/id_dsa.ppk")});

This succesfully loads the private key into the whole framework but it never then uses that key to authenticate further.

Any help or direction most warmly received

Answer 1

After much digging around I finally got to the answer myself. Seems that much of my bother was to do with the format of the private and public key

privateKey must be in openSSH format publicKey for whatever reason can only be pasted from the puttyGen window (exporting the public key always seemed to give it with missing headers meaning the freeSSHD windows server could not use it)

Anyhow below is my code I finally came up with including the javadoc, so hopefully should save some others the pain I went through

/**
* Fetches a file from a remote sftp server and copies it to a local file location.  The authentication method used
* is public/private key authentication. <br><br>

* IMPORTANT: Your private key must be in the OpenSSH format, also it must not have a passphrase associated with it.
*    (currently the apache-commons-vfs2 library does not support passphrases)<p>
* 
* Also remember your public key needs to be on the sftp server.  If you were connecting as user 'bob' then your
* public key will need to be in '.ssh/bob' on the server (the location of .ssh will change depending on the type
* of sftp server)
* 
* @param server The server we care connection to 
* @param userName The username we are connection as
* @param openSSHPrivateKey The location of the  private key (which must be in openSSH format) on the local machine
* @param remoteDir The directory from where you want to retrieve the file on the remote machine (this is in reference to SSH_HOME, SSH_HOME is the direcory you 
* automatically get directed to when connecting)
* @param remoteFile The name of the file on the remote machine to be collected (does not support wild cards)
* @param localDir The direcoty on the local machine where you want the file to be copied to
* @param localFileName The name you wish to give to retrieved file on the local machine
* @throws IOException - Gets thrown is there is any problem fetching the file
*/
public static void sftpGetFile_keyAuthentication(String server, String userName, String openSSHPrivateKey,
    String remoteDir,String remoteFile, String localDir, String localFileName) throws IOException
{

   FileSystemOptions fsOptions = new FileSystemOptions();
   FileSystemManager fsManager = null;
   String remoteURL = "sftp://" + userName + "@" + server + "/" + remoteDir + "/" + remoteFile;
   String localURL  = "file://" + localDir + "/" + localFileName;

    try {
        SftpFileSystemConfigBuilder.getInstance().setStrictHostKeyChecking(fsOptions, "no");
        SftpFileSystemConfigBuilder.getInstance().setIdentities(fsOptions, new File[]{new File(openSSHPrivateKey)});
        fsManager = VFS.getManager();
        FileObject remoteFileObject = fsManager.resolveFile(remoteURL, fsOptions);
        LocalFile localFile =
                   (LocalFile) fsManager.resolveFile(localURL);
        localFile.copyFrom(remoteFileObject,
                   new AllFileSelector());
    } catch (FileSystemException e) {
        LOGGER.error("Problem retrieving from " + remoteURL + " to " + localURL,e );
        throw new IOException(e);
    }
}