Best way to prevent "mux_client_request_session: read from master failed: Broken pipe" ssh errors

Question

I've got a perl script that creates an ssh tunnel and sets up a Perl DBI connection across it to query a database on a remote host:

1 my $ssh = Net::OpenSSH->new('[email protected]');
2 $pid = $ssh->spawn({ssh_opts => '-L 127.0.0.1:12345:127.0.0.1:3306'}, 'sleep 3');

3 return DBI->connect($dsn, $db_user, $db_pass);

This works about 80 to 90% of the time but the rest of the time I get this error when trying to connect to the database:

mux_client_request_session: read from master failed: Broken pipe

In the process of troubleshooting this, I noticed if I sleep the program very briefly after line 2 with usleep (10000), it works 100% of the time. I'm not sure why this is but I'm curious know and how I can properly fix the issue.

Thanks.

Answer 1

There is a race condition in your code: spawn runs the ssh command that starts the tunnel on the background, sometimes that command is faster than DBI connect starting the TCP socket, sometimes it isn't and so connect fails.

Anyway, the way I usually recommend to create tunnels is to use a control command:

 $ssh->system({ssh_opts => [-O => 'forward', '-L3066:localhost:3066']})

If you want to remove the tunnel once the connection to the database has been established:

 $ssh->system({ssh_opts => [-O => 'cancel', '-L3066:localhost:3066']});

Also, note that you have to keep the Net::OpenSSH object alive as long as you want to use the DB connection.

In summary:

sub dbi_connect {
    my $mysql_port = 3066;
    my $local_port = 12345;
    my $dsn = "DBI:mysql:database=$database;host=localhost;port=$local_port";
    my $tunnel = join ':', $local_port, 'localhost', $mysql_port;
    my $ssh = Net::OpenSSH->new($host, ...);
    $ssh->system({ssh_opts => [-O => 'forward', "-L$tunnel"]});
    my $dbi = DBI->connect($dsn, $db_user, $db_pass);
    $ssh->system({ssh_opts => [-O => 'cancel', "-L$tunnel"]});
    return ($dbi, $ssh);
}

See also this post at PerlMonks.