Embedding an Expect script inside a Bash script

Question

I have the following script:

#!/bin/bash

if [ `hostname` = 'EXAMPLE' ]
then

/usr/bin/expect << EOD

spawn scp -rp host:~/outfiles/ /home/USERNAME/outfiles/
expect "id_rsa':"
send "PASSWORD\r"
interact

spawn scp -rp host:~/errfiles/ /home/USERNAME/errfiles/
expect "id_rsa':"
send "PASSWORD\r"
interact

expect eof

EOD

echo 'Successful download'
fi

Unfortunately it doesn't seem to work and I get an error message:

spawn scp -rp host:~/outfiles/ /home/USERNAME/outfiles/
Enter passphrase for key '/home/USERNAME/.ssh/id_rsa': interact: spawn id exp0 not open
    while executing
"interact"

I don't know what it means and why it doesn't work. However, when I wrote the above code using a not-embedded Expect script:

#!/usr/bin/expect

spawn scp -rp host:~/outfiles/ /home/USERNAME/outfiles/
expect "id_rsa':"
send "PASSWORD\r"
interact

spawn scp -rp host:~/errfiles/ /home/USERNAME/errfiles/
expect "id_rsa':"
send "PASSWORD\r"
interact

It worked without any problems. So what am I doing wrong?

NOTE: Often when someone posts a question about using Expect to use scp or ssh the answer given is to use RSA keys. I tried, unfortunately on one of my computers there is some crappy bug with the GNOME keyring that means that I can't remove my password from the RSA key, which is exactly why I'm trying to write the above script with an if statement. So please don't tell me to use RSA keys.

Answer 1

Your Bash script is passing the Expect commands on the standard input of expect. That is what the here-document <<EOD does. However, expect... expects its commands to be provided in a file, or as the argument of a -c, per the man page. Three options are below. Caveat emptor; none have been tested.

1. Process substitution with here-document:

   expect <(cat <<'EOD'
   spawn ... (your script here)
   EOD
   )
   

   The EOD ends the here-document, and then the whole thing is wrapped in a <( ) process substitution block. The result is that expect will see a temporary filename including the contents of your here-document.

   As @Aserre noted, the quotes in <<'EOD' mean that everything in your here-document will be treated literally. Leave them off to expand Bash variables and the like inside the script, if that's what you want.

2. Edit Variable+here-document:

   IFS= read -r -d '' expect_commands <<'EOD'
   spawn ... (your script here)
   interact
   EOD
   
   expect -c "${expect_commands//
   /;}"
   

   Yes, that is a real newline after // - it's not obvious to me how to escape it. That turns newlines into semicolons, which the man page says is required.

   Thanks to this answer for the read+heredoc combo.

3. Shell variable

   expect_commands='
   spawn ... (your script here)
   interact'
   expect -c "${expect_commands//
   /;}"
   

   Note that any ' in the expect commands (e.g., after id_rsa) will need to be replaced with '\'' to leave the single-quote block, add a literal apostrophe, and then re-enter the single-quote block. The newline after // is the same as in the previous option.