Is there a way to ssh to different server and run BashOperator using Airbnb's Airflow? I am trying to run a hive sql command with Airflow but I need to SSH to a different box in order to run the hive shell. My tasks should look like this:
Thanks!
I think that I just figured it out:
Create a SSH connection in UI under Admin > Connection. Note: the connection will be deleted if you reset the database
In the Python file add the following
from airflow.contrib.hooks import SSHHook
sshHook = SSHHook(conn_id=<YOUR CONNECTION ID FROM THE UI>)
Add the SSH operator task
t1 = SSHExecuteOperator(
task_id="task1",
bash_command=<YOUR COMMAND>,
ssh_hook=sshHook,
dag=dag)
Thanks!