How to continue a task when Fabric receives an error

Question

When I define a task to run on several remote servers, if the task runs on server one and exits with an error, Fabric will stop and abort the task. But I want to make fabric ignore the error and run the task on the next server. How can I make it do this?

For example:

$ fab site1_service_gw
[site1rpt1] Executing task 'site1_service_gw'

[site1fep1] run: echo 'Nm123!@#' | sudo -S route
[site1fep1] err:
[site1fep1] err: We trust you have received the usual lecture from the local System
[site1fep1] err: Administrator. It usually boils down to these three things:
[site1fep1] err:
[site1fep1] err:     #1) Respect the privacy of others.
[site1fep1] err:     #2) Think before you type.
[site1fep1] err:     #3) With great power comes great responsibility.
[site1fep1] err: root's password:
[site1fep1] err: sudo: route: command not found

Fatal error: run() encountered an error (return code 1) while executing 'echo 'Nm123!@#' | sudo -S route '

Aborting.

Answer 1

From the docs:

... Fabric defaults to a “fail-fast” behavior pattern: if anything goes wrong, such as a remote program returning a nonzero return value or your fabfile’s Python code encountering an exception, execution will halt immediately.

This is typically the desired behavior, but there are many exceptions to the rule, so Fabric provides env.warn_only, a Boolean setting. It defaults to False, meaning an error condition will result in the program aborting immediately. However, if env.warn_only is set to True at the time of failure – with, say, the settings context manager – Fabric will emit a warning message but continue executing.

Looks like you can exercise fine-grained control over where errors are ignored by using the settings context manager, something like so:

from fabric.api import settings

sudo('mkdir tmp') # can't fail
with settings(warn_only=True):
    sudo('touch tmp/test') # can fail
sudo('rm tmp') # can't fail