=============== Getting started

Welcome! This tutorial highlights Fabric's core features; for further details, see the links within, or the documentation index which has links to conceptual and API doc sections.

A note about imports

Fabric composes a couple of other libraries as well as providing its own layer on top; user code will most often import from the fabric package, but you'll sometimes import directly from invoke or paramiko too:

• Invoke <https://www.pyinvoke.org>_ implements CLI parsing, task organization, and shell command execution (a generic framework plus specific implementation for local commands.)

  • Anything that isn't specific to remote systems tends to live in Invoke, and it is often used standalone by programmers who don't need any remote functionality.
  • Fabric users will frequently import Invoke objects, in cases where Fabric itself has no need to subclass or otherwise modify what Invoke provides.

• Paramiko <https://www.paramiko.org>_ implements low/mid level SSH functionality - SSH and SFTP sessions, key management, etc.

  • Fabric mostly uses this under the hood; users will only rarely import from Paramiko directly.

• Fabric glues the other libraries together and provides its own high level objects too, e.g.:

  • Subclassing Invoke's context and command-runner classes, wrapping them around Paramiko-level primitives;
  • Extending Invoke's configuration system by using Paramiko's ssh_config parsing machinery;
  • Implementing new high-level primitives of its own, such as port-forwarding context managers. (These may, in time, migrate downwards into Paramiko.)

.. TODO: we should probably rename Collection to be Namespace or something; it's too close to 'Connection'

Run commands via Connections and run

The most basic use of Fabric is to execute a shell command on a remote system via SSH, then (optionally) interrogate the result. By default, the remote program's output is printed directly to your terminal, and captured. A basic example:

.. testsetup:: basic

mock = MockRemote()
mock.expect(out=b'Linux\n')

.. testcleanup:: basic

mock.stop()

.. doctest:: basic

>>> from fabric import Connection
>>> c = Connection('web1')
>>> result = c.run('uname -s')
Linux
>>> result.stdout.strip() == 'Linux'
True
>>> result.exited
0
>>> result.ok
True
>>> result.command
'uname -s'
>>> result.connection
<Connection host=web1>
>>> result.connection.host
'web1'

Meet .Connection, which represents an SSH connection and provides the core of Fabric's API, such as ~.Connection.run. .Connection objects need at least a hostname to be created successfully, and may be further parameterized by username and/or port number. You can give these explicitly via args/kwargs::

Connection(host='web1', user='deploy', port=2202)

Or by stuffing a [user@]host[:port] string into the host argument (though this is purely convenience; always use kwargs whenever ambiguity appears!)::

Connection('deploy@web1:2202')

.Connection objects' methods (like ~.Connection.run) usually return instances of invoke.runners.Result (or subclasses thereof) exposing the sorts of details seen above: what was requested, what happened while the remote action occurred, and what the final result was.

.. note:: Many lower-level SSH connection arguments (such as private keys and timeouts) can be given directly to the SSH backend by using the :ref:connect_kwargs argument <connect_kwargs-arg>.

Superuser privileges via auto-response

Need to run things as the remote system's superuser? You could invoke the sudo program via ~.Connection.run, and (if your remote system isn't configured with passwordless sudo) respond to the password prompt by hand, as below. (Note how we need to request a remote pseudo-terminal; most sudo implementations get grumpy at password-prompt time otherwise.)

.. testsetup:: sudo-by-hand

mock = MockRemote()
mock.expect(commands=(
    Command(out=b'[sudo] password:\n'),
    Command(out=b'1001\n'),
))

.. testcleanup:: sudo-by-hand

mock.stop()

.. doctest:: sudo-by-hand

>>> from fabric import Connection
>>> c = Connection('db1')
>>> c.run('sudo useradd mydbuser', pty=True)
[sudo] password:
<Result cmd='sudo useradd mydbuser' exited=0>
>>> c.run('id -u mydbuser')
1001
<Result cmd='id -u mydbuser' exited=0>

Giving passwords by hand every time can get old; thankfully Invoke's powerful command-execution functionality includes the ability to :ref:auto-respond <autoresponding> to program output with pre-defined input. We can use this for sudo:

.. testsetup:: sudo-with-responses

mock = MockRemote()
mock.expect(out=b'[sudo] password:\nroot\n', in_=b'mypassword\n')

.. testcleanup:: sudo-with-responses

mock.stop()

.. doctest:: sudo-with-responses

>>> from invoke import Responder
>>> from fabric import Connection
>>> c = Connection('host')
>>> sudopass = Responder(
...     pattern=r'\[sudo\] password:',
...     response='mypassword\n',
... )
>>> c.run('sudo whoami', pty=True, watchers=[sudopass])
[sudo] password:
root
<Result cmd='sudo whoami' exited=0>

It's difficult to show in a snippet, but when the above was executed, the user didn't need to type anything; mypassword was sent to the remote program automatically. Much easier!

The sudo helper

Using watchers/responders works well here, but it's a lot of boilerplate to set up every time - especially as real-world use cases need more work to detect failed/incorrect passwords.

To help with that, Invoke provides a Context.sudo <invoke.context.Context.sudo> method which handles most of the boilerplate for you (as .Connection subclasses ~invoke.context.Context, it gets this method for free.) ~invoke.context.Context.sudo doesn't do anything users can't do themselves - but as always, common problems are best solved with commonly shared solutions.

All the user needs to do is ensure the sudo.password :doc:configuration value </concepts/configuration> is filled in (via config file, environment variable, or :option:--prompt-for-sudo-password) and .Connection.sudo handles the rest. For the sake of clarity, here's an example where a library/shell user performs their own getpass-based password prompt:

.. testsetup:: sudo

from unittest.mock import patch
gp_patcher = patch('getpass.getpass', side_effect=lambda x: print(x))
gp_patcher.start()
mock = MockRemote()
mock.expect(commands=(
    Command(out=b'root\n'),
    Command(),
    Command(out=b'1001\n'),
))

.. testcleanup:: sudo

mock.stop()
gp_patcher.stop()

.. doctest:: sudo :options: +ELLIPSIS

>>> import getpass
>>> from fabric import Connection, Config
>>> sudo_pass = getpass.getpass("What's your sudo password?")
What's your sudo password?
>>> config = Config(overrides={'sudo': {'password': sudo_pass}})
>>> c = Connection('db1', config=config)
>>> c.sudo('whoami', hide='stderr')
root
<Result cmd="...whoami" exited=0>
>>> c.sudo('useradd mydbuser')
<Result cmd="...useradd mydbuser" exited=0>
>>> c.run('id -u mydbuser')
1001
<Result cmd='id -u mydbuser' exited=0>

We filled in the sudo password up-front at runtime in this example; in real-world situations, you might also supply it via the configuration system (perhaps using environment variables, to avoid polluting config files), or ideally, use a secrets management system.

Transfer files

Besides shell command execution, the other common use of SSH connections is file transfer; .Connection.put and .Connection.get exist to fill this need. For example, say you had an archive file you wanted to upload:

.. testsetup:: transfers

mock = MockRemote(enable_sftp=True)

.. testcleanup:: transfers

mock.stop()

.. doctest:: transfers

>>> from fabric import Connection
>>> result = Connection('web1').put('myfiles.tgz', remote='/opt/mydata/')
>>> print("Uploaded {0.local} to {0.remote}".format(result))
Uploaded /local/myfiles.tgz to /opt/mydata/

These methods typically follow the behavior of cp and scp/sftp in terms of argument evaluation - for example, in the above snippet, we omitted the filename part of the remote path argument.

Multiple actions

One-liners are good examples but aren't always realistic use cases - one typically needs multiple steps to do anything interesting. At the most basic level, you could do this by calling .Connection methods multiple times::

from fabric import Connection
c = Connection('web1')
c.put('myfiles.tgz', '/opt/mydata')
c.run('tar -C /opt/mydata -xzvf /opt/mydata/myfiles.tgz')

You could (but don't have to) turn such blocks of code into functions, parameterized with a .Connection object from the caller, to encourage reuse::

def upload_and_unpack(c):
    c.put('myfiles.tgz', '/opt/mydata')
    c.run('tar -C /opt/mydata -xzvf /opt/mydata/myfiles.tgz')
    

As you'll see below, such functions can be handed to other API methods to enable more complex use cases as well.

Multiple servers

Most real use cases involve doing things on more than one server. The straightforward approach could be to iterate over a list or tuple of .Connection arguments (or .Connection objects themselves, perhaps via map)::

>>> from fabric import Connection
>>> for host in ('web1', 'web2', 'mac1'):
...     result = Connection(host).run('uname -s')
...     print("{}: {}".format(host, result.stdout.strip()))
...
...
web1: Linux
web2: Linux
mac1: Darwin

This approach works, but as use cases get more complex it can be useful to think of a collection of hosts as a single object. Enter .Group, a class wrapping one-or-more .Connection objects and offering a similar API; specifically, you'll want to use one of its concrete subclasses like .SerialGroup or .ThreadingGroup.

The previous example, using .Group (.SerialGroup specifically), looks like this::

>>> from fabric import SerialGroup as Group
>>> results = Group('web1', 'web2', 'mac1').run('uname -s')
>>> print(results)
<GroupResult: {
    <Connection 'web1'>: <CommandResult 'uname -s'>,
    <Connection 'web2'>: <CommandResult 'uname -s'>,
    <Connection 'mac1'>: <CommandResult 'uname -s'>,
}>
>>> for connection, result in results.items():
...     print("{0.host}: {1.stdout}".format(connection, result))
...
...
web1: Linux
web2: Linux
mac1: Darwin

Where .Connection methods return single Result objects (e.g. fabric.runners.Result), .Group methods return .GroupResult - dict-like objects offering access to individual per-connection results as well as metadata about the entire run.

When any individual connections within the .Group encounter errors, the .GroupResult is lightly wrapped in a .GroupException, which is raised. Thus the aggregate behavior resembles that of individual .Connection methods, returning a value on success or raising an exception on failure.

Bringing it all together

Finally, we arrive at the most realistic use case: you've got a bundle of commands and/or file transfers and you want to apply it to multiple servers. You could use multiple .Group method calls to do this::

from fabric import SerialGroup as Group
pool = Group('web1', 'web2', 'web3')
pool.put('myfiles.tgz', '/opt/mydata')
pool.run('tar -C /opt/mydata -xzvf /opt/mydata/myfiles.tgz')

That approach falls short as soon as logic becomes necessary - for example, if you only wanted to perform the copy-and-untar above when /opt/mydata is empty. Performing that sort of check requires execution on a per-server basis.

You could fill that need by using iterables of .Connection objects (though this foregoes some benefits of using Groups <.Group>)::

from fabric import Connection
for host in ('web1', 'web2', 'web3'):
    c = Connection(host)
    if c.run('test -f /opt/mydata/myfile', warn=True).failed:
        c.put('myfiles.tgz', '/opt/mydata')
        c.run('tar -C /opt/mydata -xzvf /opt/mydata/myfiles.tgz')

Alternatively, remember how we used a function in that earlier example? You can go that route instead::

from fabric import SerialGroup as Group

def upload_and_unpack(c):
    if c.run('test -f /opt/mydata/myfile', warn=True).failed:
        c.put('myfiles.tgz', '/opt/mydata')
        c.run('tar -C /opt/mydata -xzvf /opt/mydata/myfiles.tgz')

for connection in Group('web1', 'web2', 'web3'):
    upload_and_unpack(connection)

The only convenience this final approach lacks is a useful analogue to .Group.run - if you want to track the results of all the upload_and_unpack call as an aggregate, you have to do that yourself. Look to future feature releases for more in this space!

Addendum: the fab command-line tool

It's often useful to run Fabric code from a shell, e.g. deploying applications or running sysadmin jobs on arbitrary servers. You could use regular :ref:Invoke tasks <defining-and-running-task-functions> with Fabric library code in them, but another option is Fabric's own "network-oriented" tool, fab.

fab wraps Invoke's CLI mechanics with features like host selection, letting you quickly run tasks on various servers - without having to define host kwargs on all your tasks or similar.

.. note:: This mode was the primary API of Fabric 1.x; as of 2.0 it's just a convenience. Whenever your use case falls outside these shortcuts, it should be easy to revert to the library API directly (with or without Invoke's less opinionated CLI tasks wrapped around it).

For a final code example, let's adapt the previous example into a fab task module called fabfile.py::

from fabric import task

@task
def upload_and_unpack(c):
    if c.run('test -f /opt/mydata/myfile', warn=True).failed:
        c.put('myfiles.tgz', '/opt/mydata')
        c.run('tar -C /opt/mydata -xzvf /opt/mydata/myfiles.tgz')

Not hard - all we did was copy our temporary task function into a file and slap a decorator on it. ~fabric.tasks.task tells the CLI machinery to expose the task on the command line::

$ fab --list
Available tasks:

  upload_and_unpack

Then, when fab actually invokes a task, it knows how to stitch together arguments controlling target servers, and run the task once per server. To run the task once on a single server::

$ fab -H web1 upload_and_unpack

When this occurs, c inside the task is set, effectively, to Connection("web1") - as in earlier examples. Similarly, you can give more than one host, which runs the task multiple times, each time with a different .Connection instance handed in::

$ fab -H web1,web2,web3 upload_and_unpack