ContextQMD
Back to Salt

Developing New Modules

doc/topics/development/modules/developing.rst

2019.88.5 KB
Original Source

====================== Developing New Modules

Interactive Debugging

Sometimes debugging with print() and extra logs sprinkled everywhere is not the best strategy.

IPython is a helpful debug tool that has an interactive python environment which can be embedded in python programs.

First the system will require IPython to be installed.

.. code-block:: bash

# Debian
apt-get install ipython

# Arch Linux
pacman -Syu ipython2

# RHEL/CentOS (via EPEL)
yum install python-ipython

Now, in the troubling python module, add the following line at a location where the debugger should be started:

.. code-block:: python

test = "test123"
import IPython

IPython.embed_kernel()

After running a Salt command that hits that line, the following will show up in the log file:

.. code-block:: text

[CRITICAL] To connect another client to this kernel, use:
[IPKernelApp] --existing kernel-31271.json

Now on the system that invoked embed_kernel, run the following command from a shell:

.. code-block:: bash

# NOTE: use ipython2 instead of ipython for Arch Linux
ipython console --existing

This provides a console that has access to all the vars and functions, and even supports tab-completion.

.. code-block:: python

print(test)
test123

To exit IPython and continue running Salt, press Ctrl-d to logout.

.. _loader:

The Salt Loader

Salt's loader system is responsible for reading Special Module Contents_ and providing the context for the special Dunder Dictionaries. When modules developed for Salt's loader are imported directly, the dunder attributes won't be populated. You can use the Loader Context to work around this.

Loader Context

Given the following.

.. code-block:: python

    # coolmod.py


    def utils_func_getter(name):
        return __utils__[name]

You would not be able import coolmod and run utils_func_getter because __utils__ would not be defined. You must run coolmod.utils_func_getter in the context of a loader.

.. code-block:: python

    import coolmod
    import salt.loader

    opts = {}
    utils = salt.loader.utils(opts)
    with salt.loader.context(utils):
        func = coolmod.utils_func_getter("foo.bar")

.. _module-naming-best-practices:

Module Naming Best Practices

For optimal loader performance, name module files to match their module name or include the module name in the filename.

When the loader searches for a module, it performs the following stages:

  1. Exact filename match (e.g., test.py for module test)
  2. Partial filename matches (files containing "test" in the name)
  3. Expensive search through every module file

Naming your module files appropriately allows the loader to find modules using stages 1 or 2, avoiding the expensive full scan in stage 3. This reduces memory usage.

Examples of good naming:

  • If a module provides functionality under the name mymod, name the file mymod.py
  • If using virtual names via __virtualname__, include the virtual name in the filename (e.g., mymod_impl.py for virtual name mymod)

See :conf_minion:lazy_loader_strict_matching for a configuration option that disables the expensive stage 3 search entirely.

Special Module Contents

These are things that may be defined by the module to influence various things.

virtual

virtual_aliases

virtualname

init

Called before __virtual__()

proxyenabled

grains and proxy modules

proxyenabled as a list containing the names of the proxy types that the module supports.

load

func_alias

outputter

.. _dunder-dictionaries:

Dunder Dictionaries

Salt provides several special "dunder" dictionaries as a convenience for Salt development. These include __opts__, __context__, __salt__, and others. This document will describe each dictionary and detail where they exist and what information and/or functionality they provide.

The following dunder dictionaries are always defined, but may be empty

  • __context__
  • __grains__
  • __pillar__
  • __opts__

opts

.. versionchanged:: 3006.0

The ``__opts__`` dictionary can now be accessed via
:py:mod:`~salt.loader.context``.

Defined in: All modules

The __opts__ dictionary contains all of the options passed in the configuration file for the master or minion.

.. note::

In many places in salt, instead of pulling raw data from the __opts__
dict, configuration data should be pulled from the salt `get` functions
such as config.get

.. code-block:: python

    __salt__["config.get"]("foo:bar")


The `get` functions also allow for dict traversal via the *:* delimiter.
Consider using get functions whenever using ``__opts__`` or ``__pillar__``
and ``__grains__`` (when using grains for configuration data)

The configuration file data made available in the __opts__ dictionary is the configuration data relative to the running daemon. If the modules are loaded and executed by the master, then the master configuration data is available, if the modules are executed by the minion, then the minion configuration is available. Any additional information passed into the respective configuration files is made available

salt

Defined in: Auth, Beacons, Engines, Execution, Executors, Outputters, Pillars, Proxies, Renderers, Returners, Runners, SDB, SSH Wrappers, State, Thorium

__salt__ contains the execution module functions. This allows for all functions to be called as they have been set up by the salt loader.

.. code-block:: python

__salt__["cmd.run"]("fdisk -l")
__salt__["network.ip_addrs"]()

.. note::

When used in runners or outputters, ``__salt__`` references other
runner/outputter modules, and not execution modules.

grains

Filled in for: Execution, Pillar, Renderer, Returner, SSH Wrapper, State.

The __grains__ dictionary contains the grains data generated by the minion that is currently being worked with. In execution modules, state modules and returners this is the grains of the minion running the calls, when generating the external pillar the __grains__ is the grains data from the minion that the pillar is being generated for.

While __grains__ is defined for every module, it's only filled in for some.

pillar

Filled in for: Execution, Renderer, Returner, SSH Wrapper, State

The __pillar__ dictionary contains the pillar for the respective minion.

While __pillar__ is defined for every module, it's only filled in for some.

ext_pillar

Filled in for: Pillar

The __ext_pillar__ dictionary contains the external pillar modules.

.. _dunder-context:

context

During a state run the __context__ dictionary persists across all states that are run and then is destroyed when the state ends.

When running an execution module __context__ persists across all module executions until the modules are refreshed; such as when :py:func:saltutil.sync_all <salt.modules.saltutil.sync_all> or :py:func:state.apply <salt.modules.state.apply_> are executed.

.. code-block:: python

if not "cp.fileclient" in __context__:
    __context__["cp.fileclient"] = salt.fileclient.get_file_client(__opts__)

.. note:: Because context may or may not have been destroyed, always be sure to check for the existence of the key in context and generate the key before using it.

utils

Defined in: Cloud, Engine, Execution, File Server, Grain, Pillar, Proxy, Roster, Runner, SDB, State

proxy

Defined in: Beacon, Engine, Execution, Executor, Proxy, Renderer, Returner, State, Util

runner

Defined in: Engine, Roster, Thorium

.. note:: When used in engines, it should be called runners (plural)

executors

Defined in: Executor

ret

Defined in: Proxy

thorium

Defined in: Thorium

states

Defined in: Renderers, State

serializers

Defined in: State

sdb

Defined in: SDB

file_client

.. versionchanged:: 3006.5

The __file_client__ dunder was added to states and execution modules. This enables the use of a file client without haveing to instantiate one in the module.