tutorial 1 - basics

This tutorial demonstrates how to use doit to create simple computational pipeline.

The goal is to create a graph of python module imports in a package. graphviz’s dot tool will be used to generate the graph.

As example the requests package will be used. The result is the image below:

_images/requests.png

In the image an arrow represents an import from one module to another. For example in the left side of image you can see an arrow from requests.status_codes to requests.structures. This comes from the following line in status_code.py:

requests/status_codes.py
from .structures import LookupDict

Drawing an “import” dependency graph can be very useful to understand how the code in a package is organized. It may also help you identify potential problems like circular imports.

There are three main steps to generate this graph:

  1. read each python module and list it’s imports
  2. generate a dot (text format) file representing the graph
  3. generate an image (PNG) from the dot file

setup

required python packages

$ pip install doit pygraphviz import_deps

sample project

First create a directory that will contain the projects to be analyzed.

$ mkdir projects

Then clone the requests project

$ cd projects
$ git clone git@github.com:requests/requests.git
$ cd ..

finding a module’s import

Using import_deps list all (intra-packages) imports from a module:

For example:

$ python -m import_deps projects/requests/requests/models.py
requests._internal_utils
requests.auth
requests.compat
requests.cookies
requests.exceptions
requests.hooks
requests.status_codes
requests.structures
requests.utils

The output contains one imported module per line.

doit task

Next step we are going to wrap the above script in a doit task. In doit tasks are defined in a plain python module, by default called dodo.py.

For example a trivial task to execute the script above and save its output into a file would be:

dodo.py
 def task_imports():
     return {
         'actions': ['python -m import_deps '
                     'projects/requests/requests/models.py > requests.models.deps'],
     }

In this module you write functions that are task-creators, the role of these functions is not to execute tasks but to return task’s metadata. task-creators are any function whose name starts with task_. A task name is taken from the function name, in this case the name is imports.

The most important Task metadata is actions, this defines what will be done when a task is executed.

Note that actions is a list where its element are a string with a shell command.

task execution

doit command line by default will execute all tasks defined in dodo.py module in the current directory.

$ doit
.  imports

The output reports that the imports task was executed. You can check that a file requests.models.deps was created with a list of modules imported by requests.models.

incremental computation

One of the main purposes in the usage of doit is to make use of up-to-date checks to decide if tasks needs to be executed or not.

In this case as long as the input file is not modified we are certain that the same output would be generated…

When dealing with files, task’s metadata file_dep and targets can be used:

def task_imports():
    """find imports from a python module"""
    return {
        'file_dep': ['projects/requests/requests/models.py'],
        'targets': ['requests.models.deps'],
        'actions': ['python -m import_deps %(dependencies)s > %(targets)s'],
    }

Note how actions can make use of variable substitution for %(dependencies)s and %(targets)s.

Now let’s execute it again:

$ doit
.  imports

And then, a second time:

$ doit
--  imports

Note that the second time there is a -- instead of . preceding the task name. This means that the task was not executed, doit understood that the task output would be the same as previously generated, so it does not execute the task again.

Warning

When doit loads a dodo.py file it executes all task-creator functions in order to generate all tasks metadata. A task’s action is only executed if the task is selected to run and not up-to-date.

Expensive computation should always be done on task’s action and never on the body of a task-creator function.

rules for up-to-date checks on files

file_dep

doit uses the md5 of file_dep to determine if a dependency is changed.

$ touch projects/requests/requests/models.py
$ doit
-- imports
$ echo "# comment" >> projects/requests/requests/models.py
$ doit
.  imports

Note that simply changing a file timestamp would not trigger a new execution.

targets

For targets, it only checks if the target exists. So if a target is removed it will be re-created even without changes on dependencies.

$ rm requests.models.deps
$ doit
.  imports

graphviz dot

Next step we will create a graphviz’s dot file. dot is a language to describe to graphs.

Code below defines a python function to read a file containing import dependencies (as generated by previously defined imports task).

import pathlib
import pygraphviz

def module_to_dot(dependencies, targets):
    graph = pygraphviz.AGraph(strict=False, directed=True)
    for dep in dependencies:
        filepath = pathlib.Path(dep)
        source = filepath.stem
        with open(filepath) as fh:
            for line in fh:
                sink = line.strip()
                if sink:
                    graph.add_edge(source, sink)
    graph.write(targets[0])

Task with python action

Next we define the task dot, it is similar to previous task… but note that instead of passing a string with a shell command we directly pass the previously created python function module_to_dot.

def task_dot():
    """generate a graphviz's dot graph from module imports"""
    return {
        'file_dep': ['requests.models.deps'],
        'targets': ['requests.models.dot'],
        'actions': [module_to_dot],
    }

Also note the function takes the special parameters dependencies and targets, those values will be injected by doit.

$ doit
-- imports
.  dot

To indicate a failure, a python-action should return the value False or raise an exception.

graph image

Finally lets add another task to generate an image from the dot file using the graphviz command line tool.

def task_draw():
    """generate image from a dot file"""
    return {
        'file_dep': ['requests.models.dot'],
        'targets': ['requests.models.png'],
        'actions': ['dot -Tpng %(dependencies)s -o %(targets)s'],
    }
$ doit
-- imports
-- dot
.  draw

Opening the file requests.models.png you should get the image below:

_images/requests.models.png

doit command line

doit has a rich (and extensible) command line to manipulate your tasks. So far we have only executed doit without any parameters…

doit command line takes the form of doit <sub-command> <options> <arguments>, where options and arguments are specific to the sub-command.

If no sub-command is specified the default command run is used. run executes tasks…

doit help

doit help will list all available sub-commands.

You can get help for a specific sub-command with doit help <sub-command>, i.e. doit help run.

You can also get help for the task metadata fields with doit help task.

doit list

The command list displays the list of known tasks:

$ doit list
dot       generate a graphviz's dot graph from module imports
draw      generate image from a dot file
imports   find imports from a python module

Note how the docstring from task-creators functions were used as task’s description.

info

The info command can be used to get more information from a specific task. Information about it’s metadata, and it is state (whether it is up-to-date or not).

$ doit info imports

imports

find imports from a python module

status     : up-to-date

file_dep   :
  - projects/requests/requests/models.py

targets    :
  - requests.models.deps

run

run is the default command, and usually not explicitly typed. So $ doit and $ doit run do exactly the same thing.

Without any parameters run will execute all of your tasks. You can also select which tasks to be executed by passing a sequence of tasks names.

For example if you want to execute only the imports task you would type:

$ doit imports
-- imports

Note that even if you explicitly pass the name of task to be executed, doit will actually execute the task only if it is not up-to-date.

You can also pass more than one task:

$ doit imports dot
-- imports
-- dot

Another important point to take notice is that even if you specify only one task doit will run all of the dependencies of the specified task.

$ doit dot
-- imports
-- dot

Note how the imports task was run because task dot has file_dep that is a target of imports task.

clean

A common use-case is to be able to “revert” the operations done by a task. doit provides the clean command for that.

By default it does nothing… You need to add a parameter clean to task metadata. For the most common case where you just want to remove the created targets, just pass the value True.

You can also write custom actions (shell or python) of what should be done as a value to clean field.

Add clean to all defined tasks, like:

def task_draw():
    """generate image from a dot file"""
    return {
        'file_dep': ['requests.models.dot'],
        'targets': ['requests.models.png'],
        'actions': ['dot -Tpng %(dependencies)s -o %(targets)s'],
        'clean': True,
    }

Executing clean:

$ doit clean
draw - removing file 'requests.models.png'
dot - removing file 'requests.models.dot'
imports - removing file 'requests.models.deps'

Since targets were removed this will force the tasks to be executed on next run.

$ doit
.  imports
.  dot
.  draw

forget

doit will look for changes in dependencies, but not changes in your code… While developing a task it is common to want to force the execution of task after making changes in the code.

For example lets change the colors of nodes in graph:

def module_to_dot(dependencies, targets):
    graph = pygraphviz.AGraph(strict=False, directed=True)
    graph.node_attr['color'] = 'lightblue2'
    graph.node_attr['style'] = 'filled'
    for dep in dependencies:
        filepath = pathlib.Path(dep)
        source = filepath.stem
        with open(filepath) as fh:

To force its execution we need to doit to forget its state like:

$ doit forget dot
forgetting dot
$ doit
-- imports
.  dot
.  draw
_images/requests.models-blue.png

Note

Another option force execution of task after code changes is to use run`’s command option -a/--always-execute. That will ignore the up-to-date check and always execute tasks.

Code dodo.py.

Pipelines

So far we have build a traditional “file” based pipeline where one task’s target is used as a dependency in another task.

While doit provides first-class support for file based pipelines, they are not required.

get module imports - python

Let’s rewrite the imports task to use python action:

from import_deps import PyModule, ModuleSet

def get_imports(module_path):
    module = PyModule(module_path)
    base_path = module.pkg_path().resolve()
    mset = ModuleSet(base_path.glob('**/*.py'))
    imports = mset.get_imports(module, return_fqn=True)
    return {'modules': list(sorted(imports))}

def task_imports():
    """find imports from a python module"""
    module_path = 'projects/requests/requests/models.py'
    return {
        'file_dep': [module_path],
        'actions': [(get_imports, [module_path])],
    }

Function get_imports is used as task’s action. It returns a dictionary, this will be saved by doit in its internal database. The returned dictionary must contain only values that can be encoded in JSON.

Note that the parameter module_path is passed into task definition of actions. Instead of just specifying a callable it takes a tuple (callable, args, kwargs).

Note

Note in this example for simplicity we are using doit internal database but it is also possible to use any other external database or data source.

getargs

The task parameter getargs can be used to specify values to be retrieved that are computed in another task. It is a dictionary where the key is the parameter name used in this task’s action parameter. It’s value is 2-value tuple where the first item is the task name, and the second is the value name (key in the returned dictionary).

def module_to_dot(source, sinks, targets):
    graph = pygraphviz.AGraph(strict=False, directed=True)
    graph.node_attr['color'] = 'lightblue2'
    graph.node_attr['style'] = 'filled'
    for sink in sinks:
        graph.add_edge(source, sink)
    graph.write(targets[0])


def task_dot():
    """generate a graphviz's dot graph from module imports"""
    return {
        'targets': ['requests.models.dot'],
        'actions': [(module_to_dot, (), {'source': 'requests.models'})],
        'getargs': {'sinks': ('imports', 'modules')},
        'clean': True,
    }

Note how module_to_dot takes 3 parameters:

  • source: value is passed directly when task’s actions is defined
  • imports: value is taken from imports task result
  • targets: values is taken from Task metadata

Everything should work as before, but without the creation of intermediate files.

doit can determine if task imports is up-to-date even without a target file (it just look at the file_dep).

doit can also determine if dot is up-to-date by comparing the value returned by imports (instead of checking for file changes, it checks for changes in the JSON object).

Code dodo.py.

package imports

So far we are creating a graph of only one module. Let’s process all modules in the package.

doit has the concept of a task-group. A task group performs the same operation over a set of instances. To create a task group the task-creator function should yield one or more task dictionaries with task metadata.

Note that each task is still independent. Since each task needs to be independently identified an extra parameter name must be provided.

def get_imports(pkg_modules, module_path):
    module = pkg_modules.by_path[module_path]
    imports = pkg_modules.get_imports(module, return_fqn=True)
    return {'modules': list(sorted(imports))}

def task_imports():
    """find imports from a python module"""
    base_path = pathlib.Path('projects/requests/requests')
    pkg_modules = ModuleSet(base_path.glob('**/*.py'))
    for name, module in pkg_modules.by_name.items():
        yield {
            'name': name,
            'file_dep': [module.path],
            'actions': [(get_imports, (pkg_modules, module.path))],
        }

Sub-tasks (items of task group) by default are not reported on list command. They be displayed using the --all flag.

$ doit list
dot       generate a graphviz's dot graph from module imports
draw      generate image from a dot file
imports   find imports from a python module
$ doit list --all imports
imports                            find imports from a python module
imports:requests.__init__
imports:requests.__version__
imports:requests._internal_utils
imports:requests.adapters
imports:requests.api
imports:requests.auth
imports:requests.certs
imports:requests.compat
imports:requests.cookies
imports:requests.exceptions
imports:requests.help
imports:requests.hooks
imports:requests.models
imports:requests.packages
imports:requests.sessions
imports:requests.status_codes
imports:requests.structures
imports:requests.utils

Note the task name is composed by the (base name) group task name followed by a colon : and the name specified as a parameter.

From command line a single task can executed like:

$ doit imports:requests.models
.  imports:requests.models

getargs from group-task

getargs can also be used to get values from a group-task. The difference is that its value will be a dictionary where the key is the sub-task name:

def module_to_dot(imports, targets):
    graph = pygraphviz.AGraph(strict=False, directed=True)
    graph.node_attr['color'] = 'lightblue2'
    graph.node_attr['style'] = 'filled'
    for source, sinks in imports.items():
        for sink in sinks:
            graph.add_edge(source, sink)
    graph.write(targets[0])

def task_dot():
    """generate a graphviz's dot graph from module imports"""
    return {
        'targets': ['requests.dot'],
        'actions': [module_to_dot],
        'getargs': {'imports': ('imports', 'modules')},
        'clean': True,
    }

Finally adjust task draw.

def task_draw():
    """generate image from a dot file"""
    return {
        'file_dep': ['requests.dot'],
        'targets': ['requests.png'],
        'actions': ['dot -Tpng %(dependencies)s -o %(targets)s'],
        'clean': True,
    }

Running doit you should get the file requests.png with the image below:

_images/requests.png

Code dodo.py.

printing imports

Getting rid of intermediate computation files (like requests.models.deps) was nice… but sometimes it is nice to be able to quickly list the direct imports from a module.

Let’s create another task just print it’s output in the terminal.

base_path = pathlib.Path('projects/requests/requests')
PKG_MODULES = ModuleSet(base_path.glob('**/*.py'))

def print_imports(modules):
    print('\n'.join(modules))

def task_print():
    """print on stdout list of direct module imports"""
    for name, module in PKG_MODULES.by_name.items():
        yield {
            'name': name,
            'actions': [print_imports],
            'getargs': {'modules': ('imports:{}'.format(name), 'modules')},
            'uptodate': [False],
            'verbosity': 2,
        }

Here again we used a task-group to create one task per python module. And getargs to get list of module imports that were already calculated on imports task.

Also note the usage of two metadata fields not seen before uptodate and verbosity.

custom uptodate

So far we have seen how doit can determine if a task is up-to-date by taking into account changes to file_dep, if targets exist and results from getargs

While those cover a wide range of use cases, doit also provides a way to specify completely custom checks for up-to-date, using the uptodate field.

In this case the print task actually does not perform any computation, it is being used to display some info to the user. So this task should be always executed.

update will be explained in detail in part 2 of this tutorial. For now suffice to add value False to indicate this task will never be considered up-do-date.

'uptodate': [False],

verbosity

doit output (for command run) consists of:

  • one line with task name (preceded by . or )
  • task output

The actual task output displayed can be controlled by verbosity, there are 3 levels of verbosity.

  • 0: both stdout and stderr from task are NOT displayed
  • 1: only stderr is displayed
  • 2: both stdout and stderr are displayed

The default verbosity is 1.

If print task was executed with default verbosity we would actually not see any output, so we must force its verbosity value to 2.

'verbosity': 2,
$ doit print:requests.models
-- imports:requests.models
.  print:requests.models
requests._internal_utils
requests.auth
requests.compat
requests.cookies
requests.exceptions
requests.hooks
requests.status_codes
requests.structures
requests.utils

Note verbosity can be overwritten from command-line with option -v/--verbosity.

DOIT_CONFIG

There is one last problem to be solved. Since print is used only to display some information, it should not be executed by default. It should be executed only when explicitly asked.

i.e. when you just run doit without any parameters it should create the graph image but not print out the information from print task.

Before I said by default doit run would execute all tasks. That is not exactly true… it will execute all default tasks. The default tasks can be controlled by adding a configuration dictionary with the name DOIT_CONFIG in the dodo.py.

DOIT_CONFIG = {
    'default_tasks': ['imports', 'dot', 'draw'],
}

Apart from default_tasks, DOIT_CONFIG can change the default of any command line option. For example you can globally change task’s verbosity for all tasks with:

DOIT_CONFIG = {
 'default_tasks': ['imports', 'dot', 'draw'],
 'verbosity': 2,
}

Code dodo.py.

And that’s all for part 1 of tutorial :)