doit logo

Table Of Contents

Sponsors

Your logo here

Sponsor/Donate


Why Donate? Donations will be used to sponsor further development of the project.

For corporate donations the logo of your company will be placed on this website side-bar (see above). For more information please contact schettino72@gmail.com

Hire me

Looking for a python developer with a proven record of designing and developing robust and well tested applications? The creator and maintainer of doit is available for hire. Full-time, part-time or one-off job.
Contact: schettino72@gmail.com



Tasks

Intro

doit is all about automating task dependency management and execution. Tasks can execute external shell commands/scripts or python functions (actually any callable). So a task can be anything you can code :)

Tasks are defined in plain python module with some conventions.

Note

You should be comfortable with python basics. If you don’t know python yet check Python tutorial.

A function that starts with the name task_ defines a task-creator recognized by doit. These functions must return (or yield) dictionaries representing a task. A python module/file that defines tasks for doit is called dodo file (that is something like a Makefile for make).

Take a look at this example (file dodo.py):

def task_hello():
    """hello"""

    def python_hello(targets):
        with open(targets[0], "a") as output:
            output.write("Python says Hello World!!!\n")

    return {
        'actions': [python_hello],
        'targets': ["hello.txt"],
        }

When doit is executed without any parameters it will look for tasks in a file named dodo.py in the current folder and execute its tasks.

$ doit
.  hello

On the output it displays which tasks were executed. In this case the dodo file has only one task, hello.

Actions

Every task must define actions. It can optionally define other attributes like targets, file_dep, verbosity, doc ...

Actions define what the task actually does. Actions is always a list that can have any number of elements. The actions of a task are always run sequentially. There 2 basic kinds of actions: cmd-action and python-action. The action “result” is used to determine if task execution was successful or not.

python-action

If action is a python callable or a tuple (callable, *args, **kwargs) - only callable is required. The callable must be a function, method or callable object. Classes and built-in functions are not allowed. args is a sequence and kwargs is a dictionary that will be used as positional and keywords arguments for the callable. see Keyword Arguments.

The result of the task is given by the returned value of the action function. So it must return a boolean value True, None, a dictionary or a string to indicate successful completion of the task. Use False to indicate task failed. If it raises an exception, it will be considered an error. If it returns any other type it will also be considered an error but this behavior might change in future versions.

def task_hello():
    """hello py """

    def python_hello(times, text, targets):
        with open(targets[0], "a") as output:
            output.write(times * text)

    return {'actions': [(python_hello, [3, "py!\n"])],
            'targets': ["hello.txt"],
            }

The function task_hello is a task-creator, not the task itself. The body of the task-creator function is always executed when the dodo file is loaded.

Note

The body of task-creators are executed even if the task is not going to be executed. The body of task-creators should be used to create task metadata only, not execute tasks! From now on when the documentation says that a task is executed, read “the task’s actions are executed”.

cmd-action

CmdAction’s are executed in a subprocess (using python subprocess.Popen).

If action is a string, the command will be executed through the shell. (Popen argument shell=True).

Note that the string must be escaped according to python string formatting.

It is easy to include dynamic (on-the-fly) behavior to your tasks with python code from the dodo file. Let’s take a look at another example:

def task_hello():
    """hello cmd """
    msg = 3 * "hi! "
    return {
        'actions': ['echo %s ' % msg + ' > %(targets)s',],
        'targets': ["hello.txt"],
        }

Note

The body of the task-creator is always executed, so in this example the line msg = 3 * “hi! “ will always be executed.

If action is a list of strings, by default it will be executed without the shell (Popen argument shell=False).

def task_python_version():
    return {
        'actions': [['python', '--version']]
        }

For complex commands it is also possible to pass a callable that returns the command string. In this case you must explicit import CmdAction.

from doit.action import CmdAction

def task_hello():
    """hello cmd """

    def create_cmd_string():
        return "echo hi"

    return {
        'actions': [CmdAction(create_cmd_string)],
        'verbosity': 2,
        }

You might also explicitly import CmdAction in case you want to pass extra parameters to Popen like cwd. All keyword parameter from Popen can be used on CmdAction (except stdout and stderr).

Note

Different from subprocess.Popen, CmdAction shell argument defaults to True. All other Popen arguments can also be passed in CmdAction except stdout and stderr

The result of the task follows the shell convention. If the process exits with the value 0 it is successful. Any other value means the task failed.

custom actions

It is possible to create other type of actions, check tools.LongRunning as an example.

task name

By default a task name is taken from the name of the python function that generates the task. For example a def task_hello would create a task named hello.

It is possible to explicitly set a task name with the parameter basename.

def task_hello():
    return {
        'actions': ['echo hello']
        }

def task_xxx():
    return {
        'basename': 'hello2',
        'actions': ['echo hello2']
        }
$ doit
.  hello
.  hello2

When explicit using basename the task-creator is not limited to create only one task. Using yield it can generate several tasks at once. It is also possible to yield a generator that generate tasks. This is useful to write some generic/reusable task-creators.

def gen_many_tasks():
    yield {'basename': 't1',
           'actions': ['echo t1']}
    yield {'basename': 't2',
           'actions': ['echo t2']}

def task_all():
    yield gen_many_tasks()
$ doit
.  t2
.  t1

sub-tasks

Most of the time we want to apply the same task several times in different contexts.

The task function can return a python-generator that yields dictionaries. Since each sub-task must be uniquely identified it requires an additional field name.

def task_create_file():
    for i in range(3):
        filename = "file%d.txt" % i
        yield {'name': filename,
               'actions': ["touch %s" % filename]}
$ doit
.  create_file:file0.txt
.  create_file:file1.txt
.  create_file:file2.txt

avoiding empty sub-tasks

If you are not sure sub-tasks will be created for a given basename but you want to make sure that a task exist, you can yield a sub-task with name equal to None. This can also used to set the task doc and watch attribute.

import glob

def task_xxx():
    """my doc"""
    LIST = glob.glob('*.xyz') # might be empty
    yield {
        'basename': 'do_x',
        'name': None,
        'doc': 'docs for X',
        'watch': ['.'],
        }
    for item in LIST:
        yield {
            'basename': 'do_x',
            'name': item,
            'actions': ['echo %s' % item]
            }
$ doit
$ doit list
do_x   docs for X

Dependencies & Targets

One of the main ideas of doit (and other build-tools) is to check if the tasks/targets are up-to-date. In case there is no modification in the dependencies and the targets already exist, it skips the task execution to save time, as it would produce the same output from the previous run.

Dependency
A dependency indicates an input to the task execution.
Target
A target is the result/output file produced by the task execution.

i.e. In a compilation task the source file is a file_dep, the object file is a target.

def task_compile():
    return {'actions': ["cc -c main.c"],
            'file_dep': ["main.c", "defs.h"],
            'targets': ["main.o"]
            }

doit automatically keeps track of file dependencies. It saves the signature (MD5) of the dependencies every time the task is completed successfully.

So if there are no modifications to the dependencies and you run doit again. The execution of the task’s actions is skipped.

$ doit
.  compile
$ doit
-- compile

Note the -- (2 dashes, one space) on the command output on the second time it is executed. It means, this task was up-to-date and not executed.

file_dep (file dependency)

Different from most build-tools dependencies are on tasks, not on targets. So doit can take advantage of the “execute only if not up-to-date” feature even for tasks that don’t define targets.

Let’s say you work with a dynamic language (python in this example). You don’t need to compile anything but you probably want to apply a lint-like tool (i.e. pyflakes) to your source code files. You can define the source code as a dependency to the task.

def task_checker():
    return {'actions': ["pyflakes sample.py"],
            'file_dep': ["sample.py"]}
$ doit
.  checker
$ doit
-- checker

Note the -- again to indicate the execution was skipped.

Traditional build-tools can only handle files as “dependencies”. doit has several ways to check for dependencies, those will be introduced later.

targets

Targets can be any file path (a file or folder). If a target doesn’t exist the task will be executed. There is no limitation on the number of targets a task may define. Two different tasks can not have the same target.

Lets take the compilation example again.

def task_compile():
    return {'actions': ["cc -c main.c"],
            'file_dep': ["main.c", "defs.h"],
            'targets': ["main.o"]
            }
  • If there are no changes in the dependency the task execution is skipped.
  • But if the target is removed the task is executed again.
  • But only if it does not exist. If the target is modified but the dependencies do not change the task is not executed again.
$ doit
.  compile
$ doit
-- compile
$ rm main.o
$ doit
.  compile
$ echo xxx > main.o
$ doit
-- compile

execution order

If your tasks interact in a way where the target (output) of one task is a file_dep (input) of another task, doit will make sure your tasks are executed in the correct order.

def task_modify():
    return {'actions': ["echo bar > foo.txt"],
            'file_dep': ["foo.txt"],
            }

def task_create():
    return {'actions': ["touch foo.txt"],
            'targets': ["foo.txt"]
            }
$ doit
.  create
.  modify

Task selection

By default all tasks are executed in the same order as they were defined (the order may change to satisfy dependencies). You can control which tasks will run in 2 ways.

Another example

DOIT_CONFIG = {'default_tasks': ['t3']}

def task_t1():
    return {'actions': ["touch task1"],
            'targets': ['task1']}

def task_t2():
    return {'actions': ["echo task2"]}

def task_t3():
    return {'actions': ["echo task3"],
            'file_dep': ['task1']}

DOIT_CONFIG -> default_tasks

dodo file defines a dictionary DOIT_CONFIG with default_tasks, a list of strings where each element is a task name.

$ doit
.  t1
.  t3

Note that only the task t3 was specified to be executed by default. But its dependencies include a target of another task (t1). So that task was automatically executed also.

command line selection

From the command line you can control which tasks are going to be execute by passing its task name. Any number of tasks can be passed as positional arguments.

$ doit t2
.  t2

You can also specify which task to execute by its target:

$ doit task1
.  t1

sub-task selection

You can select sub-tasks from the command line specifying its full name.

def task_create_file():
    for i in range(3):
        filename = "file%d.txt" % i
        yield {'name': filename,
               'actions': ["touch %s" % filename]}
$ doit create_file:file2.txt
.  create_file:file2.txt

wildcard selection

You can also select tasks to be executed using a glob like syntax (it must contains a *).

$ doit create_file:file*
.  create_file:file1.txt
.  create_file:file2.txt
.  create_file:file3.txt

arguments

It is possible to pass option parameters to the task through the command line.

Just add a params field to the task dictionary. params must be a list of dictionaries where every entry is an option parameter. Each parameter must define a name, and a default value. It can optionally define a “short” and “long” names to be used from the command line (it follows unix command line conventions). It may also specify additional attributes, such as type and help (see below).

See the example:

def task_py_params():
    def show_params(param1, param2):
        print param1
        print 5 + param2
    return {'actions':[(show_params,)],
            'params':[{'name':'param1',
                       'short':'p',
                       'default':'default value'},

                      {'name':'param2',
                       'long':'param2',
                       'type': int,
                       'default':0}],
            'verbosity':2,
            }

def task_cmd_params():
    return {'actions':["echo mycmd %(flag)s xxx"],
            'params':[{'name':'flag',
                       'short':'f',
                       'long': 'flag',
                       'default': '',
                       'help': 'helpful message about this flag'}],
            'verbosity': 2
            }

For python-actions the python function must define arguments with the same name as a task parameter.

$ doit py_params -p abc --param2 4
.  py_params
abc
9

For cmd-actions use python string substitution notation:

$ doit cmd_params -f "-c --other value"
.  cmd_params
mycmd -c --other value xxx

All parameters attributes

Here is the list of all attributes param accepts:

name

Name of the parameter, identifier used as name of the the parameter on python code. It should be unique among others.

required:True
type:str
default

Default value used when it is set through command-line.

required:True
short

Short parameter form, used for e.g. -p value.

required:optional
type:str
long

Long parameter form, used for e.g. --parameter value when it differs from its name.

required:optional
type:str
type

Actually it can be any python callable. It coverts the string value received from command line to whatever value to be used on python code.

If the type is bool the parameter is treated as an option flag where no value should be specified, value is set to True. Example: doit mytask --flag.

required:optional
type:callable (e.g. a function)
default:str
help

Help message associated to this parameter, shown when help is called for this task, e.g. doit help mytask.

required:optional
type:str
inverse

[only for bool parameter] Set inverse flag long parameter name, value will be set to False (see example below).

required:optional
type:str

Example, given following code:

def task_with_flag():
    def _task(flag):
        print "Flag {0}".format("On" if flag else "Off")

    return {
        'params': [{
            'name': 'flag',
            'long': 'flagon',
            'short': 'f',
            'type': bool,
            'default': True,
            'inverse': 'flagoff'}],
        'actions': [(_task, )],
        'verbosity': 2
        }

calls to task with_flag show flag on or off:

$ doit with_flag
.  with_flag
Flag On
$ doit with_flag --flagoff
.  with_flag
Flag Off

positional arguments

Tasks might also get positional arguments from the command line as standard unix commands do, with positional arguments after optional arguments.

def task_pos_args():
    def show_params(param1, pos):
        print('param1 is: {0}'.format(param1))
        for index, pos_arg in enumerate(pos):
            print('positional-{0}: {1}'.format(index, pos_arg))
    return {'actions':[(show_params,)],
            'params':[{'name':'param1',
                       'short':'p',
                       'default':'default value'},
                      ],
            'pos_arg': 'pos',
            'verbosity': 2,
            }
$ doit pos_args -p 4 foo bar
.  pos_args
param1 is: 4
positional-0: foo
positional-1: bar

Warning

If a task accepts positional arguments, it is not allowed to pass other tasks after it in the command line. For example if task1 takes positional arguments you can not call:

$ doit task1 pos1 task2

As the string task2 would be interpreted as positional argument from task1 not as another task name.

command line variables (doit.get_var)

It is possible to pass variable values to be used in dodo.py from the command line.

from doit import get_var

config = {"abc": get_var('abc', 'NO')}

def task_echo():
    return {'actions': ['echo hi %s' % config],
            'verbosity': 2,
            }
$ doit
.  echo
hi {abc: NO}
$ doit abc=xyz x=3
.  echo
hi {abc: xyz}

private/hidden tasks

If task name starts with an underscore ‘_’, it will not be included in the output.

title

By default when you run doit only the task name is printed out on the output. You can customize the output passing a “title” function to the task:

def show_cmd(task):
    return "executing... %s" % task.name

def task_custom_display():
    return {'actions':['echo abc efg'],
            'title': show_cmd}
$ doit
.  executing... Cmd: echo abc efg

verbosity

By default the stdout from a task is captured and its stderr is sent to the console. If the task fails or there is an error the stdout and a traceback (if any) is displayed.

There are 3 levels of verbosity:

0:
capture (do not print) stdout/stderr from task.
1 (default):
capture stdout only.
2:
do not capture anything (print everything immediately).

You can control the verbosity by:

  • task attribute verbosity
def task_print():
    return {'actions': ['echo hello'],
            'verbosity': 2}
$ doit
.  print
hello

custom task definition

Apart from collect functions that start with the name task_. The doit loader will also execute the create_doit_tasks callable from any object that contains this attribute.

def make_task(func):
    """make decorated function a task-creator"""
    func.create_doit_tasks = func
    return func

@make_task
def sample():
    return {
        'verbosity': 2,
        'actions': ['echo hi'],
        }

The project letsdoit has some real-world implementations.

For simple examples to help you create your own check this blog post.

importing tasks

The doit loader will look at all objects in the namespace of the dodo. It will look for functions staring with task_ and objects with create_doit_tasks. So it is also possible to load task definitions from other modules just by importing them into your dodo file.

# import task_ functions
from get_var import task_echo

# import tasks with create_doit_tasks callable
from custom_task_def import sample


def task_hello():
    return {'actions': ['echo hello']}
$ doit list
echo
hello
sample

Note

Importing tasks from different modules is useful if you want to split your task definitions in different modules.

The best way to create re-usable tasks that can be used in several projects is to call functions that return task dict’s. For example take a look at a reusable pyflakes task generator. Check the project doit-py for more examples.