doit logo

Table Of Contents

Sponsors

Your logo here

Sponsor/Donate

Gratipay weekly funding goal is $?.?? USD.
$?.??

Why Donate? Help keep the project alive! Donations will be used for project maintanance tasks (bug fixes, merge patches, reply users).

For corporate donations the logo of your company will be placed on this website side-bar (see above). For more information please contact schettino72@gmail.com

Hire me

Looking for a python developer with a proven record of designing and developing robust and well tested applications? The creator and maintainer of doit is available for hire. Full-time, part-time or one-off job.
Contact: schettino72@gmail.com



More on dependencies

uptodate

Apart from file dependencies you can extend doit to support other ways to determine if a task is up-to-date through the attribute uptodate.

This can be used in cases where you need to some kind of calculation to determine if the task is up-to-date or not.

uptodate is a list where each element can be True, False, None, a callable or a command(string).

  • False indicates that the task is NOT up-to-date
  • True indicates that the task is up-to-date
  • None values will just be ignored. This is used when the value is dynamically calculated

Note

An uptodate value equal to True does not override others up-to-date checks. It is one more way to check if task is not up-to-date.

i.e. if uptodate==True but a file_dep changes the task is still considered not up-to-date.

If an uptodate item is a string it will be executed on the shell. If the process exits with the code 0, it is considered as up-to-date. All other values would be considered as not up-to-date.

uptodate elements can also be a callable that will be executed on runtime (not when the task is being created). The section custom-uptodate will explain in details how to extend doit writing your own callables for uptodate. This callables will typically compare a value on the present time with a value calculated on the last successful execution.

Note

There is no guarantee uptodate callables or commands will be executed. doit short-circuit the checks, if it is already determined that the task is no up-to-date it will not execute remaining uptodate checks.

doit includes several implementations to be used as uptodate. They are all included in module doit.tools and will be discussed in detail later:

  • result_dep: check if the result of another task has changed
  • run_once: execute a task only once (used for tasks without dependencies)
  • timeout: indicate that a task should “expire” after a certain time interval
  • config_changed: check for changes in a “configuration” string or dictionary
  • check_timestamp_unchanged(): check access, status change/create or modify timestamp of a given file/directory

doit up-to-date definition

A task is not up-to-date if any of:

  • an uptodate item is (or evaluates to) False
  • a file is added to or removed from file_dep
  • a file_dep changed since last successful execution
  • a target path does not exist
  • a task has no file_dep and uptodate item equal to True

It means that if a task does not explicitly define any input (dependency) it will never be considered up-to-date.

Note that since a target represents an output of the task, a missing target is enough to determine that a task is not up-to-date. But its existence by itself is not enough to mark a task up-to-date.

In some situations, it is useful to define a task with targets but no dependencies. If you want to re-execute this task only when targets are missing you must explicitly add a dependency: you could add a uptodate with True value or use run_once() to force at least one execution managed by doit. Example:

def task_touch():
    return {
        'actions': ['touch foo.txt'],
        'targets': ['foo.txt'],
        # force doit to always mark the task
        # as up-to-date (unless target removed)
        'uptodate': [True],
        }

Apart from file_dep and uptodate used to determine if a task is up-to-date or not, doit also includes other kind of dependencies (introduced below) to help you combine tasks so they are executed in appropriate order.

task-dependency

It is used to enforce tasks are executed on the desired order. By default tasks are executed on the same order as they were defined in the dodo file. To define a dependency on another task use the task name (whatever comes after task_ on the function name) in the “task_dep” attribute.

Note

A task-dependency only indicates that another task should be “executed” before itself. The task-dependency might not really be executed if it is up-to-date.

Note

task-dependencies are not used to determine if a task is up-to-date or not. If a task defines only task-dependency it will always be executed.

This example we make sure we include a file with the latest revision number of the mercurial repository on the tar file.

def task_tar():
    return {'actions': ["tar -cf foo.tar *"],
            'task_dep':['version'],
            'targets':['foo.tar']}

def task_version():
    return {'actions': ["hg tip --template '{rev}' > revision.txt"]}
$ doit
.  version
.  tar

groups

You can define a group of tasks by adding tasks as dependencies and setting its actions to None.

def task_foo():
    return {'actions': ["echo foo"]}

def task_bar():
    return {'actions': ["echo bar"]}

def task_mygroup():
    return {'actions': None,
            'task_dep': ['foo', 'bar']}

Note that tasks are never executed twice in the same “run”.

calculated-dependencies

Calculation of dependencies might be an expensive operation, so not suitable to be done on load time by task-creators. For this situation it is better to delegate the calculation of dependencies to another task. The task calculating dependencies must have a python-action returning a dictionary with file_dep, task_dep, uptodate or another calc_dep.

On the example below mod_deps prints on the screen all direct dependencies from a module. The dependencies itself are calculated on task get_dep (note: get_dep has a fake implementation where the results are taken from a dict).

DOIT_CONFIG = {'verbosity': 2}

MOD_IMPORTS = {'a': ['b','c'],
               'b': ['f','g'],
               'c': [],
               'f': ['a'],
               'g': []}


def print_deps(mod, dependencies):
    print "%s -> %s" % (mod, dependencies)
def task_mod_deps():
    """task that depends on all direct imports"""
    for mod in MOD_IMPORTS.iterkeys():
        yield {'name': mod,
               'actions': [(print_deps,(mod,))],
               'file_dep': [mod],
               'calc_dep': ["get_dep:%s" % mod],
               }

def get_dep(mod):
    # fake implementation
    return {'file_dep': MOD_IMPORTS[mod]}
def task_get_dep():
    """get direct dependencies for each module"""
    for mod in MOD_IMPORTS.iterkeys():
        yield {'name': mod,
               'actions':[(get_dep,[mod])],
               'file_dep': [mod],
               }

setup-task

Some tasks may require some kind of environment setup. In this case they can define a list of “setup” tasks.

  • the setup-task will be executed only if the task is to be executed (not up-to-date)
  • setup-tasks are just normal tasks that follow all other task behavior

Note

A task-dependency is executed before checking if the task is up-to-date. A setup-task is executed after the checking if the task is up-to-date and it is executed only if the task is not up-to-date and will be executed.

teardown

Task may also define ‘teardown’ actions. These actions are executed after all tasks have finished their execution. They are executed in reverse order their tasks were executed.

Example:

### task setup env. good for functional tests!
DOIT_CONFIG = {'verbosity': 2,
               'default_tasks': ['withenvX', 'withenvY']}

def start(name):
    print "start %s" % name
def stop(name):
    print "stop %s" % name

def task_setup_sample():
    for name in ('setupX', 'setupY'):
        yield {'name': name,
               'actions': [(start, (name,))],
               'teardown': [(stop, (name,))],
               }

def task_withenvX():
    for fin in ('a','b','c'):
        yield {'name': fin,
               'actions':['echo x %s' % fin],
               'setup': ['setup_sample:setupX'],
               }

def task_withenvY():
    return {'actions':['echo y'],
            'setup': ['setup_sample:setupY'],
            }
$ doit withenvX
.  setup_sample:setupX
start setupX
.  withenvX:c
x c
.  withenvX:b
x b
.  withenvX:a
x a
stop setupX
$ doit withenvY
.  setup_sample:setupY
start setupY
.  withenvY
y
stop setupY

saving computed values

Tasks can save computed values by returning a dictionary on it’s python-actions. The values must be JSON encodable.

A cmd-action can also save it’s output. But for this you will need to explicitly import CmdAction and set its save_out parameter with the name used to save the output in values

from doit.action import CmdAction

def task_save_output():
    return {
        'actions': [CmdAction("echo x1", save_out='out')],
        }
# The task values will contain: {'out': u'x1'}

These values can be used on uptodate and getargs. Check those sections for examples.

getargs

getargs provides a way to use values computed from one task in another task. The values are taken from “saved computed values” (returned dict from a python-action).

For cmd-action use dictionary-based string formatting.

For python-action the action callable parameter names must match with keys from getargs.

getargs is a dictionary where the key is the argument name used on actions, and the value is a tuple with 2 strings: task name, “value name”.

DOIT_CONFIG = {'default_tasks': ['use_cmd', 'use_python']}

def task_compute():
   def comp():
       return {'x':5,'y':10, 'z': 20}
   return {'actions': [(comp,)]}


def task_use_cmd():
   return {'actions': ['echo x=%(x)s, z=%(z)s'],
           'getargs': {'x': ('compute', 'x'),
                       'z': ('compute', 'z')},
           'verbosity': 2,
           }


def task_use_python():
  return {'actions': [show_getargs],
          'getargs': {'x': ('compute', 'x'),
                      'y': ('compute', 'z')},
          'verbosity': 2,
          }
def show_getargs(x, y):
   print "this is x:%s" % x
   print "this is y:%s" % y

The values are being passed on to a python-action you can pass the whole dict by specifying the value name as None.

def task_compute():
   def comp():
       return {'x':5,'y':10, 'z': 20}
   return {'actions': [(comp,)]}


def show_getargs(values):
   print values

def task_args_dict():
  return {'actions': [show_getargs],
          'getargs': {'values': ('compute', None)},
          'verbosity': 2,
          }

If a group-task is used, the values from all its sub-tasks are passed as a dict.

def task_compute():
   def comp(x):
       return {'x':x}
   yield {'name': '5',
          'actions': [ (comp, [5]) ]
          }
   yield {'name': '7',
          'actions': [ (comp, [7]) ]
          }


def show_getargs(values):
    print values
    assert sum(v['x'] for v in values.itervalues()) == 12

def task_args_dict():
  return {'actions': [show_getargs],
          'getargs': {'values': ('compute', None)},
          'verbosity': 2,
          }

Note

getargs creates an implicit setup-task.

keywords on actions

It is common situation to use task information such as targets, dependencies, or changed in its own actions. Note: Dependencies here refers only to file-dependencies.

For cmd-action you can use the python notation for keyword substitution on strings. The string will contain all values separated by a space (” ”).

For python-action create a parameter in the function, doit will take care of passing the value when the function is called. The values are passed as list of strings.

def task_hello():
    """hello"""

    def python_hello(targets):
        with open(targets[0], "a") as output:
            output.write("Python says Hello World!!!\n")

    return {
        'actions': [python_hello],
        'targets': ["hello.txt"],
        }

You can also pass the keyword task to have a reference to all task metadata.

def who(task):
    print 'my name is', task.name
    print task.targets

def task_x():
    return {
        'actions': [who],
        'targets': ['asdf'],
        'verbosity': 2,
        }