doit comes from the idea of bringing the power of build-tools to execute any kind of task


python logo
pip install doit

People often compare doit to tools like make, grunt, rake, scons, snakemake.

They appreciate doit strong features, flexibility, simplicity of authoring and ease of use.

Sample Code

Define functions returning python dict with task's meta-data.

Snippet from tutorial.

def task_imports():
    """find imports from a python module"""
    for name, module in PKG_MODULES.by_name.items():
        yield {
            'name': name,
            'file_dep': [module.path],
            'actions': [(get_imports, (PKG_MODULES, module.path))],
        }

def task_dot():
    """generate a graphviz's dot graph from module imports"""
    return {
        'targets': ['requests.dot'],
        'actions': [module_to_dot],
        'getargs': {'imports': ('imports', 'modules')},
        'clean': True,
    }

def task_draw():
    """generate image from a dot file"""
    return {
        'file_dep': ['requests.dot'],
        'targets': ['requests.png'],
        'actions': ['dot -Tpng %(dependencies)s -o %(targets)s'],
        'clean': True,
    }
$ doit list
dot       generate a graphviz's dot graph from module imports
draw      generate image from a dot file
imports   find imports from a python module
$ doit
.  imports:requests.models
.  imports:requests.__init__
.  imports:requests.help
(...)
.  dot
.  draw


Task Runner

doit allows you to easily define ad-hoc tasks, helping you to organize all your project related tasks in an unified easy-to-use & discoverable way.

python powered

doit uses plain python to define tasks.

Task's meta-data are better described in a declarative way, but often you want to create this meta-data programmatically.

easy authoring

NO API: Tasks are described by a python dict (can also be easily customized)

Tasks can execute external process (shell commands) or python code

debugger & self documented

Since plain python is used to define your tasks the python debugger pdb is available

doit command allows you to list and obtain help/documentation for tasks


Build tool & Pipelines

Simple task runners simply do not scale-up. doit as other build-tools can be much more efficient at repeateadly running tasks.

cache task results
aka incremental-builds

doit creates a DAG and ensures that only required tasks will be executed and in the correct order.

doit checks if the task is up-to-date and skips its execution if the task would produce the same result of a previous execution.

up-to-date check

dependencies can be dynamically calculated by other tasks

the up-to-date check to cache task results is not restricted to looking for file modification on dependencies. Nor requires target files.

Pipelines

Traditional build-tools were created mainly to deal with compile/link process of source code. doit was designed to solve a broader range of workflows.

results from a task can be used by another task without resorting to the creation of intermediate files


Advanced Features

Extensible

Custom output
Command line output can be completely customized through reporters

Plugins
allow you to create/modify sub-commands, storage backend, task loader, and output reporter

Framework
API is exposed so you can create new applications/tools leveraging doit functionality

Batteries included

parallel execution
built-in support for parallel (threaded or multi-process) task execution

watch/auto execution
built-in support watching for file changes and automatically re-execute tasks based on file changes by external process [linux/mac only]

tab-completion
built-in support tab-completion for commands/task (supports bash and zsh)

DAG Visualisation
create task's dependency-graph image using graphviz

IPython
IPython integration provide %doit magic function that loads tasks defined directly in IPython's global namespace

strace
Integration with strace helps you understand effects of third-part commands

Testimonials

Status

doit is under active development. Version 0.31 released on 2018-02.

doit runs on Python 3.4 through 3.6 (including PyPy). For python 2 support please use doit version 0.29.

This blog post explains how everything started in 2008.

doit core features are quite stable. If there is no recent development, it does NOT mean the project is not being maintained... The project has 100% unit-test code coverage.

Development is done based on real world use cases. It is well designed and has a small code base, so adding new features is not hard. Contributions are welcome.

Project Details

LICENSE
This is an open-source project (MIT license) written in python.
DOWNLOAD
from PyPi
SUPPORT
See support page
DEVELOP
Project management (bug tracker, feature requests and source code ) on github
WEBSITE
This web site is hosted on http://pages.github.com
Powered by Universal template and Sphinx.