doit is a task management & automation tool
doit comes from the idea of bringing the power of build-tools to execute any kind of task
doit is a modern open-source build-tool written in python designed to be simple to use and flexible to deal with complex work-flows. It is specially suitable for building and managing custom work-flows where there is no out-of-the-box solution available.
doit has been successfully used on: systems test/integration automation, scientific computational pipelines, content generation, configuration management, etc. Check some success stories …
introduction¶
A task describes some computation to be done (actions), and contains some extra meta-data.
def task_example():
return {
'actions': ['myscript'],
'file_dep': ['my_input_file'],
'targets': ['result_file'],
}
actions:
can be external programs (executed as shell commands) or python functions.
a single task may define more than one action.
task meta-data:
task meta-data includes a description of input file (dependencies) for the actions and result files (targets)
there are many other meta-data fields to control how and when a task is executed…
doit uses the task’s meta-data to:
cache task results (aka incremental-builds)
doit checks if the task is up-to-date and skips its execution if the task would produce the same result of a previous execution.
correct execution order
By checking the inter-dependency between tasks doit ensures that tasks will be execute in the correct order.
parallel execution
built-in support for parallel (threaded or multi-process) task execution (more)
Traditional build-tools were created mainly to deal with compile/link process of source code. doit was designed to solve a broader range of tasks.
powerful dependency system
Task’s metadata (actions, dependencies, targets…) are better described in a declarative way, but often you want to create this metadata programmatically.
flexible task definition
doit uses plain python modules to create tasks (and its meta-data)
customizable task definition
By default tasks are described by a python dict. But it can be easily customized. (more)
debugger
Since plain python is used to define your tasks the python debugger (pdb) is available as in any other python application
Other features…
self documented
doit command allows you to list and obtain help/documentation for tasks (more)
inotify integration
built-in support for a long-running process that automatically re-execute tasks based on file changes by external process (linux/mac only) (more)
custom output
process output can be completely customized through reporters (more)
tab-completion
built-in support tab-completion for commands/task (supports bash and zsh) (more)
IPython integration
provide %doit magic function that loads tasks defined directly in IPython’s global namespace (more)
extensible
Apart from using doit to automate your project it also expose its API so you can create new applications/tools using doit functionality (more)
Check the documentation for more features…
What people are saying about doit¶
Congratulations! Your tool follows the KISS principle very closely. I always wondered why build tools had to be that complicated. - Elena
Let me start by saying I’m really lovin doit, at first the interface seemed verbose but quickly changed my mind when I started using it and realized the flexibility. Many thanks for the great software! - Michael Gliwinski
I love all the traditional unix power tools, like cron, make, perl, …, I also like new comprehensive configuration management tools like CFEngine and Puppet. But I find doit to be so versatile and so productive. - Charlie Guo
I went back and forth on different Pythonic build tools for awhile. Scons is pretty great if you’re doing ‘standard’ sorts of builds, but I found it a little heavy for my tastes and really hard to customize to my tool flow (in FPGA land, there are all kinds of nonstandard vendor tools that all need to play together). I have been using doit more and more over the past few months, and I’m continually impressed by the tool (aside from the goofy name). It works amazingly well for automating tricky/exotic build processes. Check it out! SkOink
I needed a sort of ‘make’ tool to glue things together and after trying out all kinds, doit … has actually turned out to be beautiful. Its easy to add and manage tasks, even complex ones– gluing things together with decorators and ‘library’ functions I have written to do certain similar things. - Matthew
Some time ago, I grew frustrated with Make and Ant and started porting my build files to every build tool I found (SCons, Waf, etc.). Each time, as soon as I stepped out of already available rules, I ran into some difficult to overcome stumbling blocks. Then I discovered this little gem of simplicity: doit. It’s Python-based. It does not try to be smart, it does not try to be cool, it just works. If you are looking for a flexible little build tool for different languages and tasks, give it a chance. (…) - lelele
Project Details¶
LICENSE This is an open-source project (MIT license) written in python. Runs on Python 3.4 through 3.6 (including PyPy support). For python 2 support please use doit version 0.29.
DOWNLOAD from PyPi
CONTRIBUTE Please check the community guidelines before asking questions and reporting issues.
DEVELOP Project management (bug tracker, feature requests and source code ) on github.
QUESTIONS on StackOverflow, use tag doit.
FORUM (questions/feedback/discussion) on Google group. Please do not send questions to my private email.
doit projects contains a collection of third-party projects, plugins, extensions, non-trivial examples and re-usable task creators for doit.
This web site is hosted on http://pages.github.com
Professional support and consulting services available from doit creator & maintainer (schettino72 at gmail.com).
Status¶
This blog post explains how everything started in 2008.
doit is under active development. Version 0.31 released on 2018-02.
doit core features are quite stable. If there is no recent development, it does NOT mean doit is not being maintained… The project has 100% unit-test code coverage.
Development is done based on real world use cases. It is well designed and has a small code base, so adding new features is not hard. Contributions are welcome.