Making tasks persist#
Sometimes you want to skip the execution of a task and pretend nothing has changed.
A typical scenario is that you formatted the task’s source files with black which would rerun the task.
In this case, you can apply the @pytask.mark.persist
decorator to the task, which will skip its execution as long as all products exist.
Internally, the state of the dependencies, the source file, and the products are updated in the database such that the subsequent execution will skip the task successfully.
When is this useful?#
You ran a formatter like Black on the files in your project and want to prevent the longest-running tasks from being rerun.
You extend a repetition of a task function but do not want to rerun all tasks.
You want to integrate a task that you have already run elsewhere. Copy over the dependencies and products and the task definition and make the task persist.
Caution
This feature can corrupt the integrity of your project. Document why you have applied the decorator out of consideration for yourself and other contributors.
How to do it?#
To create a persisting task, apply the correct decorator, and, et voilà, it is done.
First, we create a task and its dependency.
from pathlib import Path
import pytask
@pytask.mark.persist
def task_make_input_bold(
input_path: Path = Path("input.md"), output_path: Path = Path("output.md")
) -> None:
output_path.write_text("**" + input_path.read_text() + "**")
<!-- Content of input.md. -->
Here is the text.
Running pytask will execute the task since the product is missing.
$ pytask
──────────────────────────── Start pytask session ────────────────────────────
Platform: win32 -- Python <span style="color: var(--termynal-blue)">3.10.0</span>, pytask <span style="color: var(--termynal-blue)">0.4.0</span>, pluggy <span style="color: var(--termynal-blue)">1.0.0</span>
Root: C:\Users\pytask-dev\git\my_project
Collected <span style="color: var(--termynal-blue)">1</span> task.
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┓
┃ Task ┃ Outcome ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━┩
│ <span class="termynal-dim">task_module.py::</span>task_make_input_bold │ <span class="termynal-success">. </span> │
└────────────────────────────────────────────────┴─────────┘
<span class="termynal-dim">──────────────────────────────────────────────────────────────────────────────</span>
<span class="termynal-success">╭───────────</span> <span style="font-weight: bold;">Summary</span> <span class="termynal-success">────────────╮</span>
<span class="termynal-success">│</span> <span style="font-weight: bold;"> 1 Collected tasks </span> <span class="termynal-success">│</span>
<span class="termynal-success">│</span> <span class="termynal-success-textonly"> 1 Succeeded (100.0%) </span> <span class="termynal-success">│</span>
<span class="termynal-success">╰────────────────────────────────╯</span>
<span class="termynal-success">───────────────────────── Succeeded in 0.07 seconds ──────────────────────────</span>
After that, we accidentally changed the task’s source file by formatting the file with
Black. Without the @pytask.mark.persist decorator, the
task would run again since the source has changed. With the decorator, a green p signals
that the execution is skipped.
$ pytask
──────────────────────────── Start pytask session ────────────────────────────
Platform: win32 -- Python <span style="color: var(--termynal-blue)">3.10.0</span>, pytask <span style="color: var(--termynal-blue)">0.4.0</span>, pluggy <span style="color: var(--termynal-blue)">1.0.0</span>
Root: C:\Users\pytask-dev\git\my_project
Collected <span style="color: var(--termynal-blue)">1</span> task.
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┓
┃ Task ┃ Outcome ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━┩
│ <span class="termynal-dim">task_module.py::</span>task_make_input_bold │ <span class="termynal-success">p </span> │
└────────────────────────────────────────────────┴─────────┘
<span class="termynal-dim">──────────────────────────────────────────────────────────────────────────────</span>
<span class="termynal-success">╭───────────</span> <span style="font-weight: bold;">Summary</span> <span class="termynal-success">────────────╮</span>
<span class="termynal-success">│</span> <span style="font-weight: bold;"> 1 Collected tasks </span> <span class="termynal-success">│</span>
<span class="termynal-success">│</span> <span class="termynal-success-textonly"> 1 Persisted (100.0%) </span> <span class="termynal-success">│</span>
<span class="termynal-success">╰────────────────────────────────╯</span>
<span class="termynal-success">────────────────────────── Succeeded in 0.0 seconds ──────────────────────────</span>
If we rerun the task, it is skipped because nothing has changed and not because it is
marked with @pytask.mark.persist.
$ pytask --verbose 2
──────────────────────────── Start pytask session ────────────────────────────
Platform: win32 -- Python <span style="color: var(--termynal-blue)">3.10.0</span>, pytask <span style="color: var(--termynal-blue)">0.4.0</span>, pluggy <span style="color: var(--termynal-blue)">1.0.0</span>
Root: C:\Users\pytask-dev\git\my_project
Collected <span style="color: var(--termynal-blue)">1</span> task.
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┓
┃ Task ┃ Outcome ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━┩
│ <span class="termynal-dim">task_module.py::</span>task_make_input_bold │ <span class="termynal-success">s </span> │
└────────────────────────────────────────────────┴─────────┘
<span class="termynal-dim">──────────────────────────────────────────────────────────────────────────────</span>
<span class="termynal-success">╭────────────────</span> <span style="font-weight: bold;">Summary</span> <span class="termynal-success">─────────────────╮</span>
<span class="termynal-success">│</span> <span style="font-weight: bold;"> 1 Collected tasks </span> <span class="termynal-success">│</span>
<span class="termynal-success">│</span> <span class="termynal-success-textonly"> 1 Skipped because unchanged (100.0%) </span> <span class="termynal-success">│</span>
<span class="termynal-success">╰──────────────────────────────────────────╯</span>
<span class="termynal-success">────────────────────────── Succeeded in 0.0 seconds ──────────────────────────</span>