Docker
Processing nodes can run models directly on the host system, as Airflow describes. The platform nonetheless encourages each model application to be containerised using Docker.
Why containerise
The handbook gives three reasons, and they are about the system as a whole rather than about any individual model:
| Reason | What it protects |
|---|---|
| Consistent DAG structures | Every model is invoked the same way, whatever it is written in |
| Consistent data management structures | Inputs and outputs cross a defined boundary |
| An organised host system | The host does not accumulate every model's dependencies |
The underlying problem
The models in an IMP have, as the overview puts it, different architectures and computing requirements. MATSim needs a Java Virtual Machine; another model may need a specific Python version, a compiled hydraulic solver or a particular library.
Installing all of them on one host makes that host fragile, and makes the platform hard to rebuild elsewhere. A container gives each model its own environment.
The versioning workflow
With each new model version, three steps follow:
- A Docker container is built
- It is exported to an ISO file
- It is uploaded to connected storage
When the DAG is run, the model version specified by the user is imported, extracted and executed.
Version is a run parameter
Because images are stored per version rather than installed, the model version becomes something the user selects at run time alongside their other configuration.
That is what makes a past scenario re-runnable: the model binary that produced a result is still retrievable, not overwritten by the next release.
Where this appears in a DAG
Two nodes in the MATSim DAG shown on the Airflow page implement this workflow:
| Node | Does |
|---|---|
prepare_image | Imports and extracts the specified model container |
run_matsim | Executes it, using a CustomDockerOperator |
The surrounding nodes, which stage data and post results, run on the host as ordinary Python or Bash operators. The container boundary sits precisely around the model execution.
How this supports lineage
Storing a discrete, versioned image per model release is half of what makes historical lineage possible. The other half is recording which image a given run used, which is handled through metadata and version control.
See version control.