IMP Orchestration
Every preceding chapter describes a model. This one describes the problem of running them together.
Why an IMP becomes complex
An IMP can consist of any number of data sources, models, and transitional steps between models.
The difficulty is that models are usually developed in isolation from one another. They have different purposes and rely on different technologies, which makes them dissimilar in several ways at once:
| They differ in | Consequence |
|---|---|
| Architecture | No shared way of being invoked |
| Computing requirements | Different runtimes, libraries and hardware needs |
| Interfaces | No common calling convention |
| User accessibility | Different skills needed to operate each one |
| Input and output data criteria | Formats that do not line up |
On top of that, the transitional programs written to manipulate data between models are often bespoke, written once for one pairing.
Consequently, IMPs tend to be complex. That complexity produces five concrete problems:
- Standardisation
- Efficient data management
- Model setup
- Model execution
- Version control
This chapter is about the joins, not the models
Read through the handbook so far and the framework looks tidy: hazard models feed transport models, which feed metrics.
In practice each of those arrows is a piece of bespoke code moving files between programs that know nothing about each other. Orchestration is the discipline of making those arrows into a system rather than a collection of scripts.
The response
To address this complexity, a prototype orchestration platform has been developed, with Apache Airflow at its core.
The platform addresses each of the five problems above through a different component:
| Problem | Addressed by | Covered on |
|---|---|---|
| Model execution and standardisation | Airflow DAGs and processing nodes | Airflow |
| Model setup and environment consistency | Docker containerisation | Docker |
| Efficient data management | MinIO object storage and PostgreSQL metadata | Data storage |
| Version control and lineage | Git, combined with metadata | Version control |
The overall shape of the system is described on the architecture page.
What orchestration achieves
In building an orchestration platform, the complex and dynamic systems, processes and data or model linkages that are ordinarily performed manually throughout an IMP become embedded and codified into a working system.
In such a system, model automation also presents as a possibility, along with subsequent improvements in accessibility through ease of use.
What it does not replace
Human interaction is indispensable
The handbook is careful to qualify the automation claim, and the qualification deserves as much weight as the claim.
Oftentimes an intimate understanding of a model is required to operate it sufficiently for sound results. In terms of operation, validation and maintenance, human interaction in an orchestration platform is indispensable.
Automation removes the manual labour of moving data between models. It does not remove the expertise needed to judge whether the results are sound.
That distinction matters for how the platform should be presented to stakeholders. It lowers the operational barrier to running the IMP; it does not lower the expertise barrier to interpreting it.
What this chapter covers
| Page | What you will find |
|---|---|
| Architecture | How the servers, storage and code repository fit together |
| Airflow | DAGs, processing nodes and DAG stacking |
| Docker | Containerising models, and versioned images |
| Data storage | MinIO, BLOBs, the S3 API and metadata |
| Version control | Git and historical lineage |
| Further reading | The DAReflow repository |