Skip to content

Architecture

The orchestration platform is split across three servers, with code pulled from a public repository and users accessing two points in the system.

Orchestration architecture diagram. At the top left, an External Server holds a data backup. Beside it, a Local or Cloud Storage Server contains a Metadata DB running PostgreSQL and an S3 Object Database holding buckets labelled A, B and C, annotated as encrypted at rest; both send copies to the data backup. Below, a Deployment Server contains the Orchestrator running Airflow, which drives MATSim, CityCAT, FloodEvent and further models. An Airflow bot moves data between the Deployment Server and the storage server, downloading and uploading. At the bottom, a GitHub Open Code repository is pulled into the Deployment Server. On the right, User Access connects both to the Orchestrator and to the S3 Object Database.

Figure 3. Orchestration architecture.

The three servers

ServerHolds
Deployment ServerThe Orchestrator (Airflow) and the models it drives: MATSim, CityCAT, FloodEvent and others
Local or Cloud Storage ServerThe Metadata DB (PostgreSQL) and the S3 Object Database
External ServerA data backup, receiving copies from both stores

Separating compute from storage

The models run in one place and their data lives in another. That separation is what makes the storage layer swappable, and it is why the same mechanism works against either a locally hosted object store or cloud storage. See data storage.

How data moves

An Airflow bot downloads and uploads data between the Deployment Server and the storage server. Models running on the deployment side fetch their inputs and write their outputs through it, rather than reading and writing storage directly.

Where code comes from

The Deployment Server pulls from a GitHub repository of open code. The platform itself is published at DAReHub/DAReflow.

This is the version-control route in practice: what runs on the deployment server is whatever the repository says it should be. See version control.

How users interact

The diagram shows two user entry points, not one:

Entry pointPurpose
The OrchestratorConfiguring and running DAGs
The S3 Object DatabaseRetrieving model outputs directly

Why direct data access matters

Researchers across DARe need model outputs without needing to operate the models.

Giving them a route to the object store that does not pass through Airflow is what makes the outputs genuinely shared rather than locked behind the pipeline that produced them.

Security

The diagram annotates the object database as encrypted at rest, and both the metadata database and the object store replicate to an external backup server.

Noted from the figure

The encryption and backup arrangements appear in the architecture diagram but are not described in the handbook's text. They are recorded here because they are operationally significant, and should be written up properly in the source.