Data storage
The problem
The models used within the DARe IMP rely on filesystem storage rather than any form of database. Their outputs can also be significant in size, which makes database compatibility a challenge.
How significant
The UrbClim chapter gives a concrete figure: a single NetCDF output file of roughly 7 GB per scenario.
Files of that size are not what relational databases are designed to hold, and the models producing them expect to read and write files, not rows.
There is also a second requirement beyond storage. Model outputs need to be accessible to researchers across DARe, not just to the machine that produced them.
The solution: object storage
Model output data files are transformed into Binary Large Objects (BLOBs) and uploaded to MinIO, a locally hosted object storage system.
| What is stored | Model output data files, as BLOBs |
| Where | MinIO, locally hosted |
| Accessed via | An S3 compatible API |
Object storage suits this shape of data
Object storage is built for large, write-once, read-many files addressed by key. That is exactly what model outputs are.
It also gives every output a stable address, which is what allows researchers to retrieve data directly without going through the pipeline that produced it.
The S3 API, and why it matters
Airflow interacts with MinIO via an S3 compatible API.
The consequence is portability: the current mechanism by which Airflow uploads and downloads data can also facilitate Amazon S3 cloud storage.
A local store today, a cloud store tomorrow, no rewrite
Because MinIO speaks the same API as Amazon S3, the storage layer can move from locally hosted to cloud without changing how Airflow talks to it.
This is why the architecture diagram labels that component a "Local/Cloud Storage Server" rather than committing to either.
Metadata
Alongside the object store, a PostgreSQL instance provides database storage for detailed metadata.
The division of responsibility is clean:
| Store | Holds | Suits |
|---|---|---|
| MinIO | The output files themselves | Large binary objects |
| PostgreSQL | Detailed metadata about them | Structured, queryable records |
Metadata is what makes the object store navigable
An object store is excellent at holding a file and poor at answering "which run produced this, with which model version, under which scenario?"
Pairing it with a relational metadata store is what turns a pile of BLOBs into a searchable record. Combined with version control, this is what delivers historical lineage. See version control.
Backup and encryption
The architecture diagram shows both the metadata database and the object store replicating to an external backup server, and annotates the object database as encrypted at rest.
These arrangements appear in the figure rather than the handbook text, and should be documented properly in the source.