Skip to content

Architecture

This document describes the architecture and design principles of the EOPF GeoZarr library.

Overview

The EOPF GeoZarr library is designed to convert EOPF (Earth Observation Processing Framework) datasets to GeoZarr format while maintaining scientific accuracy and optimizing performance.

This implementation follows our GeoZarr Mini Spec, which defines the specific subset of the GeoZarr specification that this library implements, including implementation-specific details for chunking, CF compliance, and multiscale dataset organization.

Design Principles

1. Scientific Integrity First

  • Native CRS Preservation: Maintains original coordinate reference systems to avoid reprojection artifacts
  • Data Accuracy: Preserves original data values without unnecessary transformations
  • Metadata Fidelity: Ensures all scientific metadata is properly transferred and enhanced

2. Performance Optimization

  • Aligned Chunking: Optimizes chunk sizes to prevent partial chunks and improve I/O performance
  • Lazy Loading: Uses xarray and Dask for memory-efficient processing
  • Parallel Processing: Supports distributed computing for large datasets

3. Cloud-Native Design

  • Storage Agnostic: Works with local filesystems, S3, and other cloud storage
  • Scalable: Designed for processing large Earth observation datasets
  • Robust: Includes retry logic and error handling for network operations

System Architecture

graph TB
    A[EOPF DataTree Input] --> B[Conversion Engine]
    B --> C[GeoZarr Output]

    B --> D[Metadata Processing]
    B --> E[Spatial Processing]
    B --> F[Storage Management]

    D --> D1[CF Conventions]
    D --> D2[Grid Mapping]
    D --> D3[Multiscales]

    E --> E1[Chunking Strategy]
    E --> E2[Overview Generation]
    E --> E3[CRS Handling]

    F --> F1[Local Storage]
    F --> F2[S3 Storage]
    F --> F3[Validation]

Core Components

1. Conversion Engine (conversion/geozarr.py)

The main conversion engine orchestrates the transformation process:

# test: skip
def create_geozarr_dataset(
    dt_input: xr.DataTree,
    groups: List[str],
    output_path: str,
    **kwargs
) -> xr.DataTree

Key Functions:

  • setup_datatree_metadata_geozarr_spec_compliant(): Sets up GeoZarr-compliant metadata
  • write_geozarr_group(): Writes individual groups with proper structure
  • create_geozarr_compliant_multiscales(): Creates multiscales metadata

2. File System Utilities (conversion/fs_utils.py)

Handles storage operations across different backends:

Local Storage:

  • Path normalization and validation
  • Zarr group operations
  • Metadata consolidation

S3 Storage:

  • S3 path parsing and validation
  • Credential management
  • S3-specific Zarr operations

Key Functions:

  • get_storage_options(): Unified storage configuration
  • validate_s3_access(): S3 access validation
  • consolidate_metadata(): Metadata consolidation

3. Processing Utilities (conversion/utils.py)

Core processing algorithms:

Chunking:

# test: skip
def calculate_aligned_chunk_size(
    dimension_size: int,
    target_chunk_size: int
) -> int

Downsampling:

The library uses xarray's built-in .coarsen() method for efficient downsampling operations, providing better integration with lazy evaluation and memory management.

Sentinel-2 Optimization:

The S2 optimization module uses a functional programming approach with stateless functions for improved testability and maintainability:

# test: skip
def convert_s2_optimized(
    dt_input: xr.DataTree,
    output_path: str,
    **kwargs
) -> xr.DataTree

4. Command Line Interface (cli.py)

Provides user-friendly command-line access:

  • convert: Main conversion command
  • validate: GeoZarr compliance validation
  • info: Dataset information display

Data Flow

1. Input Processing

graph LR
    A[EOPF DataTree] --> B[Group Selection]
    B --> C[Metadata Extraction]
    C --> D[CRS Analysis]
    D --> E[Dimension Analysis]
  1. DataTree Loading: Load EOPF dataset using xarray
  2. Group Selection: Select specific measurement groups to process
  3. Metadata Extraction: Extract coordinate and variable metadata
  4. CRS Analysis: Determine native coordinate reference system
  5. Dimension Analysis: Calculate optimal chunking and overview levels

2. Conversion Process

graph TB
    A[Input Dataset] --> B[Prepare Datasets]
    B --> C[Create Native Resolution]
    C --> D[Generate Overviews]
    D --> E[Apply Metadata]
    E --> F[Write to Storage]

    B --> B1[Chunking Strategy]
    B --> B2[CRS Preparation]

    D --> D1[Level 1: /2 Factor]
    D --> D2[Level 2: /4 Factor]
    D --> D3[Level N: /2^N Factor]

    E --> E1[CF Conventions]
    E --> E2[Grid Mapping]
    E --> E3[Multiscales]

3. Output Structure

The library creates a hierarchical structure compliant with GeoZarr specification:

output.zarr/
├── .zattrs                    # Root attributes with multiscales
├── measurements/
│   ├── r10m/                  # Resolution group
│   │   ├── .zattrs           # Group attributes
│   │   ├── 0/                # Native resolution
│   │   │   ├── b02/          # Band data
│   │   │   ├── b03/
│   │   │   ├── b04/
│   │   │   ├── b08/
│   │   │   ├── x/            # X coordinates
│   │   │   ├── y/            # Y coordinates
│   │   │   └── spatial_ref/  # CRS information
│   │   ├── 1/                # Overview level 1 (/2)
│   │   └── 2/                # Overview level 2 (/4)
│   ├── r20m/                 # 20m resolution group
│   └── r60m/                 # 60m resolution group
└── .zmetadata                # Consolidated metadata

STAC Integration and Zarr URL Resolution

Every Zarr Group Path Can Be Opened by a Client

Zarr is a key/value store protocol, not a file format. For clients, this means that any Zarr group path is itself a valid store entry point. A URL like:

s3://bucket/S2A_MSIL2A_20251008T100041.zarr/measurements/reflectance

is not a path that needs to be split or reverse-parsed to find some "real" store root. It is the store. The Zarr spec defines the existence of a node by whether {path}/zarr.json resolves to valid metadata. Clients like xarray, zarr-python, GDAL, and OpenLayers should therefore open the asset href directly as a Zarr store, without needing to know anything about the hierarchy above it.

import xarray as xr

# Open the asset href directly — no splitting or parsing needed
asset_href = "s3://bucket/S2A_MSIL2A_20251008T100041.zarr/measurements/reflectance"
dt = xr.open_datatree(asset_href, engine="zarr")
r10m = dt["r10m"].to_dataset()

This is the fundamental principle: the STAC asset href is the URL to open, and it works as a complete, self-contained Zarr store.

Asset Hrefs Point to the Multiscales Root

For multiscale data, the STAC asset href points to the group that carries the multiscales metadata (the multiscales root), not to one of its resolution levels. For Sentinel-2 L2A, this is measurements/reflectance, and the levels (r10m, r20m, r60m, r120m, r360m, r720m) are its children.

This lets clients that support multiscales discover all available levels from the asset href, and select the level that fits the current view. Clients do not need a separate group option to find the multiscales below the asset href (see openlayers/openlayers#17378).

Consolidated Metadata Enables Standalone Group Access

A Zarr group becomes fully self-contained for clients when it carries consolidated metadata. Consolidated metadata embeds the metadata of all descendant nodes inside the group's own zarr.json (Zarr v3) or .zmetadata (Zarr v2). A client can thus discover the entire sub-hierarchy in a single request, with no traversal and no requests to parent groups.

All EOPF-produced Zarr groups pointed to by STAC assets MUST have consolidated metadata. The multiscales root carries it, so the metadata of all its resolution levels is available in one request. The resolution-level groups (e.g. r10m) do not carry their own consolidated metadata, which is one more reason not to use them as asset hrefs.

This is indicated in the STAC asset using the Zarr STAC Extension field zarr:consolidated: true.

"assets": {
  "reflectance": {
    "href": "s3://bucket/S2A_MSIL2A_20251008T100041.zarr/measurements/reflectance",
    "type": "application/vnd.zarr; version=3",
    "zarr:consolidated": true,
    "zarr:node_type": "group",
    "zarr:zarr_format": 3
  }
}

The STAC Zarr Best Practices define a "store" relationship for exactly this purpose. All EOPF-produced STAC Items and Collections MUST include this link:

"links": [
  {
    "rel": "store",
    "href": "s3://bucket/S2A_MSIL2A_20251008T100041.zarr",
    "type": "application/vnd.zarr; version=3",
    "title": "Zarr Store Root"
  }
]

Its purpose is navigation and discovery, not URL parsing:

  • It lets clients traverse or inspect the full Zarr hierarchy above the asset group (siblings, parent groups, global attributes).
  • It provides a single stable reference to the underlying storage location, useful for tools that need to know where the data lives (e.g., to construct pre-signed URLs, or list all groups in a store).
  • It allows a client to verify that all assets in the STAC object belong to the same store.

Note

Opening the rel: store href directly is equivalent to opening the top-level Zarr root. This is useful to explore the complete dataset structure, but it is not required to use any individual asset.

URL Naming Constraint

Group names, array names, and any intermediate path segments MUST NOT end with .zarr. The .zarr suffix SHOULD appear at most once in a full URL — only at the store root level — as a human-readable convention. This avoids confusion when reading URLs, even though no client should rely on this suffix for parsing.

✅  s3://bucket/S2A_MSIL2A_20251008T100041.zarr/measurements/reflectance
❌  s3://bucket/S2A_MSIL2A_20251008T100041.zarr/measurements.zarr/reflectance

EOPF Product URL Anatomy

For a Sentinel-2 L2A EOPF product, the store and asset relationship looks like this:

rel: store  →  s3://bucket/S2A_MSIL2A_20251008T100041.zarr   (top-level root, consolidated)
                │
                └── measurements/
                    └── reflectance/    ← asset href: multiscales root, consolidated
                        ├── r10m/       ← resolution level (listed in the multiscales layout)
                        │   ├── b02     ← array: asset_href + "/r10m/" + band_name
                        │   ├── b03
                        │   └── b04
                        ├── r20m/
                        ├── r60m/
                        ├── r120m/      ← overview levels
                        ├── r360m/
                        └── r720m/

A client reads the multiscales layout from the asset href, selects a level, and joins the level and band name to the asset href to get the full Zarr array URL:

import zarr

asset_href = "s3://bucket/S2A_MSIL2A_20251008T100041.zarr/measurements/reflectance"
reflectance = zarr.open_group(asset_href, mode="r")
red_band = reflectance["r10m/b04"]  # same as asset_href + "/r10m/b04"
  • Zarr v3 specification — defines the abstract store interface, hierarchy paths, and zarr.json metadata documents
  • STAC Zarr Best Practices — defines the rel: store link, asset media types, band representation patterns, and consolidated metadata guidance
  • Zarr STAC Extension — adds zarr:node_type, zarr:zarr_format, and zarr:consolidated fields to STAC assets

Metadata Architecture

1. CF Conventions Compliance

The library ensures full CF (Climate and Forecast) conventions compliance:

# Coordinate variables
x_attrs = {
    'standard_name': 'projection_x_coordinate',
    'long_name': 'x coordinate of projection',
    'units': 'm',
    '_ARRAY_DIMENSIONS': ['x']
}

y_attrs = {
    'standard_name': 'projection_y_coordinate', 
    'long_name': 'y coordinate of projection',
    'units': 'm',
    '_ARRAY_DIMENSIONS': ['y']
}

2. Grid Mapping Variables

Each dataset includes proper grid mapping information:

# test: skip
grid_mapping_attrs = {
    'grid_mapping_name': 'transverse_mercator',  # or appropriate mapping
    'projected_crs_name': crs.to_string(),
    'crs_wkt': crs.to_wkt(),
    'spatial_ref': crs.to_wkt(),
    'GeoTransform': transform_string
}

3. Multiscales Metadata

The converter writes the multiscales convention attributes on the parent group. Each level entry points at a sibling subgroup (r{2**level}) carrying the downsampled dataset:

# test: skip
multiscales = {
    "resampling_method": "mean",
    "layout": [
        {
            "asset": "r10m",                     # native, group root
            "transform": {"scale": [1, 1], "translation": [0, 0]},
        },
        {
            "asset": "r20m",                     # overview
            "derived_from": "r10m",
            "transform": {"scale": [2, 2], "translation": [0, 0]},
        },
        {
            "asset": "r40m",
            "derived_from": "r10m",
            "transform": {"scale": [4, 4], "translation": [0, 0]},
        },
    ],
}

Performance Considerations

1. Chunking Strategy

The library implements intelligent chunking to optimize performance:

def calculate_aligned_chunk_size(dimension_size: int, target_chunk_size: int) -> int:
    """Calculate chunk size that divides evenly into dimension size."""
    if target_chunk_size >= dimension_size:
        return dimension_size

    # Find largest divisor <= target_chunk_size
    for chunk_size in range(target_chunk_size, 0, -1):
        if dimension_size % chunk_size == 0:
            return chunk_size
    return 1

Benefits:

  • Prevents partial chunks that waste storage
  • Improves read/write performance
  • Reduces memory fragmentation
  • Better Dask integration

2. Memory Management

Lazy Loading:

  • Uses xarray's lazy loading capabilities
  • Processes data in chunks to manage memory usage
  • Supports out-of-core processing for large datasets

Band-by-Band Processing:

# test: skip
def write_dataset_band_by_band_with_validation(
    ds: xr.Dataset,
    output_path: str,
    max_retries: int = 3
) -> None

3. Parallel Processing

Dask Integration:

  • Supports Dask distributed computing
  • Automatic parallelization of chunk operations
  • Configurable cluster setup

Retry Logic:

  • Robust error handling for network operations
  • Configurable retry attempts
  • Graceful degradation on failures

Storage Architecture

1. Storage Abstraction

The library provides a unified interface for different storage backends:

# test: skip
def get_storage_options(path: str, **kwargs) -> Optional[Dict[str, Any]]:
    """Get storage options based on path type."""
    if is_s3_path(path):
        return get_s3_storage_options(path, **kwargs)
    return None

2. S3 Integration

Features:

  • Automatic credential detection
  • Custom endpoint support
  • Bucket validation
  • Optimized multipart uploads

Configuration:

# test: skip
s3_options = {
    'key': os.environ.get('AWS_ACCESS_KEY_ID'),
    'secret': os.environ.get('AWS_SECRET_ACCESS_KEY'),
    'endpoint_url': os.environ.get('AWS_ENDPOINT_URL'),
    'region_name': os.environ.get('AWS_DEFAULT_REGION', 'us-east-1')
}

3. Metadata Consolidation

Zarr metadata consolidation for improved performance:

def consolidate_metadata(output_path: str, **storage_kwargs) -> None:
    """Consolidate Zarr metadata for faster access."""
    store = get_zarr_store(output_path, **storage_kwargs)
    zarr.consolidate_metadata(store)

Error Handling and Validation

1. Input Validation

  • DataTree structure validation
  • Group existence checks
  • CRS compatibility verification
  • Dimension consistency checks

2. Processing Validation

  • Chunk alignment verification
  • Memory usage monitoring
  • Progress tracking
  • Intermediate result validation

3. Output Validation

  • GeoZarr specification compliance
  • Metadata completeness checks
  • Data integrity verification
  • Performance metrics collection

Extensibility

1. Plugin Architecture

The library is designed to support extensions:

  • Custom storage backends
  • Additional metadata formats
  • Custom processing algorithms
  • Validation plugins

2. Configuration System

Flexible configuration through:

  • Environment variables
  • Configuration files
  • Runtime parameters
  • Default value inheritance

Testing Architecture

1. Unit Tests

  • Individual function testing
  • Mock external dependencies
  • Edge case coverage
  • Performance benchmarks

2. Integration Tests

  • End-to-end conversion workflows
  • Storage backend testing
  • Real dataset processing
  • Cloud environment testing

3. Local Test Data

The library uses an efficient testing approach with lightweight JSON-based Zarr groups that contain only the structure and metadata (no chunked array data). This provides:

  • Faster Test Execution: Tests run locally without downloading large datasets
  • No Remote Dependencies: Eliminates need for network access during testing
  • Lightweight Fixtures: JSON files define Zarr group structure using pydantic-zarr

Test fixtures are created from JSON schemas stored in tests/test_data_api/{s1_examples,s2_examples}/ directories, making the test suite both comprehensive and efficient.

4. Validation Tests

  • GeoZarr specification compliance
  • Metadata accuracy verification
  • Data integrity checks
  • Performance regression testing

This architecture ensures the EOPF GeoZarr library is robust, performant, and maintainable while meeting the specific needs of Earth observation data processing.