🌍 🌦️ 🧠 TerraCliMind: Climate-Aware Generative Multimodality for Earth Observation

🧑‍🎓Stef Lhermitte | Dept Earth & Environmental Sciences, KULeuven, Belgium | https://orcid.org/0000-0002-1622-0177
🧑‍🎓Kristof Van Tricht | VITO, Belgium | https://orcid.org/0000-0001-5727-9098
✉️ stef.lhermitte @ kuleuven.be
🌐 www.earthmapps.io

As we collectively push the frontier of Earth Observation (EO) with foundation models, one critical dimension remains notably underrepresented: climate awareness. In this post, we introduce TerraCliMind—a novel extension to the TerraMind framework that incorporates real-world climate data directly into the generative EO modeling pipeline.

This project builds upon recent advances in multimodal generative EO models, especially the groundbreaking TerraMind model. TerraMind showed how a powerful transformer-based architecture can encode and generate across nine key EO modalities (e.g. optical, SAR, DEM, NDVI), achieving "any-to-any" generation via its Thinking-in-Modalities (TiM) approach.

However, TerraMind—and other EO foundation models to date—are climate-agnostic. That is, they do not explicitly encode or condition on the meteorological and environmental processes that shape EO observations. This omission is more than a technical detail—it represents a fundamental gap in modeling Earth's dynamic systems.

🌦 Why Climate Awareness Matters for EO Models

Earth observation data is not static. The same location can look dramatically different depending on the season, weather, and climate conditions at the time of image acquisition. For instance:

• Agricultural fields in Western Europe appear lush green in spring, golden brown in late summer, or snow-covered in winter.
  
• **Vegetation health and ecosystem properties ** shift visibly in response to interannual variability: compare a wet summer (e.g. 2017) with a dry, drought-stricken one (e.g. 2018).
  
• Cloud patterns and atmospheric conditions vary predictably across seasons and times of day—affecting both the appearance and usability of optical satellite imagery.

These effects are not just “noise” but carry essential signals that inform how EO data should be interpreted. A model trained to classify crops or detect anomalies without understanding whether it’s spring or summer, wet or dry, risks conflating natural variability with true change.

Yet, most foundation models treat EO scenes as disembodied pixels and tokens, divorced from the processes that generated them. As a result, they may misclassify seasonal senescence as drought, or hallucinate unrealistic images in generative tasks. We argue that bridging this gap requires explicitly integrating climate context into the generative modeling pipeline.

A picture is worth a thousand words
but understanding the process is probably worth a thousand pictures

🧠 Introducing TerraCliMind

TerraCliMind is our proposed solution: a climate-aware generative EO model that builds on TerraMind’s powerful backbone while introducing a dedicated climate-conditioning branch.

We designed TerraCliMind around several core principles:

1. Climate matters: Incorporate weather and climate variables such as temperature, precipitation, and solar radiation into EO generation.
2. Leverage what works: Reuse the pretrained TerraMind transformer for EO spatial encoding and decoding.
3. Train efficiently: Use a modular finetuning approach without retraining the entire model.
4. Stay grounded: Align climate and EO data across space and time to ensure meaningful conditioning.

🏗 Architecture

Our TerraCliMind architecture, proposed for the TerraMind Blue-Sky Challenge, includes two main branches that come together in the newly proposed TerraCLiMind model:

• 🛰 TerraMind Backbone:

  This branch encodes static spatial inputs—Digital Elevation Model (DEM) and geographic coordinates, potentially enriched with additional conditions on Land Use Land Cover (LULC)—into rich spatial embeddings using the original TerraMind encoder-decoder pipeline. For this branch we will test frozen Terramind’s backbone as well as a lightweight trainable adapter layer (e.g. LoRA).

• 🌡 Climate Conditioning Branch:

  This new branch ingests time series of climate variables (e.g. ERA5 data: temperature, precipitation, shortwave radiation, relative humidity) leading up to the EO acquisition date. These are processed using a temporal encoder (e.g. Transformer or GRU) to extract a climate-aware embedding. When encoding the climate data we will focus on embedding both the high-temporal resolution of weather data shaping the short-term landscape-weather response and the lower temporal resolution climate data shaping the long-term landscape-climate response.

These two streams are fused before EO generation, such that the model can condition output imagery not only on the spatial layout of the scene, but also on the environmental conditions that shaped it.

Only the parameters of the climate branch and fusion layers are finetuned; the TerraMind core remains frozen or minimally adapted using low-rank adaption to make the model training process efficient and fast. This makes training feasible without access to large-scale compute resources or full model weights—allowing us to “bolt on” climate awareness in a modular way.

🌍 Data Strategy

We are constructing a global dataset composed of:

• Randomly sampled Sentinel-2 scenes across diverse latitudes, land types, and time periods
• Static context: DEM, and geographic coordinates with optional extra conditions for LULC
• ERA5 climate time series for days/months/years preceding the scene acquisition

By covering regions and times with strong seasonal or interannual variability, we ensure the model learns meaningful correlations between EO appearance and its underlying climate drivers.

🧪 Evaluation Plan

We plan to evaluate TerraCliMind in the context of the TerraMind Blue-Sky Challenge, focusing on one baseline and three high-impact use cases:

1. Baseline:

   Benchmarking the original TerraMind with TerraCliMind to assess the additional value of adding climate awareness to the generative process.

2. Land Cover Classification:

   Climate-aware generation may improve the model’s ability to distinguish between vegetative states that are similar spatially but differ temporally (e.g. spring wheat vs. late-summer barley).

3. Anomaly & Change Detection:

   By conditioning on expected climate, the model can distinguish between natural seasonal variation and true anomalies. For example, it can avoid mislabeling fall senescence as drought.

4. Climate Scenario Simulation:

   Perhaps the most exciting application: given a baseline EO scene and a hypothetical future climate input (e.g. hotter, drier), TerraCliMind can generate counterfactual imagery—providing synthetic previews of how a landscape might look under different climate scenarios.

This opens doors for impact modeling, climate adaptation planning, and digital twin simulations of Earth under change.

✅ Hypothesized Benefits

We anticipate that TerraCliMind will bring several unique advantages:

• Higher accuracy in climate-sensitive tasks: By ingesting climate context, the model should better interpret seasonal or long-term patterns. For instance, it can learn that certain land features appear differently in wet vs. dry seasons, reducing misclassification.
• Physically consistent generation: Conditioning on climate data (e.g. precipitation maps, physicsbased masks) should prevent the model from hallucinating unrealistic events. Prior work showed that adding physics constraints (like flood masks) makes generated images more accurate (Lutjens et. al. 2021) TerraCliMind’s climate tokens serve a similar role.
• Scenario simulation: Unlike TerraMind, TerraCliMind can generate counterfactual imagery under different climate scenarios. For example, it could produce a synthetic image of a region under a future drought or warming, aiding impact assessment.
• Data augmentation via TiM-style tuning: The model can self-generate climate-augmented training samples. E.g. given a land-cover map, it can generate varied realizations under different climate perturbations, effectively increasing the diversity of training data for downstream tasks.
• Foundation integration of EO and climate sciences: TerraCliMind bridges remote sensing and climate modeling into one model. This could facilitate end-to-end learning of Earth system processes (similar in spirit to the way EarthNet merges satellite inputs to recreate the atmosphere.

Moreover, this model paves the way for a new generation of EO foundation models that do more than just generate pixels—they learn and reflect the physical Earth system that underlies them.

🔜 What’s Next?

We are currently developing and fine-tuning TerraCliMind and preparing our submission to the challenge. Code, pretrained weights, and evaluation benchmarks will be released once finished to facilitate further research and reproducibility. A research paper will be written based on the results.

We’re eager to collaborate, learn from others, and see what the community builds with climate-aware generative EO models. Feel free to reach out if you're working on similar challenges—we believe this is just the beginning of a broader effort to embed Earth system intelligence into AI.