PiCo: Physics-Informed Concept Probing for Geo-Foundation Models
Yihang She, Andrew Blake, Jianbo Qi, Srinivasan Keshav
Geo-foundation models (GFMs) are evaluated by downstream predictive performance. This reveals little about which physical properties their embeddings are sensitive to or which are used in predictions. This limits developers' ability to test hypotheses about model behaviour and users' ability to select models for new tasks. We propose Physics-Informed Concept Probing (PiCo), a post-hoc framework for explaining GFM predictions. PiCo identifies concept vectors in GFM embedding space using synthetic multispectral satellite observations from a 3D radiative transfer model coupled with a game engine. These vectors characterise GFM sensitivities to physical properties, informing domain-based hypotheses about downstream performance. Projecting embeddings onto concept vectors produces low-dimensional representations for quantifying concept importance in predictions. We evaluate PiCo on three GFMs with different input modalities using seven physical concepts covering forest leaf optics and canopy structure. In ground-cover estimation, RGB-only inputs reduce dry-matter sensitivity and disproportionately degrade predictions for dry vegetation and bare ground, as anticipated by PiCo's sensitivity analysis. For hierarchical forest type and tree species classification, concept representations retain 70.0--93.8% of full-embedding accuracy. Different GFMs capture physical differences between classes, such as canopy structure between broadleaf forest and cleared ground and leaf-water-content between oak and beech. Yet they differ in the strength of these contrasts and the concepts they use for classification. PiCo provides an interpretable basis for hypothesis testing and model assessment, with potential applicability beyond Earth observation wherever physical models enable controlled simulation and pretrained embeddings are accessible for probing. Code is available at https://github.com/yihshe/PiCo.git.
尚无 AI 解读，展示原文摘要