Abstract
The filtering distribution in hidden Markov models evolves according to the law of a mean-field model in state–observation space. The ensemble Kalman filter (EnKF) approximates this mean-field model with an ensemble of interacting particles, employing a Gaussian ansatz for the joint distribution of the state and observation at each observation time. These methods are robust, but the Gaussian ansatz limits accuracy. We address this shortcoming by using machine learning to map the joint predicted state and observation to the updated state estimate. A novel form of neural operator taking probability distributions as input—a measure neural mapping (MNM)—is used to design the MNM-enhanced ensemble filter (MNMEF). The ensemble method is implemented using a set transformer, which is invariant to ensemble permutation and allows the same basic parameterization to be deployed at different ensemble sizes. Fine-tuning a small number of ensemble-size-specific parameters further improves accuracy. Experiments demonstrate superior root-mean-square-error performance relative to leading filtering methods on Lorenz-96 and Kuramoto–Sivashinsky models.
Type
Publication
Journal of Computational Physics, 547, 114550
This work introduces the measure neural mapping enhanced ensemble filter (MNMEF),
a learning-based data-assimilation method derived from a mean-field formulation of the
filtering problem. Measure neural mappings extend neural operators to maps acting on
probability measures; their finite-ensemble implementation uses a permutation-invariant set
transformer.
The mean-field construction allows most learned parameters to be shared across ensemble
sizes. A model can therefore be trained efficiently using a small ensemble and deployed at
other ensemble sizes, while lightweight fine-tuning adapts a limited number of parameters
such as inflation and localization.
Experiments on Lorenz-63, Lorenz-96, and Kuramoto–Sivashinsky systems show improved
filtering accuracy relative to optimized classical ensemble methods across both small and
larger ensembles.
