Population-Scale User Simulation for Disaster Risk Management: A Methodological Analysis of the MatrAIx Infrastructure

Traditional AI evaluations rely on static benchmarks. They check if a system completes tasks—passes unit tests, clicks the right buttons, or generates correct outputs. But these evaluations fail to capture how diverse users formulate requests, interact with systems, and judge results. A novice user may need detailed explanations and frequent confirmations. An expert may prefer terse responses and greater autonomy. These differences shape interaction trajectories, trust in results, and willingness to continue after a failure.

For disaster risk management, this limitation is critical. Crisis communication systems must reach everyone—elderly individuals, non-native speakers, people with disabilities, and those with limited digital literacy. Offline benchmarks cannot model how these groups respond to evacuation orders, early warnings, or relief information.

Population-scale user simulation addresses this gap. It creates heterogeneous simulated populations to test systems before deployment, exposing subgroup-specific failure modes that aggregate scores miss.

MatrAIx, developed by researchers from MIT, Harvard, Stanford, and other institutions, represents a significant advance in this direction. The system contains 8.3 billion persona records under a shared 1,290-dimensional schema, with a curated coreset of approximately 1 million records released for research.

MatrAIx represents human variation through a shared categorical schema with 1,290 dimensions. These are organized into five top-level groups:

This structured representation makes missing values, contradictions, and population queries inspectable, unlike free-text profiles which are flexible but opaque. The complete three-layer taxonomy ships with the release, and every attribute has an assignment at each layer, allowing independent verification.

Matching marginal distributions alone does not produce a coherent population. Independent sampling breaks age-education, region-language, employment-seniority, and other dependencies, producing implausible profiles despite accurate aggregate counts.

MatrAIx uses a directed acyclic graph (DAG) where each of the 1,290 schema dimensions is a node. An edge indicates that one dimension is sampled conditionally on another. For example, education level is drawn given age bracket, while English proficiency is drawn given primary language and region.

The proposal distribution factorizes as:

For a non-root dimension, a candidate value is scored by combining three components:

This separation ensures statistical rarity is never confused with logical invalidity. A rare but admissible profile is downweighted by the factors but can still be sampled, while an inadmissible combination is removed by the mask and can never appear.

Human-grounded records are de-identified by removing direct identifiers such as names and contact details while retaining only extracted attributes and descriptions. The public Persona 1M coreset includes a manifest, audit of achieved demographic shares, and per-file hashes for reproducibility.

Each environment records what each persona agent thinks, says, and does, including task duration, completion status, and verifier results.

The Mileti-Sorensen (1990) Social Warning Response Model describes how individuals respond to emergency warnings through six stages: Hearing, Understanding, Believing, Personalising, Deciding, Acting. MatrAIx can test each stage at scale.

The hesitation response to a price increase, while appearing as a financial decision, serves as a proxy for cognitive resistance and hesitation index under changing conditions. In the Candy Land price sensitivity case study, 1,000 personas were tested on their reaction to a price increase. The hesitation rate varied dramatically depending on the persona-agent model:

While the absolute numbers differ, the cohort ranking remained consistent, demonstrating the system's ability to identify which demographic subgroups are most sensitive to changes in messaging. For crisis communication, this suggests that warning message framing may produce dramatically different responses across demographic groups. Testing multiple model configurations can reveal which groups are most likely to hesitate or dismiss critical instructions.

Traditional tabletop exercises are limited by participant availability and diversity. MatrAIx enables running thousands of scenarios with specifically sampled populations, including vulnerable groups.

The Sendai Framework for Disaster Risk Reduction 2015-2030, specifically Target G on multi-hazard early warning systems, emphasises inclusivity. MatrAIx supports this through:

Persona attributes like trust level and risk tolerance significantly affect behaviour. In the OpenBB validation task, the trust level dimension consistently predicted behaviour across all three persona-agent models (Cramér's V = 0.228-0.363, all q < 10⁻⁸), with all three models ordering the four trust groups identically.

This supports using MatrAIx for pre-deployment testing of evacuation messaging, identifying which population segments may require additional communication or alternative channels.

A 400-trial controlled study across ten behavioural attributes and all four environments found that declared behaviour was expressed or correctly suppressed in 366 out of 400 trials (91.5%).

Running the same cohort and task across three frontier models produces dramatically different outcomes:

This variance reveals a critical issue for disaster management applications: simulation results may reflect model priors more than human behaviour. Three independent random orderings of four groups coincide with probability (1/4!)² = 0.0017, yet the trust-level ordering was consistent across models. This supports subgroup ranking validity while highlighting the absolute outcome dependence on model choice.

When the model playing the persona and the model behind the system under test share a backbone, agreement may mean the system served the user well or that the model recognised and preferred its own output. The direction of the bias is not benign: a persona may accept an answer a person would have pushed back on, inflating satisfaction and suppressing the friction the evaluation exists to surface.

Recommended practice: Run the same cohort under at least one persona model that does not share a backbone with the system under test. Treat agreement between matched models as a hypothesis to check, not a result to report.

Zhou et al. (2026) identify the sim-to-real gap in user simulation for agentic tasks. The present experiments do not directly measure whether MatrAIx personas withhold context, push back, or give up the way people do.

Both would need to be reported per agent model given the model dependence observed earlier.

MatrAIx personas are simulation instruments rather than people. A cohort of them is not a probability sample of a real population. They cannot be used to:

• Impersonate named individuals
• Attribute opinions or behaviour to real people or identifiable communities
• Assemble profiles that target individuals
• Script persuasion, exclusion, or price discrimination aimed at protected groups

Running a simulated cohort does not replace consulting the people a product actually affects, particularly where decisions have consequences in health, finance, employment, or other regulated settings. The volunteer instrument asks for no name, contact detail, or account identifier and carries a decline option on all 1,290 items.

Every result is a persona-agent result, not a direct claim about human behaviour. The agent model is recorded as part of the evaluation configuration. Important findings should be checked with more than one model and traced back to the underlying interactions. Human studies remain necessary before applying conclusions to real populations.

MatrAIx provides an end-to-end infrastructure for population-scale simulated-user evaluation. Its three core components—Persona 8B with 8.3 billion records, the MatrAIx Playground with four evaluation environments, and MatrAIx Applications with 1,010 versioned tasks—support pre-deployment screening, subgroup analysis, stress testing, and product version comparison.

However, the system has critical limitations:

• Model dependence: Outcomes vary dramatically across persona-agent models (e.g., 23.2% to 93.9% on the same task).
• Self-preference bias: Shared backbones between persona and system can inflate satisfaction scores.
• Safety alignment suppression: Models may refuse to simulate extreme behaviours (panic, aggression) due to built-in training, introducing an over-compliance risk that may hide dangerous real-world responses.

Used responsibly, with model configurations reported and human validation maintained, MatrAIx can help disaster managers make better-informed decisions about system design and crisis communication strategies. But it remains a pre-screening tool, not a substitute for engaging the people a disaster actually affects.