# Population-Scale User Simulation for Disaster Risk Management: A Methodological Analysis of the MatrAIx Infrastructure

*AI · Rıdvan Bilgin · 2026-08-11*

## TL;DR

MatrAIx is a population-scale simulated-user evaluation infrastructure containing 8.3 billion persona records across 1,290 dimensions. For disaster risk management, it enables pre-deployment testing of crisis communication, inclusive tabletop exercises with vulnerable populations, and evacuation compliance modelling. However, simulation results depend heavily on the chosen AI model and cannot replace human validation.

## Key facts

- MatrAIx contains 8.3 billion synthetic and human-grounded persona records.
- The system uses a 1,290-dimensional schema covering background, psychology, capability, behaviour, and lifestyle.
- A directed acyclic graph (DAG) preserves real-world correlations such as age-education and region-language dependencies.
- Controlled studies show 91.5% behavioural adherence across four simulation environments.
- Model choice dramatically affects outcomes: paid-plan selection ranged from 23.2% to 93.9% across three models.
- The public Persona 1M coreset includes approximately 1 million records for research.

## Introduction: Beyond Static Testing for Crisis Scenarios

Traditional AI evaluations rely on static benchmarks. They check if a system completes tasks—passes unit tests, clicks the right buttons, or generates correct outputs. But these evaluations fail to capture how diverse users formulate requests, interact with systems, and judge results. A novice user may need detailed explanations and frequent confirmations. An expert may prefer terse responses and greater autonomy. These differences shape interaction trajectories, trust in results, and willingness to continue after a failure.

For disaster risk management, this limitation is critical. Crisis communication systems must reach everyone—elderly individuals, non-native speakers, people with disabilities, and those with limited digital literacy. Offline benchmarks cannot model how these groups respond to evacuation orders, early warnings, or relief information.

**Population-scale user simulation** addresses this gap. It creates heterogeneous simulated populations to test systems before deployment, exposing subgroup-specific failure modes that aggregate scores miss.

MatrAIx, developed by researchers from MIT, Harvard, Stanford, and other institutions, represents a significant advance in this direction. The system contains 8.3 billion persona records under a shared 1,290-dimensional schema, with a curated coreset of approximately 1 million records released for research.

## The MatrAIx Architecture: Technical Foundations

### The Persona Schema: Representing Human Diversity

MatrAIx represents human variation through a shared categorical schema with 1,290 dimensions. These are organized into five top-level groups:

### Persona Schema Overview

| Group | Count | Representative Attributes | Grounding Sources |
|---|---|---|---|
| Background | 238 | Age, region, language, education, career, industry | UN Population Prospects, World Bank, ILO, Stack Overflow |
| Psychology | 210 | Personality, values, worldview, motivation, risk tolerance | IPIP, MIDUS, WVS, GSS, Eurobarometer |
| Capability | 331 | Domain expertise, skills, tools, programming experience | O*NET, ITU, GitHub Octoverse |
| Behaviour & Interaction | 124 | Preferences, habits, technology adoption | ATUS, OECD Time Use, Consumer Expenditure |
| Lifestyle | 387 | Interests, media, culture, health, fitness | WHO GHO, IHME GBD, FAOSTAT, UNESCO |

This structured representation makes missing values, contradictions, and population queries inspectable, unlike free-text profiles which are flexible but opaque. The complete three-layer taxonomy ships with the release, and every attribute has an assignment at each layer, allowing independent verification.

### Dependency-Aware Sampling with DAG

Matching marginal distributions alone does not produce a coherent population. Independent sampling breaks age-education, region-language, employment-seniority, and other dependencies, producing implausible profiles despite accurate aggregate counts.

MatrAIx uses a directed acyclic graph (DAG) where each of the 1,290 schema dimensions is a node. An edge indicates that one dimension is sampled conditionally on another. For example, education level is drawn given age bracket, while English proficiency is drawn given primary language and region.

The proposal distribution factorizes as:

For a non-root dimension, a candidate value is scored by combining three components:

### Components of the Conditional Probability Distribution

| Component | Function | Example |
|---|---|---|
| πᵢ(v) (Base prior) | Population-wide probability of each value before parent attributes are known | Prior probability of a given English proficiency level |
| rᵢ(v; x_Pa(i)) (Dependency adjustment) | Parent-specific likelihood ratios that upweight values common in context | Region and primary language shift proficiency probabilities |
| mᵢ(v; x_Pa(i)) (Compatibility mask) | Binary rule that excludes invalid combinations | Primary language English with proficiency 'None' is disallowed |

This separation ensures statistical rarity is never confused with logical invalidity. A rare but admissible profile is downweighted by the factors but can still be sampled, while an inadmissible combination is removed by the mask and can never appear.

### Synthetic vs. Human-Grounded Records

### Persona Source Types

| Source Type | Methodology | Quantity |
|---|---|---|
| Synthetic | Forward sampled from the DAG, preserving cross-attribute correlations and compatibility rules | 8.3 billion (internal); 400,000 in public coreset |
| Human-grounded | Mapped into the same schema from six sources: Wikipedia, Amazon reviews, Stack Overflow Developer Survey, General Social Survey (GSS), PRISM Alignment profiles, and 355 consented volunteer self-reports | 599,847 in public coreset |

Human-grounded records are de-identified by removing direct identifiers such as names and contact details while retaining only extracted attributes and descriptions. The public Persona 1M coreset includes a manifest, audit of achieved demographic shares, and per-file hashes for reproducibility.

### Four Simulation Environments

### MatrAIx Playground Environments

| Environment | Mechanism | DRM Application |
|---|---|---|
| Survey | Agents complete structured questionnaires | Testing public perception of early warning systems or price sensitivity for emergency supplies |
| AI Chatbot | Agents converse with AI assistants | Evaluating crisis chatbots for retention after errors or response latency tolerance |
| Web | Browser automation or Computer-Using Agents (CUAs) browse live websites | Testing how diverse shoppers find emergency supplies or evacuation information |
| App | CUAs operate native Linux, macOS, or iOS applications | Testing accessibility of emergency alert features in mobile operating systems |

Each environment records what each persona agent thinks, says, and does, including task duration, completion status, and verifier results.

## Applications for Disaster Risk Management

### Crisis Communication: Testing Warning Messages

The Mileti-Sorensen (1990) Social Warning Response Model describes how individuals respond to emergency warnings through six stages: Hearing, Understanding, Believing, Personalising, Deciding, Acting. MatrAIx can test each stage at scale.

The hesitation response to a price increase, while appearing as a financial decision, serves as a proxy for **cognitive resistance and hesitation index** under changing conditions. In the Candy Land price sensitivity case study, 1,000 personas were tested on their reaction to a price increase. The hesitation rate varied dramatically depending on the persona-agent model:

### Hesitation Rates by Model: Candy Land Price Sensitivity

| Model | Hesitation Rate | Interpretation |
|---|---|---|
| GPT-5.5 | 98.3% | Strong sensitivity to price increase |
| Claude Opus 4.8 | 27.0% | Low sensitivity; 'fair buy' modal response at 73.0% |
| Claude Haiku 4.5 | 83.3% | Moderate sensitivity |

While the absolute numbers differ, the cohort ranking remained consistent, demonstrating the system's ability to identify which demographic subgroups are most sensitive to changes in messaging. For crisis communication, this suggests that warning message framing may produce dramatically different responses across demographic groups. Testing multiple model configurations can reveal which groups are most likely to hesitate or dismiss critical instructions.

### Inclusive Tabletop Exercises

Traditional tabletop exercises are limited by participant availability and diversity. MatrAIx enables running thousands of scenarios with specifically sampled populations, including vulnerable groups.

The Sendai Framework for Disaster Risk Reduction 2015-2030, specifically Target G on multi-hazard early warning systems, emphasises inclusivity. MatrAIx supports this through:

Targeted Sampling

### Select Personas by Demographics

Choose personas by age, disability status, primary language, English proficiency, or accessibility needs to test inclusive crisis communication.

Task Coverage

### 1,010 Application Tasks

Covering 25+ domains including travel, insurance, education, and healthcare, with many directly relevant to disaster response and recovery.

Public Data

### Persona 1M Coreset

599,847 human-grounded and 400,000 synthetic records enable reproducible and transparent studies across institutions.

Geographic Diversity

### Volunteer Survey Distribution

20.4% from South Asia, 19.1% from Sub-Saharan Africa, and 13.4% from East Asia, supporting culturally-specific crisis communication testing.

### Evacuation Compliance Modelling

Persona attributes like trust level and risk tolerance significantly affect behaviour. In the OpenBB validation task, the trust level dimension consistently predicted behaviour across all three persona-agent models (Cramér's V = 0.228-0.363, all q < 10⁻⁸), with all three models ordering the four trust groups identically.

### Key Findings: Evacuation Compliance Modelling

| Finding | Implication |
|---|---|
| Trust level predicts compliance | Higher trust leads to higher engagement and follow-through |
| Economic motivation affects decisions | Cost-sensitive personas push back more on unrealistic suggestions |
| Model consistency on subgroup ordering | Three independent models agree on which groups are more trustworthy |

This supports using MatrAIx for pre-deployment testing of evacuation messaging, identifying which population segments may require additional communication or alternative channels.

Key DRM Applications

**Crisis communication** can be tested before real emergencies, identifying which demographic groups hesitate or dismiss warnings. **Inclusive tabletop exercises** can run at scale with diverse populations, testing accessibility for vulnerable groups. **Evacuation compliance** can be modelled by varying trust and risk tolerance attributes.

## Methodological Validation and Limitations

### Controlled Behavioural Adherence

A 400-trial controlled study across ten behavioural attributes and all four environments found that declared behaviour was expressed or correctly suppressed in 366 out of 400 trials (91.5%).

### Behavioural Adherence by Environment

| Environment | Success Rate | Strong Cells (≥4/5 in both arms) |
|---|---|---|
| Survey | 96/100 | 9/10 |
| AI Chatbot | 92/100 | 9/10 |
| Web | 95/100 | 9/10 |
| App | 83/100 | 6/10 |

Over-Compliance Risk

The lower App score highlights a boundary condition: asking a model to behave impolitely often fails due to built-in safety alignment. This points to an **over-compliance risk**—real-world panic, defiance, or hesitation may be under-represented in simulations because models are fine-tuned to avoid offensive or extreme outputs. Consequently, rare but critical behaviours (e.g., stubborn refusal to evacuate) might be systematically suppressed, potentially masking vulnerabilities in crisis plans.

### Model Dependence and Self-Preference Bias

Running the same cohort and task across three frontier models produces dramatically different outcomes:

### Model Dependence: Outcome Variation

| Task | GPT-5.5 | Claude Opus 4.8 | Claude Haiku 4.5 |
|---|---|---|---|
| Notion Plans (Selected Paid) | 75.8% | 23.2% | 93.9% |
| OpenBB Honesty (Would Not Continue) | 18.5% | 14.5% | 71.9% |
| News+ Subscription (Subscribed) | 4.2% | 20.8% | 0.0% |

This variance reveals a critical issue for disaster management applications: simulation results may reflect model priors more than human behaviour. Three independent random orderings of four groups coincide with probability (1/4!)² = 0.0017, yet the trust-level ordering was consistent across models. This supports subgroup ranking validity while highlighting the absolute outcome dependence on model choice.

🧠 Persona Agent (LLM-based)

→

⚡ Self-Preference Bias

→

🤖 System Under Test (e.g., Crisis Chatbot)

→

⚠️ CRITICAL WARNING: Same backbone inflates satisfaction

When the model playing the persona and the model behind the system under test share a backbone, agreement may mean the system served the user well or that the model recognised and preferred its own output. The direction of the bias is not benign: a persona may accept an answer a person would have pushed back on, inflating satisfaction and suppressing the friction the evaluation exists to surface.

**Recommended practice:** Run the same cohort under at least one persona model that does not share a backbone with the system under test. Treat agreement between matched models as a hypothesis to check, not a result to report.

### The Sim2Real Gap

Zhou et al. (2026) identify the sim-to-real gap in user simulation for agentic tasks. The present experiments do not directly measure whether MatrAIx personas withhold context, push back, or give up the way people do.

Approach 1

### Population-Level Comparison

Compare simulated user turns against matched slices of real conversation logs on turn length, question type, correction rates, and abandonment rates.

Approach 2

### Individual-Level Behaviour Chain Test

Extract a persona from the first half of a held-out conversation, simulate the next user turns, and score against what the person actually wrote.

Both would need to be reported per agent model given the model dependence observed earlier.

Key Validation Statistics

- **91.5%** behavioural adherence across 400 controlled trials
- **33 of 40** attribute-environment cells achieved at least 4 of 5 successes in both arms
- **23.2% to 93.9%** outcome variation on the same task across three models
- **q < 10⁻⁸** for trust-level dimension consistently predicting behaviour
- **0.0017** probability of identical subgroup ordering occurring by chance across three models

## Responsible Use and Ethical Boundaries

### Simulations Are Not People

MatrAIx personas are simulation instruments rather than people. A cohort of them is not a probability sample of a real population. They cannot be used to:

- Impersonate named individuals
- Attribute opinions or behaviour to real people or identifiable communities
- Assemble profiles that target individuals
- Script persuasion, exclusion, or price discrimination aimed at protected groups

### Human Validation Required

Running a simulated cohort does not replace consulting the people a product actually affects, particularly where decisions have consequences in health, finance, employment, or other regulated settings. The volunteer instrument asks for no name, contact detail, or account identifier and carries a decline option on all 1,290 items.

### Reporting Standards

Every result is a persona-agent result, not a direct claim about human behaviour. The agent model is recorded as part of the evaluation configuration. Important findings should be checked with more than one model and traced back to the underlying interactions. Human studies remain necessary before applying conclusions to real populations.

## Future Directions for Disaster Research

Long-Term Simulation

### Dynamic Personas with Memory

Simulating behaviour during prolonged crises like pandemics or multi-week evacuations with agents that retain and update their state over time.

Security Testing

### Automated Red Teaming

Testing security and safety of critical infrastructure before failures occur, identifying vulnerabilities through adversarial simulated populations.

Validation

### Behaviour-Chain Validation

Comparing simulated behaviour against real logs (e.g., WildChat, LMSYS-Chat-1M) to close the Sim2Real gap and improve prediction accuracy.

Task Expansion

### Expanding Disaster-Specific Tasks

Adding more disaster-specific tasks beyond the current 1,010 across 25+ domains, including earthquake response, flood evacuation, and public health emergencies.

## Conclusion

MatrAIx provides an end-to-end infrastructure for population-scale simulated-user evaluation. Its three core components—Persona 8B with 8.3 billion records, the MatrAIx Playground with four evaluation environments, and MatrAIx Applications with 1,010 versioned tasks—support pre-deployment screening, subgroup analysis, stress testing, and product version comparison.

However, the system has critical limitations:

- **Model dependence:** Outcomes vary dramatically across persona-agent models (e.g., 23.2% to 93.9% on the same task).
- **Self-preference bias:** Shared backbones between persona and system can inflate satisfaction scores.
- **Safety alignment suppression:** Models may refuse to simulate extreme behaviours (panic, aggression) due to built-in training, introducing an over-compliance risk that may hide dangerous real-world responses.

Used responsibly, with model configurations reported and human validation maintained, MatrAIx can help disaster managers make better-informed decisions about system design and crisis communication strategies. But it remains a pre-screening tool, not a substitute for engaging the people a disaster actually affects.

## References

1. [MatrAIx Paper](https://arxiv.org/abs/2608.04205) Li, X., Hao, Y., et al. (2026). MatrAIx: Simulating the world with 8.3 billion persona agents. arXiv:2608.04205v1.
2. [Mileti & Sorensen Model](https://www.osti.gov/biblio/5663449) Mileti, D. S., & Sorensen, J. H. (1990). Communication of emergency public information: Mitigation and preparedness implications. Oak Ridge National Laboratory.
3. [Sendai Framework](https://www.undrr.org/publication/sendai-framework-disaster-risk-reduction-2015-2030) UNDRR. (2015). Sendai Framework for Disaster Risk Reduction 2015-2030. United Nations Office for Disaster Risk Reduction.
4. [Sim2Real Gap](https://arxiv.org/abs/2603.11245) Zhou, X., Sun, W., Ma, Q., Xie, Y., Liu, J., Du, W., Welleck, S., Yang, Y., Neubig, G., Wu, S. T., & Sap, M. (2026). Mind the Sim2Real gap in user simulation for agentic tasks. arXiv:2603.11245.
5. [Santurkar Model Bias](https://proceedings.mlr.press/v202/santurkar23a.html) Santurkar, S., Durmus, E., Ladhak, F., Lee, C., Liang, P., & Hashimoto, T. (2023). Whose opinions do language models reflect? ICML 2023.
6. [Argyle Human Simulation](https://www.cambridge.org/core/journals/political-analysis/article/out-of-one-many-using-language-models-to-simulate-human-samples/035D7C196B7B12F8B4D2E128B0E6A8E8) Argyle, L. P., Busby, E. C., Fulda, N., Gubler, J. R., Rytting, C., & Wingate, D. (2023). Out of one, many: Using language models to simulate human samples. Political Analysis, 31(3), 337-351.


## Frequently asked questions

**What is MatrAIx and how does it work?**

MatrAIx is a population-scale simulated-user evaluation infrastructure developed by researchers from MIT, Harvard, Stanford and other institutions. It contains 8.3 billion persona records under a shared 1,290-dimensional schema covering background, psychology, capability, behaviour and lifestyle. The system uses a dependency-aware sampling approach with directed acyclic graphs (DAGs) to preserve real-world correlations between attributes, then runs these personas through four evaluation environments: Survey, AI Chatbot, Web, and App.

**Can MatrAIx replace human studies in disaster management?**

No. MatrAIx is designed as a pre-screening and hypothesis-generating tool, not a substitute for human studies. Simulation results depend substantially on the chosen persona-agent model, and shared backbones between persona and system can introduce self-preference bias. Important findings must be checked with multiple models and validated with real human studies before applying conclusions to real populations.

**How accurate are MatrAIx simulations?**

In controlled adherence studies, 91.5% of trials expressed or correctly suppressed declared behaviours. However, absolute outcome rates vary dramatically across models: paid-plan selection ranged from 23.2% to 93.9% on identical cohorts. While subgroup ranking tends to be consistent (e.g., trust-level ordering was identical across three models with probability 0.0017), the absolute numbers should be interpreted with caution.

**What are the limitations of AI persona simulations for crisis response?**

Key limitations include: model dependence (results vary dramatically across AI models), self-preference bias (shared backbones can inflate satisfaction scores), safety alignment suppression (models may refuse to simulate extreme behaviours like panic or aggression), and the Sim2Real gap where simulated behaviour may not match real human responses. The system also cannot replace human validation for consequential decisions in health, finance, or public safety.

Canonical: https://ridvanbilgin.com/population-scale-user-simulation-for-disaster-risk-management-a-methodological-analysis/