← Research

SYN-DIGITS: A Synthetic Control Framework for Calibrated Digital Twin Simulation

Grace Jiarui Fan, Chengpiao Huang, Tianyi Peng, Kaizheng Wang, Yuhang Wu

Persona simulation based on large language models---often referred to as digital twin simulation---is increasingly used for market research, recommender systems, and social science research. However, these simulations often exhibit systematic bias and miscalibration relative to real human behavior. We propose SYN-DIGITS, a model-agnostic, post-hoc calibration framework that learns relationships among questions from digital twins and transfers them to predict human responses. We focus on a cold-start setting in which no human response to the target question is observed, so calibration must rely on digital twins and human data on reference questions. Our analysis shows that, though digital twins may not reproduce individual human responses accurately, they can capture nontrivial relationships among questions that can be effectively transferred to the human side. When individual-level data is unavailable and we only have access to marginal information on reference questions, our framework also supports distribution-level calibration for predicting the distribution of human responses on the target question. Across three datasets and thirteen persona constructions, experiments show that SYN-DIGITS achieves 11% to 417% relative improvements in individual-level correlation and 46%--57% relative reductions in distributional discrepancy compared to uncalibrated baselines.

PDF ↗

Models

Loading
EVIDENCE / COMPARISON

Benchmarks

Explore every benchmark

Loading

These charts contain model-linked results. The charts above also include unlinked source profiles. Compare scores within each evaluation. Model quality, usage, transparency and system performance measure different things.

Install TensorTrace