Conditions for credible aggregation

The evidence that different claims about LLM agents in markets require. The checklist is built from Table 6 of the review and lists what each claim cannot do without; it is not a validity score. The five conditions and five threats come from Sections 8 and 7.

Check a claim

The first step is to state the claim: that an LLM population reproduces a human market outcome, that the LLM component explains outcomes beyond simpler agent rules, that configured or deployed agents behave in documented ways or change outcomes, that a decision tool performs, or that an instrument measures a construct.

Choose the type of claim above to see the comparison, controls and replication it requires, and the diagnostics and documentation that help to judge the evidence.

A study can support one claim and not another; therefore, no single score can summarize it. Section 8

Table 6 as printed in the review

Table 6 Evidence required for common aggregation claims

Claim	Necessary comparison	Necessary controls and replication	Useful diagnostics	Documentation that supports assessment
Human surrogate: an LLM population reproduces a human market outcome	A human benchmark under the same design, concurrent or published, on the outcome claimed; agreement on the distributional features the claim relies on, not merely similar dispersion	Remembered outcomes excluded; familiarity with a replicated design checked; independently repeated markets	Dependence relative to humans; stability across wordings and versions; non-LLM agents under the same rules	Dated model identifiers, full prompts, raw outputs
Explanatory value: the LLM component explains outcomes beyond simpler agent rules	Non-LLM agents (random, heuristic, or rule-based) under the same institution and information	Independently repeated markets	Ablations of LLM-specific features, such as reasoning, memory, or personas; dependence among agents	Rules of the baseline agents; configuration of the LLM agents
AI economic participant: configured or deployed LLM agents behave in documented ways or change market outcomes	Simulations: the configurations represented and varied, such as models, prompts, interfaces, and LLM shares. Production records: a defined population and observation window; a comparison group for relative claims and a credible counterfactual for effects on prices or stability	Simulations: remembered outcomes excluded when real episodes are replayed; independently repeated markets. Production records: sampling frame, exposure definitions, and timing; for effects, a design addressing selection into exposure and feedback between adoption and outcomes	Dependence in errors, actions, and holdings; human-likeness not required	Configuration surfaces and dated identifiers of the deployed agents
Financial decision tool: a system performs in markets	Baseline strategies under the same rules, net of frictions	Time-consistent evaluation that keeps future target information out of model knowledge, retrieval, data preparation, and execution, through a defensible post-cutoff test or an appropriately time-restricted model; repeated runs; relevant implementation frictions, including price impact when performance depends on prices that other agents move	Dependence when many copies are deployed	Test window, cost model, code
Experimental instrument: an LLM-based measure captures a construct	Meaning-preserving alternative wordings; a market-level observation when the construct is market-level	Remembered outcomes excluded; repeated runs and dated model versions	Sources of variation stated; varied institutions when the construct depends on them	Dated versions, prompts, raw outputs

Note: The necessary evidence depends on the claim rather than the agent’s role alone, and a study may make multiple claims. Online Resource 1 lists, for each of the 30 price-setting designs and each claimed role, the necessary components that are documented, not documented, or unknown; it reports no summary scores. Whether a study claims explanatory value is not coded; therefore, the non-LLM baseline is reported for every design. Documentation indicators support assessment but do not establish validity, and the real-news indicator records exposure, not contamination. The requirements that the coding does not record, such as institutional justification, time-consistent and net-of-friction performance, credible counterfactuals for the effects of deployed agents, and version stability, are discussed in the text rather than counted. Online Resource 2 lists the component rules and their corresponding limits.

Five conditions

The evidence in Sections 4 to 7 supports five proposed criteria for assessing whether individual LLM decisions provide credible building blocks for models of market interaction. Each addresses a documented failure; together, they guide validation rather than constitute a proven sufficiency test. Table 5 lists each condition and reports how many of the 30 price-setting designs document each of its components. This audit examines designs where agents set prices, while Section 6.3 discusses interactions through payoffs and networks, such as bank runs. Table 6, discussed after the five conditions, states what each of the five common claims requires.

Interaction

Aggregate outcomes arise from agents’ interaction through prices, payoffs, or networks

Observe the outcomes that arise from the interaction. Averaged expectations can track survey means (Bybee, 2023; Graham et al., 2026), and pooled digital-twin signals can predict returns (Bowles et al., 2026) without showing behavior when agents move prices. A rule measured among copies of one model can also change when the other agents differ (Section 6.4). Institutions can reverse model rankings even with agents and payoffs fixed (Chupilkin, 2026).

Documented among the 30 price-setting designs: 30 of 271 primary studies (4 of them also elicit decisions independently); 215 elicit decisions independently

Illustrative designs: Henning et al. (2025); Chupilkin (2026); Saxena et al. (2026)

See the component audit for condition 1

Heterogeneity

Sources of heterogeneity stated; heterogeneity validated for the claimed role; dependence reported; effective agent count when averaging assumptions apply

Measure heterogeneity and dependence. Surrogate populations require validation against relevant human populations, and participant populations require defensible representations of the deployed configurations. The three market studies that compare dispersion with same-design human data reach different verdicts (Section 6.1), and a comparison is a test, not a pass. Initializing traits from investor records (Yang et al., 2025) does not establish a validated population.

The five dependence studies measured different objects: error diagnostics in two, action correlation or overlap in two, and holdings overlap and common buying in one (Section 6.5). Where outcomes permit, designs should report error covariance, alongside mean bias and individual variance. Effective agent counts require the averaging assumptions stated above. Human histories, mixed families, and epistemic roles are candidate interventions whose effects need validation, rather than automatic solutions to dependence.

Documented among the 30 price-setting designs: 3 compare between-agent dispersion with human markets of the same design (2 unknown), 1 finding similar dispersion; 1 initializes traits from investor records; 14 homogeneous by design; 5 measure dependence (2 unknown; errors in 2, action correlation or overlap in 2, holdings and common buying in 1)

Illustrative designs: Begin et al. (2026); Saxena et al. (2026); outside the 30: Feng et al. (2025)

See the component audit for condition 2

Information

Information structure stated; shared inputs identified; remembered outcomes excluded

Specify the information environment and exclude remembered outcomes. Higher-order information demands (Galanis, 2026), shared narratives (Ross et al., 2026), and networked rumors (Yang et al., 2025) change the aggregation process. Researchers should identify shared inputs and use synthetic environments or appropriately time-restricted models when replaying actual events. Five price-setting designs use historical news or episodes, and only one replication checks for familiarity with the original experiment (Section 7.1).

Documented among the 30 price-setting designs: 5 feed agents real news or replay real episodes (2 unknown); none of the 28 read in full measures the effect of remembered outcomes on prices; 1 of 9 replications checks whether models know the original design (2 designs unknown)

Illustrative designs: Gao et al. (2024); Lin (2025); Galanis (2026)

See the component audit for condition 3

Institutions

Institution varied or justified; non-LLM baseline under the same rules when the LLM’s explanatory contribution is claimed

Justify or vary the institution and include a non-LLM baseline under the same rules whenever the claim is that the LLM component explains outcomes. Random or heuristic agents can reproduce important market outcomes (Gode & Sunder, 1993; Ruano & Rajan, 2026), so realism alone does not identify the LLM component’s explanatory contribution. A baseline is not needed to show that a specified LLM population differs from humans under a matched design or to describe what a deployed fleet did. Only four of the 30 designs include non-LLM agents under the same rules, and 20 compare with theory or equilibrium; an equilibrium prediction is not an interacting counterfactual population.

Documented among the 30 price-setting designs: 4 vary the auction or trading mechanism (2 unknown); 4 include non-LLM agents under the same rules (2 unknown); 20 compare with a theoretical or equilibrium prediction

Illustrative designs: Shah et al. (2025b); Chupilkin (2026); outside the 30: Ruano and Rajan (2026)

See the component audit for condition 4

Validation

Human benchmark matched to the claim; market as the unit of replication; dated configuration, prompts, and raw outputs

Match validation to the claim and replicate it at the level of the claim. Surrogate claims require a human benchmark under the same design, and simulation claims require independent repeated markets. Table 5 shows how unevenly they are documented. Claims drawn from production records, such as a live fleet (Barton et al., 2026b), require scrutiny of the sampling frame, exposure definitions, comparison group, and timing. Such scrutiny supports the description of observed behaviors and exposures. Estimating the effects on prices or stability additionally requires a credible counterfactual and a design that addresses selection into exposure and feedback between adoption and outcomes. Performance claims for decision tools likewise require time-consistent evaluation that keeps future information out of model knowledge, retrieval, data preparation, and execution. Controlled synthetic tasks can test mechanisms or capabilities but do not by themselves establish performance in actual markets. Online Resource 1 records the source-level evidence and uncertainty for each component, while Online Resource 2 contains the audit notes.

Documented among the 30 price-setting designs: 1 runs same-design human markets; 5 compare with published same-design human evidence, including summary statistics (2 unknown); 19 report repeated markets or runs (10 unknown, 1 not applicable); of the 28 read in full, 12 name a dated identifier for at least one model and 17 state that full prompts are published (documentation indicators)

Illustrative designs: Henning et al. (2025); Galanis (2026); outside the 30: Coqueret et al. (2026)

See the component audit for condition 5

Threats to validity

The five documented threats of Section 7, with the opening paragraph of each subsection.

7.1 Look-ahead, memorization, and contamination

Within training windows, recall and inference are difficult to separate. GPT-4o recalls macroeconomic series and prominent stock prices and assigns dates to undated headlines, with these abilities deteriorating after its cutoff date (Lopez-Lira et al., 2025). A fictitious 2010 cutoff in the system prompt does not erase later knowledge: the classification of subsequent gross domestic product growth remains 98% correct, against 40% for six releases after the actual cutoff. Firm knowledge also varies with size, prominence, and recency (Shah et al., 2025a).

7.2 Prompt sensitivity, frames, and personas

The behavior of LLMs is frame-dependent in ways that matter for aggregation. A behavioral mapping fitted under one wording transfers poorly to a payoff-equivalent wording, with a pooled mean total variation distance of 0.349 that ranges from 0.052 for a lottery task to 0.525 for dictator and trust games (Dong et al., 2026). Small changes to a prompt can change a market. Including the word “aversion” in the prompt through which the LLM decides clients’ willingness to trade produced no client trades in 200 simulated markets, all of which collapsed early (Vidler & Walsh, 2025). Irrelevant notes from the requestor, including an unrelated number, raise the incoherence of probability forecasts (Andrews & Sarkar, 2026).

7.3 Output variability, drift, and replicability

Repeated outputs can vary under a fixed model configuration, and setting the temperature to zero does not guarantee determinism. Through one provider’s application programming interface (API) at temperature zero, 200 identical runs classifying the same 100 annual-report excerpts yield four distinct regression t-statistics, whereas a locally served open-weight model returns a single value (Coqueret et al., 2026). A large open model returns identical outputs in only 12.5% of runs at temperature zero, while two small models are fully repeatable (Khatchadourian & Franco, 2025). The variation reaches financial outcomes. At the default temperature, about 5% of repeated runs of one textual-analysis design produce a tone coefficient significant at the 10% level by chance alone (Coqueret et al., 2026). Across three repetitions of one headline-sentiment strategy driven by LLaMA-2-7B, the standard deviation of total return rises from 0.25 to 2.72 percentage points as temperature rises from 0 to 1, whereas with GPT-3.5 it stays near 0.9 (Yu, 2023). According to its abstract, runs of one multi-agent trading system under varied sampling settings return 18.1% ± 2.8% with one backbone and 15.8% ± 4.2% with another, against 19.1% for buying and holding the stock (Koviazin et al., 2026).

7.4 Hyper-rationality and the human benchmark

Human-likeness depends on the comparator, and a human comparator is necessary only for claims about human fidelity. Of the 271 primary studies, 147 reported no human comparator in the material read (134 of them read in full), but only 10 of these 147 claimed a human-surrogate role. Of the 93 studies that claimed this role, 80 compared with human data: 33 with published values, 30 with a prior matched dataset, 14 with participants in the same study, and 3 with an external survey. Of the remaining 13, 10 reported no comparison (one coded from its abstract), and 3 could not be classified. The 44 comparisons with same-study participants or a prior matched dataset used matched human data, whereas published values may come from a different design or offer only summary statistics. Twenty-two studies compared the dispersion of behavior with human data, and six validated a calibration on held-out human behavior. These counts record comparisons made, not successful validations, and the coding of a surrogate role partly reflects a study’s aim to compare with humans. Comparators range from lay adults (Powdthavee, 2026) and undergraduates (Bertomeu et al., 2026) to advisers surveyed a decade earlier (Rumpf et al., 2026) and laboratory professionals (Hansen & Lee, 2025), so a claim of superiority requires specifying both population and task.

7.5 Evaluation practice and the inferential unit

In audits that re-run published LLM trading systems over later or longer windows or with realistic costs, the reported gains shrink or disappear (Li et al., 2025b; Li et al., 2026a; Ye et al., 2026). Re-running two published trading agents over a later one-year window with commissions, token costs, spreads, and market impact lowers TradingAgents’ portfolio Sharpe ratio from 0.43 gross to 0.22 net and QuantAgent’s from −0.96 to −1.15, leaving both below buying and holding (Ye et al., 2026). The same audit found that 35 of the 40 system-by-friction cells across five published systems were not modeled at all. In the audit for this review, 68 of the 140 full-text system papers that evaluate an implementable trading or investment strategy model trading costs, or 62 if six partial treatments are excluded (Table 3). Another audit recovers at least partial cost treatment for 14 of the 30 trade-relevant studies (Yao et al., 2026). In its demonstration with simple rules, replacing same-close execution without costs by next-close execution with costs of 10 basis points cuts a high-turnover strategy’s final equity multiple from 27.38 to 1.50. Only 26.8% of the 164 main-conference papers acknowledge look-ahead bias, and 1.2% acknowledge survivorship bias (Kong et al., 2026).