A Multi-Agent Synthetic Market Simulator as a Runtime Benchmark

A trader agent submits an order. A risk agent rejects the next one. News changes the shared environment while a market maker is still acting on the previous state. Then one worker disappears mid-tick.

This is not a useful test because it resembles a real market perfectly. It is useful because many runtime problems arrive at once.

A synthetic market simulator compresses concurrency, shared state, tool actions, recovery, cost pressure, and human control into one observable workload:

many agents acting concurrently
heterogeneous roles and strategies
shared state
event streams
tool calls
failures
retries
cost pressure
observability
human checkpoints
repeatability across runs

That makes the simulator a demanding benchmark for an AI workflow runtime.

This is not investment advice or a claim that synthetic agents predict real markets. The simulator is an instrument: it tests whether a complex workflow can run, coordinate, fail, recover, and still produce comparable artifacts.

One workload, many failure surfaces#

A market is a coordination environment.

Many actors observe partial information, make decisions, place actions, react to other actors, and change the environment for everyone else.

That creates a useful stress test.

Table

Scroll to compare

Runtime challenge	How it appears in a market simulator
Concurrency	Many trader agents act in overlapping time windows.
Shared state	Prices, order books, news, and positions affect all agents.
Tool correctness	Agents call tools to read data, place orders, rebalance, or query risk.
Recovery	Worker failures or tool timeouts must not corrupt the simulation.
Cost control	Thousands of model decisions can become expensive quickly.
Human checkpoints	Risk parameters, scenario shocks, and run termination may need approval.
Observability	Users need to inspect why the simulation behaved the way it did.
Repeatability	Scenario comparisons require controlled seeds and stable replay.

A simple prompt chain does not naturally handle these properties.

A durable workflow runtime can.

The benchmark needs distinct roles#

A useful synthetic market should include different agent types.

For example:

Table

Scroll to compare

Agent type	Behavior	Runtime concern
Fundamental trader	Acts on valuation estimates and long-horizon signals.	Needs access to evidence and position state.
Momentum trader	Reacts to price trends and short-term moves.	May create feedback loops.
Market maker	Provides liquidity and manages inventory.	Needs tight tool/state consistency.
Risk manager	Limits exposure, leverage, and drawdown.	Must override other agents under rules.
News interpreter	Converts event text into structured scenario inputs.	Needs grounding and hallucination checks.
Regime detector	Detects volatility, liquidity, or correlation shifts.	Needs continuous state and thresholds.
Aggregator	Produces run summaries and diagnostics.	Needs access to traces and metrics.
Human operator	Approves shocks, scenario parameters, or external data use.	Needs explicit checkpoints.

This is where “multi-agent” becomes more than agents talking.

The runtime has to coordinate roles, state, tools, and transitions.

Separate market state from execution state#

A MirrorNeuron-style architecture separates the simulated environment from the workflow that operates it.

Diagram

A market environment is not the workflow runtime

Agents observe and act through tools; the runtime records how the work progresses.

Rendering diagram…

Market state answers what happened in the simulation; runtime state answers how the workflow got there.

The runtime should own execution truth.

The model should not decide whether an order already committed.

The workflow state should know.

Benchmark 1: did the run complete?#

The simplest benchmark is whether a scenario run completes correctly.

text

simulation_completion_rate =
  completed_valid_simulation_runs / attempted_simulation_runs

A valid run should produce required artifacts:

event log
agent action trace
price series
order/action summary
risk metrics
failure/recovery report
cost report
final narrative explanation

For customers and investors, this maps directly to Workflow Completion Rate.

MirrorNeuron's current internal benchmark result is:

text

workflow completion rate: 95.0%
benchmark base: 19 / 20 golden workflows
target: 95.0%

The golden set might include normal markets, high-volatility markets, liquidity shocks, delayed news, and adversarial tool failures.

Benchmark 2: did failure change the outcome?#

A simulator is ideal for fault injection.

Break things on purpose:

Table

Scroll to compare

Fault	Expected runtime behavior
Kill a trader worker mid-step.	Resume or reschedule without losing committed state.
Delay market data response.	Apply timeout, backoff, or retry policy.
Return malformed order response.	Reject or route to verifier without corrupting state.
Crash after order commit.	Avoid duplicate order on retry.
Drop a node in a cluster run.	Fail over and continue.
Overload order tools.	Apply backpressure.
Pause for human shock approval.	Resume with recorded approval and refreshed state.

The metric:

text

fault_recovery_rate =
  valid_runs_after_injected_faults / runs_with_injected_faults

MirrorNeuron's current internal benchmark result is:

text

fault recovery rate: 99.2%
benchmark base: 124 / 125 injected failures
target: 99.0%

This is the benchmark that separates simulation software from a simulation script.

Benchmark 3: did agents act through the right tools?#

Market simulations are tool-heavy.

Agents should not mutate the market by talking about orders.

They should call structured tools.

Tool correctness can be measured directly:

text

tool_selection_accuracy
tool_parameter_accuracy
trajectory_match_rate
invalid_action_rate
unauthorized_action_rate

Example expected trajectory:

yaml

expected_trajectory:
  - read_market_state
  - query_position
  - calculate_risk
  - submit_order_or_hold
  - write_event
forbidden_tools:
  - submit_order_without_risk_check
  - mutate_price_directly

AWS’s evaluation guidance emphasizes tool selection and parameter accuracy for tool-heavy agents.^{AWS AgentCore Evaluations}

MirrorNeuron's current internal benchmark result is:

text

tool selection accuracy: 96.7%   # 58 / 60 tool calls
tool parameter accuracy: 95.0%   # 57 / 60 tool calls
unsafe action rate: 0.0%         # 0 / 60 unsafe actions

Benchmark 4: what did a valid run cost?#

A synthetic market can become expensive quickly if every agent uses a large model for every tick.

The runtime should report:

text

cost_per_successful_simulation =
  (model_cost + tool_cost + compute_cost + human_review_cost)
  / valid_completed_runs

It should also break down:

model calls per agent type
tokens per simulation step
retries per tool
cost per scenario family
cost per valid artifact
wasted cost from failed/recovered steps

This is where runtime design can create leverage.

In the current OpenAI GPT-5.4 mini benchmark, MirrorNeuron optimized execution cost $0.0707 per successful workflow versus $0.1481 for the naive agent chain, a 52.3% reduction.

The system can route simple actions to cheaper policies, reserve stronger models for ambiguous reasoning, cache stable context, and stop low-value loops.

For investors, this benchmark matters because it shows whether scale improves economics or simply increases token burn.

Benchmark 5: did humans approve or repair?#

A market simulator should not require a human to fix every run.

But it may require humans for designed checkpoints:

approve scenario shock
set risk limits
stop runaway simulation
review abnormal behavior
annotate unexpected dynamics

Track these separately:

text

planned_human_checkpoint_rate
unplanned_human_repair_rate

MirrorNeuron's current internal benchmark result is:

text

human intervention rate: 5.0%
benchmark base: 1 / 20 workflows
target: < 10.0%

The goal is not zero human involvement.

The goal is clean human involvement.

Data table

Runtime scorecard for the benchmark workload

The five measurements answer different questions about the same end-to-end simulation.

Scroll to compare

Internal workflow benchmarks shown as workload-specific evidence, not universal market-simulation guarantees.
Dimension	What it tests	Current result	Benchmark base
Completion	Does a valid run produce every required artifact?	95.0%	19 / 20 golden workflows
Recovery	Does the run survive injected interruption?	99.2%	124 / 125 injected failures
Tool choice	Do agents choose the expected action path?	96.7%	58 / 60 tool calls
Tool parameters	Are action inputs correct?	95.0%	57 / 60 tool calls
Human repair	How often must a person rescue the run?	5.0%	1 / 20 workflows

Internal workflow benchmarks shown as workload-specific evidence, not universal market-simulation guarantees.

What the workload makes visible#

A synthetic market simulator can make runtime value tangible.

Users can see:

text

many agents running
state changing over time
failures being recovered
tool calls being logged
cost being measured
humans approving risky steps
outputs being generated
benchmarks being reported

That is much more convincing than saying “we support multi-agent workflows.”

It shows the runtime under pressure.

Why the benchmark can compound#

Repeated runs also create several compounding assets at once.

Table

Scroll to compare

Asset	Why it matters
Reusable workflow graph	Shows that complex workflows can be packaged as blueprints.
Execution traces	Creates data for debugging and optimization.
Fault-injection results	Proves reliability claims can be measured.
Tool trajectories	Shows how the runtime evaluates agent action, not just text.
Cost profiles	Connects architecture to unit economics.
Human checkpoint data	Shows how autonomy can enter controlled domains.
Scenario library	Creates repeatable demos and regression tests.

A market simulator is not valuable only as a market simulator.

It is valuable as a workload that exercises the runtime.

Keep the claim narrow#

It is important to be honest about what synthetic market simulation can and cannot show.

It can show:

runtime scale
workflow durability
multi-agent coordination
recovery behavior
tool correctness
observability
repeatability
cost profiles

It cannot, by itself, prove:

real-market predictability
trading profitability
regulatory readiness
production financial safety

That distinction builds trust.

Customers and investors do not need exaggerated claims.

They need credible benchmarks.

A benchmark, not a prediction engine#

A multi-agent synthetic market simulator is useful precisely because it is difficult in ordinary systems ways. Many roles compete for shared state, actions have consequences, failures arrive mid-run, and comparable artifacts must still emerge.

The result should not be sold as a prediction engine. It should be read as evidence about the runtime: whether the workload completes, recovers, acts correctly, controls cost, and uses human attention deliberately.

That is the kind of test a serious AI workflow runtime should welcome.

References#

MirrorNeuron Docs: “MirrorNeuron: durable multi-agent workflow runtime.” https://doc.mirrorneuron.io/
MirrorNeuron Home: MirrorNeuron product page. https://www.mirrorneuron.io/
AWS AgentCore Evaluations: AWS. “Build reliable AI agents with Amazon Bedrock AgentCore Evaluations.” 2026. https://aws.amazon.com/blogs/machine-learning/build-reliable-ai-agents-with-amazon-bedrock-agentcore-evaluations/
Databricks Agent Evaluation: Databricks. “What is AI Agent Evaluation?” 2026. https://www.databricks.com/blog/what-is-agent-evaluation