Good advice depends on history: what was decided for a client, why, and which rule applied. DecisionSynth tests whether your agent's memory can answer those questions, using realistic synthetic client decisions whose correct answers are known in advance.
Memory systems save space by summarizing and extracting facts. In financial advice, the detail they drop is often the one that matters: the reason a client broke from policy, or the limit that applied that year. An agent that loses it contradicts earlier advice, repeats mistakes, and can't explain itself to a compliance reviewer.
Popular memory benchmarks test whether an agent recalls a conversation. DecisionSynth tests whether it recalls decisions: questions like these.
Each system stored the same 543 client decisions, then answered 1,367 questions about them from whatever it kept. The test set is private and has never been published, so no system could have trained on it.
Held-out set, July 2026, run by the maintainers. A range means the result depends on which AI model writes the memories (Gemini 2.5 Flash or Claude Haiku 4.5). The gap between a system and the verbatim baseline is what its memory pipeline threw away. Full scoreboard and configuration
Feed a set of synthetic client decisions into your memory system. Each one records what triggered it, the options, any override and its reason, the outcome, and the IRS or Treasury figure behind it.
Your agent answers five kinds of question: recall a decision, explain its reason, find similar past cases, put events in order, and name the rule that applied.
Answers are checked against keys written by the same program that generated the decisions. Nobody labels anything by hand, and no AI grades the answers.
What one decision looks like:
{
"trigger": "promotion_or_raise",
"followed_policy": false,
"override_reason": "bracket_timing_judgment",
"outcome": "taxable_invest · approved",
"semantic_refs": ["retirement.401k.elective_deferral.under_50"]
}
Adding an AI assistant for advisors or clients? Prove it remembers client history before it goes live, and after every model upgrade.
See exactly what your pipeline keeps and what it drops on decision-shaped data, with scores you can reproduce from a fresh clone.
A documented, repeatable result for the approval file, instead of a reviewer reading transcripts by eye.
About 590 sample decisions with answers, the scoring code, and a baseline that runs with no API keys, so you get a first score in minutes.
1,979 decisions and 4,913 questions over 1,752 synthetic households, none of them in the benchmark. Test before launch, or give a new agent a realistic decision history.
Two pages covering what's included, how it's built, and pricing. Made to forward to whoever approves the purchase.
No. Every household and every decision is synthetic. There is no personal data and no data-use agreement to sign.
Public results come from a private set of households that has never been released, so no system can train on the answers. The scoring code is open, and anyone can rerun it.
Household finances and tax figures are calibrated to public sources such as the Federal Reserve's Survey of Consumer Finances and IRS publications. How often advisors override policy is set by us, not measured, and the methodology says so. Scores don't depend on those rates.
WealthSchema, which builds synthetic data and benchmarks for wealth software. Questions: support@wealthschema.com.