Can AI Beat Kalshi? How to Test an AI Trading Agent Safely
It's easy to connect an AI to a prediction market. It's much harder to know whether it's any good. A language model will give you a confident probability for anything, and on a liquid market the price is usually the better forecast. So the right first question isn't "how do I let the AI trade?" but "how do I find out whether it beats the market before any money is at risk?"
Here's a method that answers that, and how to run it for free on Kalshi.
The market price is already a forecast
A Kalshi contract trading at 63¢ means the market, as a whole, puts the event at roughly 63%. To make money, your AI has to be more accurate than that number, by enough to cover fees and the spread. Agreeing with the market earns nothing; being confidently different and wrong loses money.
Score forecasts, not trades
P&L over a few trades is mostly luck. A better test is to have the AI record a probability before it trades, then score every forecast once the market settles.
- The Brier score measures forecast accuracy: the squared difference between the forecast and what happened (1 or 0). Lower is better.
- Score the market price at the same moment the same way.
- If the AI's average score isn't clearly better than the market's over many markets, it doesn't have an edge, however convincing its reasoning sounds.
You need a lot of settled markets before any difference means something. A handful proves nothing either way.
Paper trade against the real order book
Paper trading with imaginary money against Kalshi's real markets and order books shows what the agent would actually have been filled at, including the spread, without risking anything. Run the agent this way until the forecast scores say whether it's worth more.
If you go live, limit the AI in code, not in the prompt
Telling a model "never risk more than $10" in a prompt isn't a limit. Limits need to be enforced by the software that places the orders:
- No buy without a recorded forecast that beats the price by a minimum edge after fees.
- Caps per order, per day and on open positions.
- Your approval on each live order, at least at first.
- A daily loss stop that halts new buys but never blocks exits.
- The agent can only close positions it opened.
Doing it with Krypt Trader
Krypt Trader is a free, open-source app built around exactly this method:
- It starts in Paper mode on Kalshi's real markets, with no Kalshi account needed.
- Connect Claude Code, Cursor, Claude Desktop or Codex over MCP, or run Autopilot with an Anthropic, OpenAI, OpenRouter or Gemini key, or a local Ollama or LM Studio model.
- The agent researches markets and must record a forecast before it can buy.
- The scoreboard scores every forecast against the market price (Brier score) and won't give a verdict before 30 settled markets.
- Going live needs a Kalshi API key and an explicit checklist, and the rails above are enforced by the backend.
FAQ
Can AI predict Kalshi markets?
An AI can produce forecasts, but on liquid markets the market price is usually more accurate. The only way to know if a particular agent is better is to score its forecasts against the price over many settled markets.
What is a Brier score?
A measure of forecast accuracy: the average squared difference between the probabilities forecast and the outcomes (1 if it happened, 0 if not). Lower is better, and 0 is perfect.
Can I test a Kalshi bot without money?
Yes. Paper trading against Kalshi's real order books uses imaginary money. Krypt Trader does this without needing a Kalshi account.
Is it safe to let an AI trade for me?
Only with limits enforced by software: forecasts required before buys, caps, approvals and a daily loss stop. Even then, you can lose money; nothing guarantees an edge.