I built an AI detective that isn’t allowed to have a hunch

I built an AI detective that isn't allowed to have a hunch

Four days, one graph database, and a murder mystery where nobody gets believed just because they sound confident.

The thing that bugged me

I kept noticing the same gap in every multi-agent system I read about. One agent says something. The others take it and run. There is no step anywhere in the pipeline where something stops and asks "hang on, how do you actually know that?"

We have a word for a system that believes whatever it was last told. We call it gullible. And we are busily building thousands of them.

So for Graph Hacks I built the missing step, and because an audit engine is a boring thing to look at, I wrapped a murder mystery around it.

How it works

Every suspect in the house has their own private memory. Not a shared world state they all read from, an actual separate memory, sealed off, that nobody else can see into. Things they saw. Things they were told. Things they worked out for themselves.

When a suspect makes a statement, the engine goes into that suspect's own memory and asks whether anything in there actually backs it up. Then it returns one of five verdicts: they witnessed it, their own memory contradicts it, nothing in their head touches it, they only partly remember it, or they are repeating something they heard from someone else.

The detective is an AI agent with seven tools. It picks what to ask next. It never gets to decide who is lying, because the verdicts are computed before it ever sees them. You could replace the detective with something that picks tools at random and every verdict in the case file would come out identical. You would just get a worse investigation.

That split is the whole design. The graph decides what is true. The model decides what is interesting.

The part I am most pleased with

Here is a problem I did not see coming until I was deep in it.

Someone tells you a rumour. They are not lying. They genuinely heard it from someone, and they are passing it along in good faith. Meanwhile someone else made the whole thing up.

Until you follow the chain, those two people look exactly the same. Both of them said a false thing. Both of them believed it when they said it, maybe.

So I made "X told me" into a real link in the graph. Not just a flag saying this is secondhand, but a record of who said it and which specific memory of theirs it came from. Which means I can walk into that person's memory and check. And if they heard it from someone else, I keep walking. Eventually either the trail ends at somebody who actually witnessed the thing, or it breaks.

The scoreboard counts those two roles separately. Starting a rumour costs you a lot. Repeating one costs you very little. I did not want the honest gossip getting punished like the liar.

Three things that went wrong, and what I got out of them

The time my world quietly doubled. I was regenerating a case into a database that still had the last run sitting in it. Everything appended instead of replacing. So I was auditing 252 statements in a world that should have had 126, and every single number came out wrong by a factor that looked completely plausible. Nothing crashed. No test failed. The percentages just quietly lied to me.

That one rearranged how I work. A green test suite tells you your code does what you told it to do. It tells you nothing about whether you told it the right thing. Now when a number looks fine, I go and count the rows myself.

The statement about no time at all. My engine checks whether a person's memory covers the window their statement claims. Someone says they were in the library from eight to nine, and I check how much of that hour their memory can actually account for.

Then a statement came through covering a window of zero length. Start time and end time identical. And my check said: supported. Because zero minutes of evidence does, technically, cover zero minutes of claim. The maths was perfect and the answer was garbage.

Now I have a standing rule. Any time I divide one span by another, I go and ask what happens when the bottom one is empty.

A number I left in that makes me look bad. One of my metrics came out at basically zero out of 182. It is in the README and it is in my results file and I am not hiding it.

When I dug into why, it turned out the metric had almost nothing to find. My suspects are scripted, so they barely produce the specific kind of false testimony that metric exists to catch. It is close to a structural zero rather than a broken engine.

I thought about quietly dropping it. Then I realised that a project whose entire point is "show me your evidence" cannot also be a project that hides its own worst number. So it stays.

The thing I actually learned

I set out to build a lie detector. What I ended up with is something that cannot detect lies at all, and is better for it.

A graph can prove that a statement has nothing behind it. It genuinely cannot tell you whether the person knew that when they said it. Someone with a gap in their alibi might be a murderer, or they might have gone for a smoke alone. The structure looks identical.

So I took intent out of the verdicts completely. The word "lie" appears nowhere in anything a user sees. What the system tells you is narrower and much more defensible: here is exactly how much of what this person said their own memory can account for, and here is the path to every piece of it.

That turned out to be the more interesting product anyway. "This agent is lying" is a claim I cannot back. "This claim is standing on nothing, and here is the walk I took to find that out" is one I can.

Code: https://github.com/AugustusWayne/alibi

ciao