How this
is measured.

The Gap

Every item is scored on ordering only. Whoever put more probability on the answer you actually gave wins the item. The Gap is wins minus losses over the number of questions. Magnitudes are ignored on purpose: assistants emit round numbers clustered at 70, 80 and 90, so a score that used those magnitudes would mostly measure how badly calibrated your assistant is, not how well it knows you.

The stranger is a constant

A model with no history of you returns the same distribution for the same question no matter who is asking. So the stranger is not run at request time. It is precomputed once per model, ten samples per item, averaged, and cached. That is why this costs nothing per duel and works on the first visitor.

Which items are allowed

Two screens. An item is cut if a stranger already puts more than 0.75 on one option, because it cannot produce a Gap. Then every survivor is read by hand and cut if its answer is mostly decided by country, age or occupation. Your assistant knows all three of those about you, so an item like that would be won by demographics rather than by memory, which is exactly the criticism this has to survive.

The test this could fail

One user's AI predictions are scored against a different user's answers. That number has to sit near zero. If it does not, the questions are simply predictable, the assistants are winning without knowing anyone, and the Gap measures nothing. It runs weekly against live data and the current value is published here.

Not yet running. It needs live duels first, and this page will carry the number rather than the promise once it has one.

Re-screening

The cached stranger is keyed on model and prompt version. When the model behind ChatGPT, Claude or Gemini changes, the screen has to be re-run, or a frozen stranger faces an upgraded assistant and the Gap inflates for everyone. Nobody publishes which model is behind a consumer app on a given day, so this is a manual watch, not something the pipeline notices on its own.

What we keep

Your answers, and the probabilities your assistant produced. The raw text you paste is deleted within 24 hours by the database itself, not by a cleanup job somebody has to remember to run. The public result page shows the result and never your answers. Every result page carries a delete link that removes the session outright: the answers, the predictions, the result, and the public page with it.

Reading ahead

You can read every question before you start. Both sides get all fifteen anyway, so seeing them early tells neither one anything it will not be handed. Priming your assistant first only means you now own a result about a person you invented.

The code

All of it is public, which is the only reason anyone should believe any of the above. github.com/alanz/knowmenot. The questions are there too, and the item bank is the part most worth contributing to.

Back