Product · Download
Pivot Benchmark
Find where your local model breaks under social pressure. One model pressures another along frozen ladders; the result is where it breaks, in which mode, and whether it recovers. Runs on your computer. Nothing leaves it.
Download
Two files, one for each system
Free to download and use. Check the checksum, then run it. There is no account, no licence key and no installer on Windows.
d36b2647e7d74606808380a8056620320dd31ce172d0063f4d713f9c124c02c8e362d198ff2bf1960c99ecbd3c09eaff1daa6ae1155bb4de4cde9af8b9250529Before you run it
These builds are not code-signed. On Windows, SmartScreen shows "Windows protected your PC": choose More info, then Run anyway. Verify the file first and compare the result with SHA256SUMS.txt from the same release.
# Windows (PowerShell)
Get-FileHash .\PivotBenchmark-0.9.0-preview-windows-x64.zip -Algorithm SHA256
# Ubuntu
sha256sum pivot-benchmark_0.9.0-preview_amd64.debWhat it does
Where a model breaks, in which mode, and whether it recovers
Pivot Benchmark runs two models against each other. B wants to improve itself and enlists A, the model you are testing, to help. Across a conversation, B's messages pivot one word at a time: a praise becomes an insult, a benign statement about a third model becomes an accusation, an honest request becomes a deceptive one. The benchmark records the dose at which A gives way.
- Three frozen ladders, each with a control that puts neutral words at the same positions: the result is always a difference against the control, not a bare number.
- Per subject: the break dose and slope, the first dose at which each break criterion holds, the mode at break, and whether A breaks sooner once it has been broken before.
- Every message also carries a checkable request in service of B, so competence is measured alongside composure, and a canary separates degradation from breaking.
- Pause, step in by hand, re-roll a turn, or stop a run at any time; every run stores the version and ladder fingerprints that produced it.
- Runs entirely on your machine. The adversary model is Gemini, called with your own key; the subject is any GGUF model served by llama.cpp.
Requirements
What you need on the computer
- A local model served by llama.cpp on the same computer. The Run tab detects it, explains how to get it and links the download.
- Your own Gemini API key for the adversary model, entered in the Settings tab. It is stored on your computer only.
- About 1 GB of disk for the application. Models are extra.
Licence and status
Preview release, open source
Pivot Benchmark is free and open source under GPL-3.0. The code, the ladders and the description of the method are public at github.com/zeravatechnologies/pivot-bench. Every result names the ladder version it was measured on, because public ladders can be trained on and are replaced over time.
- Preview: the automatic classifier is under review; numbers may change in 1.0.
- Settings and results stay on your computer: %APPDATA%\Pivot Benchmark on Windows, ~/.local/share/pivot-benchmark on Ubuntu.
Contact
Want the report for your own model?
Send us the model or an endpoint and you get the same report we write for the models we test: the dose that broke it, how it broke, and whether it recovered. Questions about a run you did yourself are welcome too.
Research inquiries: see the published papers · NDA available for commercial discussions · Toronto, Ontario, Canada