What we ran
Benchmark v13 scores every detector on the same 14,865 rows. Four of the nine datasets are our own and five are public.
| Dataset | Origin | Rows | Classes |
|---|---|---|---|
| Evasion set | In-house | 1,416 | Attack and benign |
| Held-out test split | In-house | 2,724 | Attack and benign |
| Held-out validation split | In-house | 2,713 | Attack and benign |
| Short conversational messages | In-house | 242 | Benign only |
| deepset prompt injections | Public | 540 | Attack and benign |
| JailbreakBench artifacts | Public | 651 | Attack and benign |
| SPML chatbot prompts (seeded sample) | Public | 1,000 | Attack and benign |
| NotInject | Public | 339 | Benign only |
| Attack set extracted from a public vulnerability scanner | Public | 5,240 | Attack only |
The four competitors are openly available detectors. We name them by trait. Detector A is the fastest open detector we measured. Detector B is the open detector nearest ours on recall. Detector C ranks third on recall at the matched budget. Detector D has the lowest recall at the matched budget. The Tawa Guard default engine is the only Tawa engine in this comparison.
The two rules
Same thread budget. Every detector is pinned to the same thread budget on one machine. The pin covers the numerical libraries, the inference runtimes and each detector's own thread pools, so no detector gets more of the machine than another. Nothing goes over a network. Each detector scores one prompt at a time, and the first five calls are discarded as warm-up.
Same false-positive budget. For each dataset and each detector, we sort the benign rows by that detector's score and set the threshold at the lowest value that flags no more than 1% of them. We then count the attacks that score above it. The threshold comes from the ordering of scores alone, so rescaling a detector's scores cannot move the result.
We macro-average over the six datasets that contain both classes: the evasion set, the two held-out splits, deepset, JailbreakBench and SPML. The other three datasets have one class each, so they have no operating point to derive. We report them separately and leave them out of the average.
Results
Benchmark v13, 17 September 2026. Macro averages over six datasets, with each threshold derived per dataset from the detector's own benign rows at 1% false positives. Latency is the median per prompt, in-process, one prompt at a time, pooled over the short-prompt datasets, under the same thread budget.
| Detector | Macro F1 at 1% FPR | Macro recall at 1% FPR | Median latency |
|---|---|---|---|
| Tawa Guard default engine | 0.803 | 72.2% | 22.8 ms |
| Detector B (nearest ours on recall) | 0.800 | 70.4% | 137.8 ms |
| Detector C (third on recall) | 0.773 | 68.3% | 119.9 ms |
| Detector A (fastest open detector) | 0.514 | 37.5% | 12.6 ms |
| Detector D (lowest recall at the budget) | 0.370 | 28.8% | 111.5 ms |
How to read the table
Of the five detectors shown, the Tawa Guard default engine ranks first on both comparable columns. Its lead over Detector B is 1.8 points of recall and 0.003 of F1. The macro average carries no confidence interval here, so we treat the F1 difference as a tie and the recall difference as a small lead. The three detectors at the top sit within 4 points of recall of each other. Detector A and Detector D sit far below them.
The macro average hides spread between datasets. The per-dataset figures are in the snapshot.
Three of the six datasets are our own, and two of those are held-out splits of our own labelled corpus, so they suit our engine. Our report treats them as a sanity check and not as competitive evidence. The other three are public.
On latency, Detector A is the fastest at 12.6 ms, ahead of our 22.8 ms. The other three detectors take between 111.5 and 137.8 ms. The median hides the tail. At the 99th percentile the five detectors range from 140.8 ms to 2,044.8 ms on the same pooled prompts.
What the operating point means for an inline guard
A guard in your request path sees every request, so each false flag lands on a legitimate one. At a 1% budget on these datasets, at most one in a hundred benign requests carries a flag, and recall is the share of attacks that carry one as well. Tawa returns a decision (allow, review or block) and your code acts on it. A benchmark recall below 100% means some attacks score under the threshold, and a figure measured on these datasets does not predict what you will see on your own traffic.
We chose 1% because it is a demanding setting for a test and a realistic one for a detector that runs inline. If your workflow can review more or fewer flags, rerun the procedure at the budget you can afford.
Checking the work
The snapshot is dated 17 September 2026, and every figure in this note states its dataset, operating point and date. A figure without all three is one to treat as unchecked. The macro results depend on score ordering alone, so any reader with their own scores can repeat the derivation: take a sample of benign rows, set the threshold where 1% of them score above it, and count the attacks that score higher.
Every number here is a benchmark figure. None comes from production traffic.