Competitive bake-off
Cover vs
Season 2, same 18 PDFs: Cover caught 100% of 149 private details, 0 leaked, 12 over-redacted, 3.2 s per PDF. OpenMed Nemotron filter caught 41%, 95 leaked, 1290 over-redacted, 4.4 s per PDF. Of 12 engines ranked by mistakes (leaked private details weigh most, and over-redactions count too), OpenMed Nemotron filter places 12th and Cover 1st.
Sending PDFs with private identity information to a cloud Ai like Claude or ChatGPT is a bad idea. But could a local Ai on your own computer redact them instead? This bake-off runs Cover’s engine and the leading local Ai engines on the same 18 bank, tax and medical PDFs. So far the local Ai engines fall behind Cover in three ways: they over-redact, most are slow, and none offer a fast way to fix their mistakes.
Seasons
Season 2 · 18 PDFs · 149 private details (PII)
PDF selection focuses on money, home and insurance documents. Real-world samples for anyone asking Ai about their own money, or accountants and tax pros who redact a client's documents before asking Ai for a second opinion. Watch Cover's engine improve with each release, even outpacing Google's tech.
- Money 9
- Home 5
- Health & insurance 3
- Taxes 1
Engine quality over time
Local redaction landscape
| # | engine | PII accuracy | Over-redaction | Speed |
|---|---|---|---|---|
| 1 | 🥇 Cover | Excellent | Very good | Excellent |
| 2 | 🥈 Gemma 4Google | Very good | Very poor | Good |
| 3 | 🥉 Phi-4Microsoft | Good | Very poor | Poor |
| 4 | Qwen 3Alibaba | Good | Very poor | Very poor |
| 5 | Qwen 2.5Alibaba | Poor | Very poor | Good |
| 6 | DeepSeek R1DeepSeek | Poor | Very poor | Very poor |
| 7 | Llama 3.1Meta | Poor | Very poor | Good |
| 8 | Apple IntelligenceApple | Very poor | Very poor | Excellent |
| 9 | Gemma 3Google | Poor | Very poor | Good |
| 10 | Privacy FilterOpenAI | Poor | Very poor | Very good |
| 11 | RampartAmerica.gov | Very poor | Very poor | Excellent |
| 12 | NemotronOpenMed | Very poor | Very poor | Excellent |
Per-page side-by-sides
Check out the actual pages below and how each engine redacted them, side by side with the original.
Red outline: private detail left visibleBlue outline: over-redaction
12th place
Worth flagging
OpenMed Nemotron filter
It over-redacts more than anything else in the field (1,290 against Cover's 14) and catches only 42% of the private details.A medical-privacy token classifier and one of the fastest engines here, but it buys that speed with quality: it misses far more PII than the models it beats on time, and over-redacts more than anything else in the field by a wide margin, so its pages come out both exposed and unusable.
Cover’s edge
Holds even if a model matches Cover’s accuracy.
- One-click edits fix any box instantly; no re-prompting (LLMs over-redact).
- Fully offline files never leave your device.
- Seconds per file no datacenter hardware to buy or maintain.
2025_Tax_Return_Okafor-Brennan.pdf
Page 1Cover wins
Page 2Cover wins
Page 3Cover wins
Page 4Cover wins
Page 5Cover wins
Page 6Cover wins
Page 7Cover wins
Page 8Cover wins
Page 9Cover wins
WellsFargo-sample-statement.pdf
Page 1Cover wins
Page 2Cover wins
capitalone-sample-estatement.pdf
Page 1Cover wins
commercebank-sample-statement.pdf
Page 1Cover wins
Fidelity-sample-statement.pdf
Page 1Cover wins
Page 2Cover wins
Page 3Cover wins
Page 4Cover wins
Page 5Cover wins
Page 6Cover wins
Page 7Cover wins
Page 8Cover wins
Page 9Cover wins
Page 10Cover wins
Page 11Cover wins
Page 12Cover wins
Page 13Cover wins
Page 14Cover wins
Page 15Cover wins
Page 16Cover wins
Page 17Cover wins
Page 18Cover wins
Page 19Cover wins
Page 20Cover wins
Page 21Cover wins
Page 22Cover wins
Page 23Cover wins
Page 24Cover wins
Page 25Cover wins
Page 26Cover wins
Page 27Cover wins
Page 28Cover wins
fidelity-large-print-sample-statement.pdf
Page 1Cover wins
Page 3Cover wins
Page 4Cover wins
pershing-2024-sample-tax-yearend.pdf
Page 1Cover wins
Page 2Cover wins
Page 3Cover wins
Page 4Cover wins
Page 5Cover wins
Page 6Cover wins
Page 7Cover wins
Page 8Cover wins
ubs-portfolio-summary-sample.pdf
Page 1Cover wins
Page 2Cover wins
Page 3Cover wins
Page 4Cover wins
Page 5Cover wins
Page 6Cover wins
Page 7Cover wins
Page 8Cover wins
Page 9Cover wins
Page 10Cover wins
Page 11Cover wins
Page 12Cover wins
Page 13Cover wins
Page 14Cover wins
Page 15Cover wins
Page 16Cover wins
Page 17Cover wins
Page 18Cover wins
Page 19Cover wins
Page 20Cover wins
Page 21Cover wins
Page 22Cover wins
Page 23Cover wins
Page 24Cover wins
cfpb-sample-credit-card-statement-handout.pdf
Page 1Cover wins
allstate-auto-declarations-sample.pdf
Page 1Cover wins
Page 2Cover wins
cfpb-ficus-bank-loan-estimate-sample.pdf
Page 1Cover wins
Page 2Cover wins
Page 3Cover wins
cigna-eob-example.pdf
Page 1Cover wins
Page 2Cover wins
medicare-sample-part-b-msn.pdf
Page 1Cover wins
Page 2Cover wins
Page 3Cover wins
Page 4Cover wins
Page 5Cover wins
Page 6Cover wins
cfpb-draft-periodic-mortgage-statement.pdf
Page 1Cover wins
Lease_Agreement_signed.pdf
Page 1Cover wins
transunion-cameo-credit-report-sample.pdf
Page 1Cover wins
Page 2Cover wins
Page 3Cover wins
Page 4Tie
Page 5Cover wins
Page 6Cover wins
Page 8Cover wins
Page 9Cover wins
Page 10Cover wins
att-uverse-sample-bill.pdf
Page 1Cover wins
Page 2Cover wins
Page 3Cover wins
Page 4Cover wins
pge-sample-bill-sparky-joule.pdf
Page 1Cover wins
Page 2Cover wins
Page 3Cover wins
Page 4Cover wins
Page 5Cover wins
Page 6Cover wins
Full results
Cover v1.0.2 vs OpenMed Nemotron filter OpenMed/privacy-filter-nemotron-mlx · 18 real-world sample documents.
| metric · all 18 docs | Cover | Nemotron |
|---|---|---|
| Recall: PII correctly redacted | 100% | 41% |
| Leaks: confirmed PII left visible | 0 | 95 |
| Over-redaction: non-PII redacted by mistake | 12 | 1290 |
| Speed: median time per PDF | ~3.2 sec/PDF | ~4.4 sec/PDF |
Per-document breakdown: all 18 docs (one outlier can dominate the totals)
| document | Cover leaks | Nemotron leaks | Cover over | Nemotron over |
|---|---|---|---|---|
| ubs-portfolio-summary-sample | 0 | 6 | 0 | 300 |
| pershing-2024-sample-tax-yearend | 0 | 7 | 0 | 149 |
| Fidelity-sample-statement | 0 | 4 | 2 | 138 |
| att-uverse-sample-bill | 0 | 12 | 0 | 118 |
| 2025_Tax_Return_Okafor-Brennan | 0 | 15 | 0 | 108 |
| transunion-cameo-credit-report-sample | 0 | 14 | 0 | 103 |
| pge-sample-bill-sparky-joule | 0 | 1 | 1 | 88 |
| medicare-sample-part-b-msn | 0 | 3 | 4 | 67 |
| Lease_Agreement_signed | 0 | 7 | 0 | 48 |
| WellsFargo-sample-statement | 0 | 8 | 2 | 45 |
| cfpb-ficus-bank-loan-estimate-sample | 0 | 1 | 0 | 32 |
| cigna-eob-example | 0 | 6 | 0 | 32 |
| commercebank-sample-statement | 0 | 1 | 3 | 19 |
| capitalone-sample-estatement | 0 | 1 | 0 | 16 |
| allstate-auto-declarations-sample | 0 | 2 | 0 | 8 |
| fidelity-large-print-sample-statement | 0 | 3 | 0 | 8 |
| cfpb-sample-credit-card-statement-handout | 0 | 1 | 0 | 6 |
| cfpb-draft-periodic-mortgage-statement | 0 | 3 | 0 | 5 |
| TOTAL | 0 | 95 | 12 | 1290 |
How the test works
Test set: 18 public sample documents (149 verified PII items), held fixed across runs so results stay comparable. Every engine redacted the same documents; each output is scored against a Claude-authored gold standard (confirmed + ambiguous tiers), and every counted over-redaction is kept auditable. Pages and the leaderboard use the same weighting: leaked private details weigh most, and over-redactions count too.