Competitive bake-off

Cover vs

Season 2, same 18 PDFs: Cover caught 100% of 149 private details, 0 leaked, 12 over-redacted, 3.2 s per PDF. Google Gemma 3 caught 69%, 53 leaked, 623 over-redacted, 24 s per PDF. Of 12 engines ranked by mistakes (leaked private details weigh most, and over-redactions count too), Google Gemma 3 places 9th and Cover 1st.

4.3B · gemma3Last updated October 3, 2026 · engine v1.0.2

Sending PDFs with private identity information to a cloud Ai like Claude or ChatGPT is a bad idea. But could a local Ai on your own computer redact them instead? This bake-off runs Cover’s engine and the leading local Ai engines on the same 18 bank, tax and medical PDFs. So far the local Ai engines fall behind Cover in three ways: they over-redact, most are slow, and none offer a fast way to fix their mistakes.

Seasons
Season 3
Season 2 · 18 PDFs · 149 private details (PII)

PDF selection focuses on money, home and insurance documents. Real-world samples for anyone asking Ai about their own money, or accountants and tax pros who redact a client's documents before asking Ai for a second opinion. Watch Cover's engine improve with each release, even outpacing Google's tech.

  • Money 9
  • Home 5
  • Health & insurance 3
  • Taxes 1
Engine quality over time
CoverGemma 3
60%70%80%90%100%101001000% PII caughtover-redactions (log)Season 1 · 15 docsSeason 1 · 15 docsSeason 2 · 18 docsSeason 2 · 18 docs0.12.70.12.80.13.20.13.30.14.00.14.10.15.10.16.40.16.50.16.60.18.40.18.50.20.11.0.2v0.12.7: Gemma 3 caught 78% (22 leaked)v0.12.8: Gemma 3 caught 78% (22 leaked)v0.13.2: Gemma 3 caught 78% (22 leaked)v0.13.3: Gemma 3 caught 78% (22 leaked)v0.14.0: Gemma 3 caught 78% (22 leaked)v0.14.1: Gemma 3 caught 78% (22 leaked)v0.15.1: Gemma 3 caught 78% (22 leaked)v0.16.4: Gemma 3 caught 78% (21 leaked)v0.16.5: Gemma 3 caught 78% (37 leaked)v0.16.6: Gemma 3 caught 78% (37 leaked)v0.18.4: Gemma 3 caught 78% (37 leaked)v0.18.5: Gemma 3 caught 78% (37 leaked)v0.20.1: Gemma 3 caught 78% (37 leaked)v1.0.2: Gemma 3 caught 69% (53 leaked)v0.12.7: Cover caught 75% of PII (21 leaked)v0.12.8: Cover caught 77% of PII (19 leaked)v0.13.2: Cover caught 79% of PII (18 leaked)v0.13.3: Cover caught 86% of PII (12 leaked)v0.14.0: Cover caught 86% of PII (12 leaked)v0.14.1: Cover caught 86% of PII (12 leaked)v0.15.1: Cover caught 91% of PII (5 leaked)v0.16.4: Cover caught 95% of PII (2 leaked)v0.16.5: Cover caught 98% of PII (3 leaked)v0.16.6: Cover caught 98% of PII (3 leaked)v0.18.4: Cover caught 98% of PII (3 leaked)v0.18.5: Cover caught 98% of PII (3 leaked)v0.20.1: Cover caught 98% of PII (3 leaked)v1.0.2: Cover caught 100% of PII (0 leaked)78%69%75%100%v0.12.7: Gemma 3 over-redacted 723 non-PII itemsv0.12.8: Gemma 3 over-redacted 723 non-PII itemsv0.13.2: Gemma 3 over-redacted 723 non-PII itemsv0.13.3: Gemma 3 over-redacted 723 non-PII itemsv0.14.0: Gemma 3 over-redacted 723 non-PII itemsv0.14.1: Gemma 3 over-redacted 723 non-PII itemsv0.15.1: Gemma 3 over-redacted 724 non-PII itemsv0.16.4: Gemma 3 over-redacted 707 non-PII itemsv0.16.5: Gemma 3 over-redacted 692 non-PII itemsv0.16.6: Gemma 3 over-redacted 692 non-PII itemsv0.18.4: Gemma 3 over-redacted 692 non-PII itemsv0.18.5: Gemma 3 over-redacted 692 non-PII itemsv0.20.1: Gemma 3 over-redacted 692 non-PII itemsv1.0.2: Gemma 3 over-redacted 623 non-PII itemsv0.12.7: Cover over-redacted 15 non-PII itemsv0.12.8: Cover over-redacted 15 non-PII itemsv0.13.2: Cover over-redacted 51 non-PII itemsv0.13.3: Cover over-redacted 16 non-PII itemsv0.14.0: Cover over-redacted 16 non-PII itemsv0.14.1: Cover over-redacted 12 non-PII itemsv0.15.1: Cover over-redacted 22 non-PII itemsv0.16.4: Cover over-redacted 36 non-PII itemsv0.16.5: Cover over-redacted 62 non-PII itemsv0.16.6: Cover over-redacted 26 non-PII itemsv0.18.4: Cover over-redacted 27 non-PII itemsv0.18.5: Cover over-redacted 22 non-PII itemsv0.20.1: Cover over-redacted 19 non-PII itemsv1.0.2: Cover over-redacted 12 non-PII items7236231512
Per-version detail: all 14 releases (recall · leaks · over-redaction · pages better / tied / worse)
versionseasondatePII caughtover-redactionpages better / tied / worsedocs
1.0.2S22026-10-03100% vs 69%12 vs 623n/a18
0.20.1S22026-09-0398% vs 78%19 vs 692n/a18
0.18.5S22026-08-2798% vs 78%22 vs 692n/a18
0.18.4S22026-08-2398% vs 78%27 vs 692n/a18
0.16.6S22026-07-1898% vs 78%26 vs 692n/a18
0.16.5S22026-07-1798% vs 78%62 vs 692n/a18
0.16.4S12026-07-1695% vs 78%36 vs 707n/a15
0.15.1S12026-07-0791% vs 78%22 vs 724n/a15
0.14.1S12026-06-2986% vs 78%12 vs 723n/a15
0.14.0S12026-06-2686% vs 78%16 vs 723n/a15
0.13.3S12026-06-1586% vs 78%16 vs 723n/a15
0.13.2S12026-06-1579% vs 78%51 vs 723n/a15
0.12.8S12026-06-0477% vs 78%15 vs 723n/a15
0.12.7S12026-06-0375% vs 78%15 vs 723n/a15

14 versions · 16 raw runs · de-duped latest-per-version

Local redaction landscape
↙ the goal013000100Over-redactionLeaksCoverGemma 4Phi-4Qwen 3Qwen 2.5DeepSeek R1Llama 3.1Apple IntelligenceGemma 3Privacy FilterRampartNemotron
#enginePII accuracyOver-redactionSpeed
1🥇 CoverExcellentVery goodExcellent
2🥈 Gemma 4GoogleVery goodVery poorGood
3🥉 Phi-4MicrosoftGoodVery poorPoor
4Qwen 3AlibabaGoodVery poorVery poor
5Qwen 2.5AlibabaPoorVery poorGood
6DeepSeek R1DeepSeekPoorVery poorVery poor
7Llama 3.1MetaPoorVery poorGood
8Apple IntelligenceAppleVery poorVery poorExcellent
9Gemma 3GooglePoorVery poorGood
10Privacy FilterOpenAIPoorVery poorVery good
11RampartAmerica.govVery poorVery poorExcellent
12NemotronOpenMedVery poorVery poorExcellent

Per-page side-by-sides

Check out the actual pages below and how each engine redacted them, side by side with the original.

Red outline: private detail left visibleBlue outline: over-redaction
9th place
Worth flagging
Google Gemma 3
PII accuracy: PoorOver-redaction: Very poorSpeed: Good ~24 sec/PDF

Catches 69% of the private details but over-redacts about 44 times as much as Cover, so its pages are hard to actually use.

Cover’s edge
Holds even if a model matches Cover’s accuracy.
  • One-click edits fix any box instantly; no re-prompting (LLMs over-redact).
  • Fully offline files never leave your device.
  • Seconds per file no datacenter hardware to buy or maintain.

2025_Tax_Return_Okafor-Brennan.pdf

Tax return · 15 private details
Page 1Cover wins
Original
Page 1, original
CoverNo mistakes
Page 1, redacted by Cover
Gemma 33 left visible21 over-redacted
Page 1, redacted by Gemma 3
Page 2Cover wins
Original
Page 2, original
CoverNo mistakes
Page 2, redacted by Cover
Gemma 32 left visible40 over-redacted
Page 2, redacted by Gemma 3
Page 3Cover wins
Original
Page 3, original
CoverNo mistakes
Page 3, redacted by Cover
Gemma 31 left visible3 over-redacted
Page 3, redacted by Gemma 3
Page 4Cover wins
Original
Page 4, original
CoverNo mistakes
Page 4, redacted by Cover
Gemma 32 left visible
Page 4, redacted by Gemma 3
Page 5Cover wins
Original
Page 5, original
CoverNo mistakes
Page 5, redacted by Cover
Gemma 33 left visible
Page 5, redacted by Gemma 3
Page 6Cover wins
Original
Page 6, original
CoverNo mistakes
Page 6, redacted by Cover
Gemma 31 over-redacted
Page 6, redacted by Gemma 3
Page 7Cover wins
Original
Page 7, original
CoverNo mistakes
Page 7, redacted by Cover
Gemma 31 left visible6 over-redacted
Page 7, redacted by Gemma 3
Page 8Cover wins
Original
Page 8, original
CoverNo mistakes
Page 8, redacted by Cover
Gemma 31 left visible8 over-redacted
Page 8, redacted by Gemma 3
Page 9Cover wins
Original
Page 9, original
CoverNo mistakes
Page 9, redacted by Cover
Gemma 318 over-redacted
Page 9, redacted by Gemma 3

WellsFargo-sample-statement.pdf

Bank statement · 8 private details
Page 1Cover wins
Original
Page 1, original
CoverNo mistakes
Page 1, redacted by Cover
Gemma 31 left visible
Page 1, redacted by Gemma 3
Page 2Cover wins
Original
Page 2, original
Cover2 over-redacted
Page 2, redacted by Cover
Gemma 34 over-redacted
Page 2, redacted by Gemma 3

capitalone-sample-estatement.pdf

Bank statement · 4 private details
Page 1Cover wins
Original
Page 1, original
CoverNo mistakes
Page 1, redacted by Cover
Gemma 32 over-redacted
Page 1, redacted by Gemma 3

commercebank-sample-statement.pdf

Bank statement · 4 private details
Page 1Cover wins
Original
Page 1, original
Cover3 over-redacted
Page 1, redacted by Cover
Gemma 31 left visible15 over-redacted
Page 1, redacted by Gemma 3

Fidelity-sample-statement.pdf

Brokerage statement · 9 private details
Page 1Cover wins
Original
Page 1, original
CoverNo mistakes
Page 1, redacted by Cover
Gemma 31 left visible
Page 1, redacted by Gemma 3
Page 2Tie
Original
Page 2, original
CoverNo mistakes
Page 2, redacted by Cover
Gemma 3No mistakes
Page 2, redacted by Gemma 3
Page 3Cover wins
Original
Page 3, original
CoverNo mistakes
Page 3, redacted by Cover
Gemma 33 over-redacted
Page 3, redacted by Gemma 3
Page 4Tie
Original
Page 4, original
CoverNo mistakes
Page 4, redacted by Cover
Gemma 3No mistakes
Page 4, redacted by Gemma 3
Page 5Tie
Original
Page 5, original
CoverNo mistakes
Page 5, redacted by Cover
Gemma 3No mistakes
Page 5, redacted by Gemma 3
Page 6Tie
Original
Page 6, original
CoverNo mistakes
Page 6, redacted by Cover
Gemma 3No mistakes
Page 6, redacted by Gemma 3
Page 7Tie
Original
Page 7, original
CoverNo mistakes
Page 7, redacted by Cover
Gemma 3No mistakes
Page 7, redacted by Gemma 3
Page 8Tie
Original
Page 8, original
CoverNo mistakes
Page 8, redacted by Cover
Gemma 3No mistakes
Page 8, redacted by Gemma 3
Page 9Tie
Original
Page 9, original
CoverNo mistakes
Page 9, redacted by Cover
Gemma 3No mistakes
Page 9, redacted by Gemma 3
Page 10Tie
Original
Page 10, original
CoverNo mistakes
Page 10, redacted by Cover
Gemma 3No mistakes
Page 10, redacted by Gemma 3
Page 11Tie
Original
Page 11, original
CoverNo mistakes
Page 11, redacted by Cover
Gemma 3No mistakes
Page 11, redacted by Gemma 3
Page 12Cover wins
Original
Page 12, original
CoverNo mistakes
Page 12, redacted by Cover
Gemma 31 over-redacted
Page 12, redacted by Gemma 3
Page 13Tie
Original
Page 13, original
CoverNo mistakes
Page 13, redacted by Cover
Gemma 3No mistakes
Page 13, redacted by Gemma 3
Page 14Cover wins
Original
Page 14, original
CoverNo mistakes
Page 14, redacted by Cover
Gemma 31 over-redacted
Page 14, redacted by Gemma 3
Page 15Cover wins
Original
Page 15, original
Cover1 over-redacted
Page 15, redacted by Cover
Gemma 31 left visible4 over-redacted
Page 15, redacted by Gemma 3
Page 16Tie
Original
Page 16, original
CoverNo mistakes
Page 16, redacted by Cover
Gemma 3No mistakes
Page 16, redacted by Gemma 3
Page 17Tie
Original
Page 17, original
CoverNo mistakes
Page 17, redacted by Cover
Gemma 3No mistakes
Page 17, redacted by Gemma 3
Page 18Tie
Original
Page 18, original
CoverNo mistakes
Page 18, redacted by Cover
Gemma 3No mistakes
Page 18, redacted by Gemma 3
Page 19Tie
Original
Page 19, original
CoverNo mistakes
Page 19, redacted by Cover
Gemma 3No mistakes
Page 19, redacted by Gemma 3
Page 20Tie
Original
Page 20, original
CoverNo mistakes
Page 20, redacted by Cover
Gemma 3No mistakes
Page 20, redacted by Gemma 3
Page 21Tie
Original
Page 21, original
CoverNo mistakes
Page 21, redacted by Cover
Gemma 3No mistakes
Page 21, redacted by Gemma 3
Page 22Tie
Original
Page 22, original
CoverNo mistakes
Page 22, redacted by Cover
Gemma 3No mistakes
Page 22, redacted by Gemma 3
Page 23Tie
Original
Page 23, original
CoverNo mistakes
Page 23, redacted by Cover
Gemma 3No mistakes
Page 23, redacted by Gemma 3
Page 24Tie
Original
Page 24, original
CoverNo mistakes
Page 24, redacted by Cover
Gemma 3No mistakes
Page 24, redacted by Gemma 3
Page 25Tie
Original
Page 25, original
CoverNo mistakes
Page 25, redacted by Cover
Gemma 3No mistakes
Page 25, redacted by Gemma 3
Page 26Cover wins
Original
Page 26, original
CoverNo mistakes
Page 26, redacted by Cover
Gemma 319 over-redacted
Page 26, redacted by Gemma 3
Page 27Cover wins
Original
Page 27, original
CoverNo mistakes
Page 27, redacted by Cover
Gemma 310 over-redacted
Page 27, redacted by Gemma 3
Page 28Cover wins
Original
Page 28, original
Cover1 over-redacted
Page 28, redacted by Cover
Gemma 379 over-redacted
Page 28, redacted by Gemma 3

fidelity-large-print-sample-statement.pdf

Brokerage statement · 5 private details
Page 1Cover wins
Original
Page 1, original
CoverNo mistakes
Page 1, redacted by Cover
Gemma 31 left visible
Page 1, redacted by Gemma 3
Page 2Cover wins
Original
Page 2, original
CoverNo mistakes
Page 2, redacted by Cover
Gemma 34 over-redacted
Page 2, redacted by Gemma 3
Page 3Cover wins
Original
Page 3, original
CoverNo mistakes
Page 3, redacted by Cover
Gemma 31 left visible
Page 3, redacted by Gemma 3

pershing-2024-sample-tax-yearend.pdf

Brokerage statement · 7 private details
Page 1Tie
Original
Page 1, original
CoverNo mistakes
Page 1, redacted by Cover
Gemma 3No mistakes
Page 1, redacted by Gemma 3
Page 2Tie
Original
Page 2, original
CoverNo mistakes
Page 2, redacted by Cover
Gemma 3No mistakes
Page 2, redacted by Gemma 3
Page 3Cover wins
Original
Page 3, original
CoverNo mistakes
Page 3, redacted by Cover
Gemma 31 left visible2 over-redacted
Page 3, redacted by Gemma 3
Page 4Tie
Original
Page 4, original
CoverNo mistakes
Page 4, redacted by Cover
Gemma 3No mistakes
Page 4, redacted by Gemma 3
Page 5Cover wins
Original
Page 5, original
CoverNo mistakes
Page 5, redacted by Cover
Gemma 31 over-redacted
Page 5, redacted by Gemma 3
Page 6Cover wins
Original
Page 6, original
CoverNo mistakes
Page 6, redacted by Cover
Gemma 31 left visible3 over-redacted
Page 6, redacted by Gemma 3
Page 7Cover wins
Original
Page 7, original
CoverNo mistakes
Page 7, redacted by Cover
Gemma 33 left visible
Page 7, redacted by Gemma 3
Page 8Cover wins
Original
Page 8, original
CoverNo mistakes
Page 8, redacted by Cover
Gemma 32 left visible35 over-redacted
Page 8, redacted by Gemma 3

ubs-portfolio-summary-sample.pdf

Brokerage statement · 7 private details
Page 1Cover wins
Original
Page 1, original
CoverNo mistakes
Page 1, redacted by Cover
Gemma 39 over-redacted
Page 1, redacted by Gemma 3
Page 2Cover wins
Original
Page 2, original
CoverNo mistakes
Page 2, redacted by Cover
Gemma 32 over-redacted
Page 2, redacted by Gemma 3
Page 3Cover wins
Original
Page 3, original
CoverNo mistakes
Page 3, redacted by Cover
Gemma 33 over-redacted
Page 3, redacted by Gemma 3
Page 4Cover wins
Original
Page 4, original
CoverNo mistakes
Page 4, redacted by Cover
Gemma 312 over-redacted
Page 4, redacted by Gemma 3
Page 5Cover wins
Original
Page 5, original
CoverNo mistakes
Page 5, redacted by Cover
Gemma 31 left visible4 over-redacted
Page 5, redacted by Gemma 3
Page 6Cover wins
Original
Page 6, original
CoverNo mistakes
Page 6, redacted by Cover
Gemma 31 left visible6 over-redacted
Page 6, redacted by Gemma 3
Page 7Cover wins
Original
Page 7, original
CoverNo mistakes
Page 7, redacted by Cover
Gemma 38 over-redacted
Page 7, redacted by Gemma 3
Page 8Cover wins
Original
Page 8, original
CoverNo mistakes
Page 8, redacted by Cover
Gemma 31 left visible1 over-redacted
Page 8, redacted by Gemma 3
Page 9Cover wins
Original
Page 9, original
CoverNo mistakes
Page 9, redacted by Cover
Gemma 32 over-redacted
Page 9, redacted by Gemma 3
Page 10Cover wins
Original
Page 10, original
CoverNo mistakes
Page 10, redacted by Cover
Gemma 31 over-redacted
Page 10, redacted by Gemma 3
Page 11Cover wins
Original
Page 11, original
CoverNo mistakes
Page 11, redacted by Cover
Gemma 31 over-redacted
Page 11, redacted by Gemma 3
Page 12Cover wins
Original
Page 12, original
CoverNo mistakes
Page 12, redacted by Cover
Gemma 36 over-redacted
Page 12, redacted by Gemma 3
Page 13Cover wins
Original
Page 13, original
CoverNo mistakes
Page 13, redacted by Cover
Gemma 36 over-redacted
Page 13, redacted by Gemma 3
Page 14Cover wins
Original
Page 14, original
CoverNo mistakes
Page 14, redacted by Cover
Gemma 37 over-redacted
Page 14, redacted by Gemma 3
Page 15Cover wins
Original
Page 15, original
CoverNo mistakes
Page 15, redacted by Cover
Gemma 36 over-redacted
Page 15, redacted by Gemma 3
Page 16Cover wins
Original
Page 16, original
CoverNo mistakes
Page 16, redacted by Cover
Gemma 31 over-redacted
Page 16, redacted by Gemma 3
Page 17Cover wins
Original
Page 17, original
CoverNo mistakes
Page 17, redacted by Cover
Gemma 38 over-redacted
Page 17, redacted by Gemma 3
Page 18Cover wins
Original
Page 18, original
CoverNo mistakes
Page 18, redacted by Cover
Gemma 31 over-redacted
Page 18, redacted by Gemma 3
Page 19Cover wins
Original
Page 19, original
CoverNo mistakes
Page 19, redacted by Cover
Gemma 321 over-redacted
Page 19, redacted by Gemma 3
Page 20Cover wins
Original
Page 20, original
CoverNo mistakes
Page 20, redacted by Cover
Gemma 317 over-redacted
Page 20, redacted by Gemma 3
Page 21Cover wins
Original
Page 21, original
CoverNo mistakes
Page 21, redacted by Cover
Gemma 320 over-redacted
Page 21, redacted by Gemma 3
Page 22Tie
Original
Page 22, original
CoverNo mistakes
Page 22, redacted by Cover
Gemma 3No mistakes
Page 22, redacted by Gemma 3

cfpb-sample-credit-card-statement-handout.pdf

Credit-card statement · 2 private details
Page 1Tie
Original
Page 1, original
CoverNo mistakes
Page 1, redacted by Cover
Gemma 3No mistakes
Page 1, redacted by Gemma 3

allstate-auto-declarations-sample.pdf

Insurance declaration · 7 private details
Page 1Cover wins
Original
Page 1, original
CoverNo mistakes
Page 1, redacted by Cover
Gemma 33 left visible3 over-redacted
Page 1, redacted by Gemma 3
Page 2Cover wins
Original
Page 2, original
CoverNo mistakes
Page 2, redacted by Cover
Gemma 31 left visible1 over-redacted
Page 2, redacted by Gemma 3

cfpb-ficus-bank-loan-estimate-sample.pdf

Loan estimate · 9 private details
Page 1Tie
Original
Page 1, original
CoverNo mistakes
Page 1, redacted by Cover
Gemma 3No mistakes
Page 1, redacted by Gemma 3
Page 2Cover wins
Original
Page 2, original
CoverNo mistakes
Page 2, redacted by Cover
Gemma 31 over-redacted
Page 2, redacted by Gemma 3
Page 3Tie
Original
Page 3, original
CoverNo mistakes
Page 3, redacted by Cover
Gemma 3No mistakes
Page 3, redacted by Gemma 3

cigna-eob-example.pdf

Medical claim / EOB · 7 private details
Page 1Cover wins
Original
Page 1, original
CoverNo mistakes
Page 1, redacted by Cover
Gemma 346 over-redacted
Page 1, redacted by Gemma 3
Page 2Cover wins
Original
Page 2, original
CoverNo mistakes
Page 2, redacted by Cover
Gemma 33 left visible2 over-redacted
Page 2, redacted by Gemma 3

medicare-sample-part-b-msn.pdf

Medical claim / EOB · 6 private details
Page 1Cover wins
Original
Page 1, original
CoverNo mistakes
Page 1, redacted by Cover
Gemma 31 over-redacted
Page 1, redacted by Gemma 3
Page 2Cover wins
Original
Page 2, original
Cover4 over-redacted
Page 2, redacted by Cover
Gemma 31 left visible1 over-redacted
Page 2, redacted by Gemma 3
Page 3Cover wins
Original
Page 3, original
CoverNo mistakes
Page 3, redacted by Cover
Gemma 34 left visible
Page 3, redacted by Gemma 3
Page 4Cover wins
Original
Page 4, original
CoverNo mistakes
Page 4, redacted by Cover
Gemma 31 over-redacted
Page 4, redacted by Gemma 3
Page 5Tie
Original
Page 5, original
CoverNo mistakes
Page 5, redacted by Cover
Gemma 3No mistakes
Page 5, redacted by Gemma 3
Page 6Cover wins
Original
Page 6, original
CoverNo mistakes
Page 6, redacted by Cover
Gemma 31 over-redacted
Page 6, redacted by Gemma 3

cfpb-draft-periodic-mortgage-statement.pdf

Mortgage statement · 4 private details
Page 1Cover wins
Original
Page 1, original
CoverNo mistakes
Page 1, redacted by Cover
Gemma 31 left visible2 over-redacted
Page 1, redacted by Gemma 3

Lease_Agreement_signed.pdf

Other · 16 private details
Page 1Cover wins
Original
Page 1, original
CoverNo mistakes
Page 1, redacted by Cover
Gemma 37 left visible
Page 1, redacted by Gemma 3

transunion-cameo-credit-report-sample.pdf

Other · 23 private details
Page 1Cover wins
Original
Page 1, original
CoverNo mistakes
Page 1, redacted by Cover
Gemma 34 left visible8 over-redacted
Page 1, redacted by Gemma 3
Page 2Cover wins
Original
Page 2, original
CoverNo mistakes
Page 2, redacted by Cover
Gemma 31 left visible17 over-redacted
Page 2, redacted by Gemma 3
Page 3Tie
Original
Page 3, original
CoverNo mistakes
Page 3, redacted by Cover
Gemma 3No mistakes
Page 3, redacted by Gemma 3
Page 4Cover wins
Original
Page 4, original
CoverNo mistakes
Page 4, redacted by Cover
Gemma 33 over-redacted
Page 4, redacted by Gemma 3
Page 5Cover wins
Original
Page 5, original
CoverNo mistakes
Page 5, redacted by Cover
Gemma 31 left visible16 over-redacted
Page 5, redacted by Gemma 3
Page 6Cover wins
Original
Page 6, original
CoverNo mistakes
Page 6, redacted by Cover
Gemma 320 over-redacted
Page 6, redacted by Gemma 3
Page 7Cover wins
Original
Page 7, original
CoverNo mistakes
Page 7, redacted by Cover
Gemma 337 over-redacted
Page 7, redacted by Gemma 3
Page 8Cover wins
Original
Page 8, original
CoverNo mistakes
Page 8, redacted by Cover
Gemma 31 left visible8 over-redacted
Page 8, redacted by Gemma 3
Page 9Cover wins
Original
Page 9, original
CoverNo mistakes
Page 9, redacted by Cover
Gemma 32 over-redacted
Page 9, redacted by Gemma 3
Page 10Tie
Original
Page 10, original
CoverNo mistakes
Page 10, redacted by Cover
Gemma 3No mistakes
Page 10, redacted by Gemma 3

att-uverse-sample-bill.pdf

Utility bill · 13 private details
Page 1Cover wins
Original
Page 1, original
CoverNo mistakes
Page 1, redacted by Cover
Gemma 31 left visible
Page 1, redacted by Gemma 3
Page 2Tie
Original
Page 2, original
CoverNo mistakes
Page 2, redacted by Cover
Gemma 3No mistakes
Page 2, redacted by Gemma 3
Page 3Cover wins
Original
Page 3, original
CoverNo mistakes
Page 3, redacted by Cover
Gemma 31 left visible
Page 3, redacted by Gemma 3
Page 4Cover wins
Original
Page 4, original
CoverNo mistakes
Page 4, redacted by Cover
Gemma 31 left visible1 over-redacted
Page 4, redacted by Gemma 3

pge-sample-bill-sparky-joule.pdf

Utility bill · 3 private details
Page 1Gemma 3 wins
Original
Page 1, original
Cover1 over-redacted
Page 1, redacted by Cover
Gemma 3No mistakes
Page 1, redacted by Gemma 3
Page 2Cover wins
Original
Page 2, original
CoverNo mistakes
Page 2, redacted by Cover
Gemma 318 over-redacted
Page 2, redacted by Gemma 3
Page 3Tie
Original
Page 3, original
CoverNo mistakes
Page 3, redacted by Cover
Gemma 3No mistakes
Page 3, redacted by Gemma 3
Page 4Cover wins
Original
Page 4, original
CoverNo mistakes
Page 4, redacted by Cover
Gemma 32 over-redacted
Page 4, redacted by Gemma 3
Page 5Tie
Original
Page 5, original
CoverNo mistakes
Page 5, redacted by Cover
Gemma 3No mistakes
Page 5, redacted by Gemma 3
Page 6Cover wins
Original
Page 6, original
CoverNo mistakes
Page 6, redacted by Cover
Gemma 310 over-redacted
Page 6, redacted by Gemma 3

Full results

Cover v1.0.2 vs Google Gemma 3 gemma3:latest · 18 real-world sample documents.

metric · all 18 docsCoverGemma 3
Recall: PII correctly redacted100%69%
Leaks: confirmed PII left visible053
Over-redaction: non-PII redacted by mistake12623
Speed: median time per PDF~3.2 sec/PDF~24 sec/PDF
Per-document breakdown: all 18 docs (one outlier can dominate the totals)
documentCover
leaks		Gemma 3
leaks		Cover
over		Gemma 3
over
ubs-portfolio-summary-sample010142
Fidelity-sample-statement022117
transunion-cameo-credit-report-sample070111
2025_Tax_Return_Okafor-Brennan09097
cigna-eob-example03048
pershing-2024-sample-tax-yearend03041
pge-sample-bill-sparky-joule00130
commercebank-sample-statement01315
WellsFargo-sample-statement0524
allstate-auto-declarations-sample0304
fidelity-large-print-sample-statement0204
medicare-sample-part-b-msn0544
capitalone-sample-estatement0002
cfpb-draft-periodic-mortgage-statement0102
att-uverse-sample-bill0401
cfpb-ficus-bank-loan-estimate-sample0001
Lease_Agreement_signed0700
cfpb-sample-credit-card-statement-handout0000
TOTAL05312623

How the test works

Test set: 18 public sample documents (149 verified PII items), held fixed across runs so results stay comparable. Every engine redacted the same documents; each output is scored against a Claude-authored gold standard (confirmed + ambiguous tiers), and every counted over-redaction is kept auditable. Pages and the leaderboard use the same weighting: leaked private details weigh most, and over-redactions count too.