Competitive bake-off

Cover vs

Season 2, same 18 PDFs: Cover caught 100% of 149 private details, 0 leaked, 12 over-redacted, 3.2 s per PDF. Meta Llama 3.1 caught 67%, 59 leaked, 260 over-redacted, 24 s per PDF. Of 12 engines ranked by mistakes (leaked private details weigh most, and over-redactions count too), Meta Llama 3.1 places 7th and Cover 1st.

8.0B · llamaLast updated October 3, 2026 · engine v1.0.2

Sending PDFs with private identity information to a cloud Ai like Claude or ChatGPT is a bad idea. But could a local Ai on your own computer redact them instead? This bake-off runs Cover’s engine and the leading local Ai engines on the same 18 bank, tax and medical PDFs. So far the local Ai engines fall behind Cover in three ways: they over-redact, most are slow, and none offer a fast way to fix their mistakes.

Seasons
Season 3
Season 2 · 18 PDFs · 149 private details (PII)

PDF selection focuses on money, home and insurance documents. Real-world samples for anyone asking Ai about their own money, or accountants and tax pros who redact a client's documents before asking Ai for a second opinion. Watch Cover's engine improve with each release, even outpacing Google's tech.

  • Money 9
  • Home 5
  • Health & insurance 3
  • Taxes 1
Engine quality over time
CoverLlama 3.1
50%60%70%80%90%100%101001000% PII caughtover-redactions (log)Season 1 · 15 to 17 docsSeason 1 · 15 to 17 docsSeason 2 · 18 docsSeason 2 · 18 docs0.12.70.12.80.13.20.13.30.14.00.14.10.15.10.16.30.16.40.16.50.16.60.18.40.18.50.20.11.0.2v0.12.7: Llama 3.1 caught 59% (36 leaked)v0.12.8: Llama 3.1 caught 59% (36 leaked)v0.13.2: Llama 3.1 caught 59% (36 leaked)v0.13.3: Llama 3.1 caught 59% (36 leaked)v0.14.0: Llama 3.1 caught 59% (36 leaked)v0.14.1: Llama 3.1 caught 59% (36 leaked)v0.15.1: Llama 3.1 caught 59% (36 leaked)v0.16.3: Llama 3.1 caught 60% (47 leaked)v0.16.4: Llama 3.1 caught 60% (35 leaked)v0.16.5: Llama 3.1 caught 72% (48 leaked)v0.16.6: Llama 3.1 caught 72% (48 leaked)v0.18.4: Llama 3.1 caught 72% (48 leaked)v0.18.5: Llama 3.1 caught 72% (48 leaked)v0.20.1: Llama 3.1 caught 72% (48 leaked)v1.0.2: Llama 3.1 caught 67% (59 leaked)v0.12.7: Cover caught 75% of PII (21 leaked)v0.12.8: Cover caught 77% of PII (19 leaked)v0.13.2: Cover caught 79% of PII (18 leaked)v0.13.3: Cover caught 86% of PII (12 leaked)v0.14.0: Cover caught 86% of PII (12 leaked)v0.14.1: Cover caught 86% of PII (12 leaked)v0.15.1: Cover caught 91% of PII (5 leaked)v0.16.3: Cover caught 94% of PII (6 leaked)v0.16.4: Cover caught 95% of PII (2 leaked)v0.16.5: Cover caught 98% of PII (3 leaked)v0.16.6: Cover caught 98% of PII (3 leaked)v0.18.4: Cover caught 98% of PII (3 leaked)v0.18.5: Cover caught 98% of PII (3 leaked)v0.20.1: Cover caught 98% of PII (3 leaked)v1.0.2: Cover caught 100% of PII (0 leaked)59%67%75%100%v0.12.7: Llama 3.1 over-redacted 228 non-PII itemsv0.12.8: Llama 3.1 over-redacted 228 non-PII itemsv0.13.2: Llama 3.1 over-redacted 228 non-PII itemsv0.13.3: Llama 3.1 over-redacted 228 non-PII itemsv0.14.0: Llama 3.1 over-redacted 228 non-PII itemsv0.14.1: Llama 3.1 over-redacted 228 non-PII itemsv0.15.1: Llama 3.1 over-redacted 228 non-PII itemsv0.16.3: Llama 3.1 over-redacted 271 non-PII itemsv0.16.4: Llama 3.1 over-redacted 242 non-PII itemsv0.16.5: Llama 3.1 over-redacted 271 non-PII itemsv0.16.6: Llama 3.1 over-redacted 271 non-PII itemsv0.18.4: Llama 3.1 over-redacted 271 non-PII itemsv0.18.5: Llama 3.1 over-redacted 271 non-PII itemsv0.20.1: Llama 3.1 over-redacted 271 non-PII itemsv1.0.2: Llama 3.1 over-redacted 260 non-PII itemsv0.12.7: Cover over-redacted 15 non-PII itemsv0.12.8: Cover over-redacted 15 non-PII itemsv0.13.2: Cover over-redacted 51 non-PII itemsv0.13.3: Cover over-redacted 16 non-PII itemsv0.14.0: Cover over-redacted 16 non-PII itemsv0.14.1: Cover over-redacted 12 non-PII itemsv0.15.1: Cover over-redacted 22 non-PII itemsv0.16.3: Cover over-redacted 40 non-PII itemsv0.16.4: Cover over-redacted 36 non-PII itemsv0.16.5: Cover over-redacted 62 non-PII itemsv0.16.6: Cover over-redacted 26 non-PII itemsv0.18.4: Cover over-redacted 27 non-PII itemsv0.18.5: Cover over-redacted 22 non-PII itemsv0.20.1: Cover over-redacted 19 non-PII itemsv1.0.2: Cover over-redacted 12 non-PII items2282601512
Per-version detail: all 15 releases (recall · leaks · over-redaction · pages better / tied / worse)
versionseasondatePII caughtover-redactionpages better / tied / worsedocs
1.0.2S22026-10-03100% vs 67%12 vs 260n/a18
0.20.1S22026-09-0398% vs 72%19 vs 271n/a18
0.18.5S22026-08-2798% vs 72%22 vs 271n/a18
0.18.4S22026-08-2398% vs 72%27 vs 271n/a18
0.16.6S22026-07-1898% vs 72%26 vs 271n/a18
0.16.5S22026-07-1798% vs 72%62 vs 271n/a18
0.16.4S12026-07-1695% vs 60%36 vs 242n/a15
0.16.3S12026-07-1494% vs 60%40 vs 271n/a17
0.15.1S12026-07-0791% vs 59%22 vs 228n/a15
0.14.1S12026-06-2986% vs 59%12 vs 228n/a15
0.14.0S12026-06-2686% vs 59%16 vs 228n/a15
0.13.3S12026-06-1586% vs 59%16 vs 228n/a15
0.13.2S12026-06-1579% vs 59%51 vs 228n/a15
0.12.8S12026-06-0477% vs 59%15 vs 228n/a15
0.12.7S12026-06-0475% vs 59%15 vs 228n/a15

15 versions · 17 raw runs · de-duped latest-per-version

Local redaction landscape
↙ the goal013000100Over-redactionLeaksCoverGemma 4Phi-4Qwen 3Qwen 2.5DeepSeek R1Llama 3.1Apple IntelligenceGemma 3Privacy FilterRampartNemotron
#enginePII accuracyOver-redactionSpeed
1🥇 CoverExcellentVery goodExcellent
2🥈 Gemma 4GoogleVery goodVery poorGood
3🥉 Phi-4MicrosoftGoodVery poorPoor
4Qwen 3AlibabaGoodVery poorVery poor
5Qwen 2.5AlibabaPoorVery poorGood
6DeepSeek R1DeepSeekPoorVery poorVery poor
7Llama 3.1MetaPoorVery poorGood
8Apple IntelligenceAppleVery poorVery poorExcellent
9Gemma 3GooglePoorVery poorGood
10Privacy FilterOpenAIPoorVery poorVery good
11RampartAmerica.govVery poorVery poorExcellent
12NemotronOpenMedVery poorVery poorExcellent

Per-page side-by-sides

Check out the actual pages below and how each engine redacted them, side by side with the original.

Red outline: private detail left visibleBlue outline: over-redaction
7th place
Worth flagging
Meta Llama 3.1
PII accuracy: PoorOver-redaction: Very poorSpeed: Good ~24 sec/PDF

Catches 67% of the private details and over-redacts about nineteen times as much as Cover, so its pages come out both exposed and hard to read.

Cover’s edge
Holds even if a model matches Cover’s accuracy.
  • One-click edits fix any box instantly; no re-prompting (LLMs over-redact).
  • Fully offline files never leave your device.
  • Seconds per file no datacenter hardware to buy or maintain.

2025_Tax_Return_Okafor-Brennan.pdf

Tax return · 15 private details
Page 1Cover wins
Original
Page 1, original
CoverNo mistakes
Page 1, redacted by Cover
Llama 3.119 over-redacted
Page 1, redacted by Llama 3.1
Page 2Cover wins
Original
Page 2, original
CoverNo mistakes
Page 2, redacted by Cover
Llama 3.11 left visible40 over-redacted
Page 2, redacted by Llama 3.1
Page 3Cover wins
Original
Page 3, original
CoverNo mistakes
Page 3, redacted by Cover
Llama 3.11 over-redacted
Page 3, redacted by Llama 3.1
Page 4Cover wins
Original
Page 4, original
CoverNo mistakes
Page 4, redacted by Cover
Llama 3.11 left visible
Page 4, redacted by Llama 3.1
Page 5Tie
Original
Page 5, original
CoverNo mistakes
Page 5, redacted by Cover
Llama 3.1No mistakes
Page 5, redacted by Llama 3.1
Page 6Cover wins
Original
Page 6, original
CoverNo mistakes
Page 6, redacted by Cover
Llama 3.12 over-redacted
Page 6, redacted by Llama 3.1
Page 7Cover wins
Original
Page 7, original
CoverNo mistakes
Page 7, redacted by Cover
Llama 3.11 left visible
Page 7, redacted by Llama 3.1
Page 8Tie
Original
Page 8, original
CoverNo mistakes
Page 8, redacted by Cover
Llama 3.1No mistakes
Page 8, redacted by Llama 3.1

WellsFargo-sample-statement.pdf

Bank statement · 8 private details
Page 1Tie
Original
Page 1, original
CoverNo mistakes
Page 1, redacted by Cover
Llama 3.1No mistakes
Page 1, redacted by Llama 3.1
Page 2Llama 3.1 wins
Original
Page 2, original
Cover2 over-redacted
Page 2, redacted by Cover
Llama 3.11 over-redacted
Page 2, redacted by Llama 3.1

capitalone-sample-estatement.pdf

Bank statement · 4 private details
Page 1Cover wins
Original
Page 1, original
CoverNo mistakes
Page 1, redacted by Cover
Llama 3.13 left visible2 over-redacted
Page 1, redacted by Llama 3.1

commercebank-sample-statement.pdf

Bank statement · 4 private details
Page 1Cover wins
Original
Page 1, original
Cover3 over-redacted
Page 1, redacted by Cover
Llama 3.12 left visible3 over-redacted
Page 1, redacted by Llama 3.1

Fidelity-sample-statement.pdf

Brokerage statement · 9 private details
Page 1Cover wins
Original
Page 1, original
CoverNo mistakes
Page 1, redacted by Cover
Llama 3.11 left visible
Page 1, redacted by Llama 3.1
Page 2Tie
Original
Page 2, original
CoverNo mistakes
Page 2, redacted by Cover
Llama 3.1No mistakes
Page 2, redacted by Llama 3.1
Page 3Cover wins
Original
Page 3, original
CoverNo mistakes
Page 3, redacted by Cover
Llama 3.12 over-redacted
Page 3, redacted by Llama 3.1
Page 4Tie
Original
Page 4, original
CoverNo mistakes
Page 4, redacted by Cover
Llama 3.1No mistakes
Page 4, redacted by Llama 3.1
Page 5Tie
Original
Page 5, original
CoverNo mistakes
Page 5, redacted by Cover
Llama 3.1No mistakes
Page 5, redacted by Llama 3.1
Page 6Tie
Original
Page 6, original
CoverNo mistakes
Page 6, redacted by Cover
Llama 3.1No mistakes
Page 6, redacted by Llama 3.1
Page 7Tie
Original
Page 7, original
CoverNo mistakes
Page 7, redacted by Cover
Llama 3.1No mistakes
Page 7, redacted by Llama 3.1
Page 8Tie
Original
Page 8, original
CoverNo mistakes
Page 8, redacted by Cover
Llama 3.1No mistakes
Page 8, redacted by Llama 3.1
Page 9Tie
Original
Page 9, original
CoverNo mistakes
Page 9, redacted by Cover
Llama 3.1No mistakes
Page 9, redacted by Llama 3.1
Page 10Cover wins
Original
Page 10, original
CoverNo mistakes
Page 10, redacted by Cover
Llama 3.11 left visible
Page 10, redacted by Llama 3.1
Page 11Tie
Original
Page 11, original
CoverNo mistakes
Page 11, redacted by Cover
Llama 3.1No mistakes
Page 11, redacted by Llama 3.1
Page 12Tie
Original
Page 12, original
CoverNo mistakes
Page 12, redacted by Cover
Llama 3.1No mistakes
Page 12, redacted by Llama 3.1
Page 13Tie
Original
Page 13, original
CoverNo mistakes
Page 13, redacted by Cover
Llama 3.1No mistakes
Page 13, redacted by Llama 3.1
Page 14Tie
Original
Page 14, original
CoverNo mistakes
Page 14, redacted by Cover
Llama 3.1No mistakes
Page 14, redacted by Llama 3.1
Page 15Cover wins
Original
Page 15, original
Cover1 over-redacted
Page 15, redacted by Cover
Llama 3.14 left visible4 over-redacted
Page 15, redacted by Llama 3.1
Page 16Tie
Original
Page 16, original
CoverNo mistakes
Page 16, redacted by Cover
Llama 3.1No mistakes
Page 16, redacted by Llama 3.1
Page 17Tie
Original
Page 17, original
CoverNo mistakes
Page 17, redacted by Cover
Llama 3.1No mistakes
Page 17, redacted by Llama 3.1
Page 18Tie
Original
Page 18, original
CoverNo mistakes
Page 18, redacted by Cover
Llama 3.1No mistakes
Page 18, redacted by Llama 3.1
Page 19Cover wins
Original
Page 19, original
CoverNo mistakes
Page 19, redacted by Cover
Llama 3.12 over-redacted
Page 19, redacted by Llama 3.1
Page 20Tie
Original
Page 20, original
CoverNo mistakes
Page 20, redacted by Cover
Llama 3.1No mistakes
Page 20, redacted by Llama 3.1
Page 21Tie
Original
Page 21, original
CoverNo mistakes
Page 21, redacted by Cover
Llama 3.1No mistakes
Page 21, redacted by Llama 3.1
Page 22Tie
Original
Page 22, original
CoverNo mistakes
Page 22, redacted by Cover
Llama 3.1No mistakes
Page 22, redacted by Llama 3.1
Page 23Tie
Original
Page 23, original
CoverNo mistakes
Page 23, redacted by Cover
Llama 3.1No mistakes
Page 23, redacted by Llama 3.1
Page 24Tie
Original
Page 24, original
CoverNo mistakes
Page 24, redacted by Cover
Llama 3.1No mistakes
Page 24, redacted by Llama 3.1
Page 25Tie
Original
Page 25, original
CoverNo mistakes
Page 25, redacted by Cover
Llama 3.1No mistakes
Page 25, redacted by Llama 3.1
Page 27Cover wins
Original
Page 27, original
CoverNo mistakes
Page 27, redacted by Cover
Llama 3.16 over-redacted
Page 27, redacted by Llama 3.1
Page 28Llama 3.1 wins
Original
Page 28, original
Cover1 over-redacted
Page 28, redacted by Cover
Llama 3.1No mistakes
Page 28, redacted by Llama 3.1

fidelity-large-print-sample-statement.pdf

Brokerage statement · 5 private details
Page 1Tie
Original
Page 1, original
CoverNo mistakes
Page 1, redacted by Cover
Llama 3.1No mistakes
Page 1, redacted by Llama 3.1
Page 3Tie
Original
Page 3, original
CoverNo mistakes
Page 3, redacted by Cover
Llama 3.1No mistakes
Page 3, redacted by Llama 3.1

pershing-2024-sample-tax-yearend.pdf

Brokerage statement · 7 private details
Page 1Tie
Original
Page 1, original
CoverNo mistakes
Page 1, redacted by Cover
Llama 3.1No mistakes
Page 1, redacted by Llama 3.1
Page 2Tie
Original
Page 2, original
CoverNo mistakes
Page 2, redacted by Cover
Llama 3.1No mistakes
Page 2, redacted by Llama 3.1
Page 3Cover wins
Original
Page 3, original
CoverNo mistakes
Page 3, redacted by Cover
Llama 3.12 over-redacted
Page 3, redacted by Llama 3.1
Page 4Cover wins
Original
Page 4, original
CoverNo mistakes
Page 4, redacted by Cover
Llama 3.11 over-redacted
Page 4, redacted by Llama 3.1
Page 5Tie
Original
Page 5, original
CoverNo mistakes
Page 5, redacted by Cover
Llama 3.1No mistakes
Page 5, redacted by Llama 3.1
Page 6Cover wins
Original
Page 6, original
CoverNo mistakes
Page 6, redacted by Cover
Llama 3.15 over-redacted
Page 6, redacted by Llama 3.1
Page 7Cover wins
Original
Page 7, original
CoverNo mistakes
Page 7, redacted by Cover
Llama 3.16 over-redacted
Page 7, redacted by Llama 3.1
Page 8Cover wins
Original
Page 8, original
CoverNo mistakes
Page 8, redacted by Cover
Llama 3.14 over-redacted
Page 8, redacted by Llama 3.1

ubs-portfolio-summary-sample.pdf

Brokerage statement · 7 private details
Page 1Cover wins
Original
Page 1, original
CoverNo mistakes
Page 1, redacted by Cover
Llama 3.14 over-redacted
Page 1, redacted by Llama 3.1
Page 2Cover wins
Original
Page 2, original
CoverNo mistakes
Page 2, redacted by Cover
Llama 3.15 over-redacted
Page 2, redacted by Llama 3.1
Page 3Cover wins
Original
Page 3, original
CoverNo mistakes
Page 3, redacted by Cover
Llama 3.11 left visible
Page 3, redacted by Llama 3.1
Page 4Tie
Original
Page 4, original
CoverNo mistakes
Page 4, redacted by Cover
Llama 3.1No mistakes
Page 4, redacted by Llama 3.1
Page 5Cover wins
Original
Page 5, original
CoverNo mistakes
Page 5, redacted by Cover
Llama 3.12 left visible
Page 5, redacted by Llama 3.1
Page 6Cover wins
Original
Page 6, original
CoverNo mistakes
Page 6, redacted by Cover
Llama 3.11 left visible
Page 6, redacted by Llama 3.1
Page 7Cover wins
Original
Page 7, original
CoverNo mistakes
Page 7, redacted by Cover
Llama 3.13 left visible2 over-redacted
Page 7, redacted by Llama 3.1
Page 8Cover wins
Original
Page 8, original
CoverNo mistakes
Page 8, redacted by Cover
Llama 3.11 left visible3 over-redacted
Page 8, redacted by Llama 3.1
Page 9Cover wins
Original
Page 9, original
CoverNo mistakes
Page 9, redacted by Cover
Llama 3.11 over-redacted
Page 9, redacted by Llama 3.1
Page 10Cover wins
Original
Page 10, original
CoverNo mistakes
Page 10, redacted by Cover
Llama 3.11 left visible3 over-redacted
Page 10, redacted by Llama 3.1
Page 11Tie
Original
Page 11, original
CoverNo mistakes
Page 11, redacted by Cover
Llama 3.1No mistakes
Page 11, redacted by Llama 3.1
Page 12Cover wins
Original
Page 12, original
CoverNo mistakes
Page 12, redacted by Cover
Llama 3.14 over-redacted
Page 12, redacted by Llama 3.1
Page 13Cover wins
Original
Page 13, original
CoverNo mistakes
Page 13, redacted by Cover
Llama 3.11 over-redacted
Page 13, redacted by Llama 3.1
Page 14Cover wins
Original
Page 14, original
CoverNo mistakes
Page 14, redacted by Cover
Llama 3.11 left visible3 over-redacted
Page 14, redacted by Llama 3.1
Page 15Cover wins
Original
Page 15, original
CoverNo mistakes
Page 15, redacted by Cover
Llama 3.11 over-redacted
Page 15, redacted by Llama 3.1
Page 16Cover wins
Original
Page 16, original
CoverNo mistakes
Page 16, redacted by Cover
Llama 3.11 left visible3 over-redacted
Page 16, redacted by Llama 3.1
Page 17Tie
Original
Page 17, original
CoverNo mistakes
Page 17, redacted by Cover
Llama 3.1No mistakes
Page 17, redacted by Llama 3.1
Page 18Cover wins
Original
Page 18, original
CoverNo mistakes
Page 18, redacted by Cover
Llama 3.11 left visible3 over-redacted
Page 18, redacted by Llama 3.1
Page 19Tie
Original
Page 19, original
CoverNo mistakes
Page 19, redacted by Cover
Llama 3.1No mistakes
Page 19, redacted by Llama 3.1
Page 20Cover wins
Original
Page 20, original
CoverNo mistakes
Page 20, redacted by Cover
Llama 3.11 left visible3 over-redacted
Page 20, redacted by Llama 3.1
Page 21Cover wins
Original
Page 21, original
CoverNo mistakes
Page 21, redacted by Cover
Llama 3.11 over-redacted
Page 21, redacted by Llama 3.1
Page 22Cover wins
Original
Page 22, original
CoverNo mistakes
Page 22, redacted by Cover
Llama 3.15 over-redacted
Page 22, redacted by Llama 3.1
Page 23Cover wins
Original
Page 23, original
CoverNo mistakes
Page 23, redacted by Cover
Llama 3.132 over-redacted
Page 23, redacted by Llama 3.1

cfpb-sample-credit-card-statement-handout.pdf

Credit-card statement · 2 private details
Page 1Tie
Original
Page 1, original
CoverNo mistakes
Page 1, redacted by Cover
Llama 3.1No mistakes
Page 1, redacted by Llama 3.1

allstate-auto-declarations-sample.pdf

Insurance declaration · 7 private details
Page 1Tie
Original
Page 1, original
CoverNo mistakes
Page 1, redacted by Cover
Llama 3.1No mistakes
Page 1, redacted by Llama 3.1
Page 2Cover wins
Original
Page 2, original
CoverNo mistakes
Page 2, redacted by Cover
Llama 3.12 over-redacted
Page 2, redacted by Llama 3.1

cfpb-ficus-bank-loan-estimate-sample.pdf

Loan estimate · 9 private details
Page 1Tie
Original
Page 1, original
CoverNo mistakes
Page 1, redacted by Cover
Llama 3.1No mistakes
Page 1, redacted by Llama 3.1
Page 2Cover wins
Original
Page 2, original
CoverNo mistakes
Page 2, redacted by Cover
Llama 3.11 left visible6 over-redacted
Page 2, redacted by Llama 3.1
Page 3Cover wins
Original
Page 3, original
CoverNo mistakes
Page 3, redacted by Cover
Llama 3.11 left visible
Page 3, redacted by Llama 3.1

cigna-eob-example.pdf

Medical claim / EOB · 7 private details
Page 1Cover wins
Original
Page 1, original
CoverNo mistakes
Page 1, redacted by Cover
Llama 3.128 over-redacted
Page 1, redacted by Llama 3.1
Page 2Tie
Original
Page 2, original
CoverNo mistakes
Page 2, redacted by Cover
Llama 3.1No mistakes
Page 2, redacted by Llama 3.1

medicare-sample-part-b-msn.pdf

Medical claim / EOB · 6 private details
Page 2Cover wins
Original
Page 2, original
Cover4 over-redacted
Page 2, redacted by Cover
Llama 3.12 left visible1 over-redacted
Page 2, redacted by Llama 3.1
Page 3Cover wins
Original
Page 3, original
CoverNo mistakes
Page 3, redacted by Cover
Llama 3.11 left visible1 over-redacted
Page 3, redacted by Llama 3.1
Page 4Cover wins
Original
Page 4, original
CoverNo mistakes
Page 4, redacted by Cover
Llama 3.18 over-redacted
Page 4, redacted by Llama 3.1
Page 5Tie
Original
Page 5, original
CoverNo mistakes
Page 5, redacted by Cover
Llama 3.1No mistakes
Page 5, redacted by Llama 3.1
Page 6Cover wins
Original
Page 6, original
CoverNo mistakes
Page 6, redacted by Cover
Llama 3.14 over-redacted
Page 6, redacted by Llama 3.1

cfpb-draft-periodic-mortgage-statement.pdf

Mortgage statement · 4 private details
Page 1Cover wins
Original
Page 1, original
CoverNo mistakes
Page 1, redacted by Cover
Llama 3.11 left visible1 over-redacted
Page 1, redacted by Llama 3.1

Lease_Agreement_signed.pdf

Other · 16 private details
Page 1Cover wins
Original
Page 1, original
CoverNo mistakes
Page 1, redacted by Cover
Llama 3.111 left visible
Page 1, redacted by Llama 3.1

transunion-cameo-credit-report-sample.pdf

Other · 23 private details
Page 1Cover wins
Original
Page 1, original
CoverNo mistakes
Page 1, redacted by Cover
Llama 3.113 left visible
Page 1, redacted by Llama 3.1
Page 2Cover wins
Original
Page 2, original
CoverNo mistakes
Page 2, redacted by Cover
Llama 3.11 left visible2 over-redacted
Page 2, redacted by Llama 3.1
Page 3Cover wins
Original
Page 3, original
CoverNo mistakes
Page 3, redacted by Cover
Llama 3.13 left visible
Page 3, redacted by Llama 3.1
Page 4Tie
Original
Page 4, original
CoverNo mistakes
Page 4, redacted by Cover
Llama 3.1No mistakes
Page 4, redacted by Llama 3.1
Page 5Cover wins
Original
Page 5, original
CoverNo mistakes
Page 5, redacted by Cover
Llama 3.111 left visible
Page 5, redacted by Llama 3.1
Page 6Cover wins
Original
Page 6, original
CoverNo mistakes
Page 6, redacted by Cover
Llama 3.16 over-redacted
Page 6, redacted by Llama 3.1
Page 7Cover wins
Original
Page 7, original
CoverNo mistakes
Page 7, redacted by Cover
Llama 3.117 over-redacted
Page 7, redacted by Llama 3.1
Page 8Cover wins
Original
Page 8, original
CoverNo mistakes
Page 8, redacted by Cover
Llama 3.11 left visible
Page 8, redacted by Llama 3.1
Page 9Cover wins
Original
Page 9, original
CoverNo mistakes
Page 9, redacted by Cover
Llama 3.110 over-redacted
Page 9, redacted by Llama 3.1
Page 10Cover wins
Original
Page 10, original
CoverNo mistakes
Page 10, redacted by Cover
Llama 3.13 left visible
Page 10, redacted by Llama 3.1

att-uverse-sample-bill.pdf

Utility bill · 13 private details
Page 1Tie
Original
Page 1, original
CoverNo mistakes
Page 1, redacted by Cover
Llama 3.1No mistakes
Page 1, redacted by Llama 3.1
Page 2Tie
Original
Page 2, original
CoverNo mistakes
Page 2, redacted by Cover
Llama 3.1No mistakes
Page 2, redacted by Llama 3.1
Page 3Cover wins
Original
Page 3, original
CoverNo mistakes
Page 3, redacted by Cover
Llama 3.17 left visible
Page 3, redacted by Llama 3.1
Page 4Cover wins
Original
Page 4, original
CoverNo mistakes
Page 4, redacted by Cover
Llama 3.12 left visible
Page 4, redacted by Llama 3.1

pge-sample-bill-sparky-joule.pdf

Utility bill · 3 private details
Page 1Llama 3.1 wins
Original
Page 1, original
Cover1 over-redacted
Page 1, redacted by Cover
Llama 3.1No mistakes
Page 1, redacted by Llama 3.1
Page 2Tie
Original
Page 2, original
CoverNo mistakes
Page 2, redacted by Cover
Llama 3.1No mistakes
Page 2, redacted by Llama 3.1
Page 3Tie
Original
Page 3, original
CoverNo mistakes
Page 3, redacted by Cover
Llama 3.1No mistakes
Page 3, redacted by Llama 3.1
Page 4Tie
Original
Page 4, original
CoverNo mistakes
Page 4, redacted by Cover
Llama 3.1No mistakes
Page 4, redacted by Llama 3.1
Page 5Tie
Original
Page 5, original
CoverNo mistakes
Page 5, redacted by Cover
Llama 3.1No mistakes
Page 5, redacted by Llama 3.1
Page 6Tie
Original
Page 6, original
CoverNo mistakes
Page 6, redacted by Cover
Llama 3.1No mistakes
Page 6, redacted by Llama 3.1

Full results

Cover v1.0.2 vs Meta Llama 3.1 llama3.1:latest · 18 real-world sample documents.

metric · all 18 docsCoverLlama 3.1
Recall: PII correctly redacted100%67%
Leaks: confirmed PII left visible059
Over-redaction: non-PII redacted by mistake12260
Speed: median time per PDF~3.2 sec/PDF~24 sec/PDF
Per-document breakdown: all 18 docs (one outlier can dominate the totals)
documentCover
leaks		Llama 3.1
leaks		Cover
over		Llama 3.1
over
ubs-portfolio-summary-sample06074
2025_Tax_Return_Okafor-Brennan02062
transunion-cameo-credit-report-sample015035
cigna-eob-example00028
pershing-2024-sample-tax-yearend00018
Fidelity-sample-statement05214
medicare-sample-part-b-msn03414
cfpb-ficus-bank-loan-estimate-sample0106
commercebank-sample-statement0233
allstate-auto-declarations-sample0002
capitalone-sample-estatement0402
WellsFargo-sample-statement0021
cfpb-draft-periodic-mortgage-statement0101
Lease_Agreement_signed01100
att-uverse-sample-bill0900
cfpb-sample-credit-card-statement-handout0000
fidelity-large-print-sample-statement0000
pge-sample-bill-sparky-joule0010
TOTAL05912260

How the test works

Test set: 18 public sample documents (149 verified PII items), held fixed across runs so results stay comparable. Every engine redacted the same documents; each output is scored against a Claude-authored gold standard (confirmed + ambiguous tiers), and every counted over-redaction is kept auditable. Pages and the leaderboard use the same weighting: leaked private details weigh most, and over-redactions count too.