Skip to content

improvement(pii): mask texts with many matches in linear time and retry failed chunks three times - #10151

Merged
waleedlatif1 merged 1 commit into
stagingfrom
improvement/pii-large-text-speed
Nov 18, 2026
Merged

waleedlatif1 merged 1 commit into
stagingfrom
improvement/pii-large-text-speed

Conversation

@waleedlatif1

Copy link
Copy Markdown
Collaborator

Summary

  • Bump presidio-analyzer / presidio-anonymizer 2.2.362 → 2.2.366. Before 2.2.365, the analyzer's remove_duplicates and the anonymizer's overlap resolution compared every match with every other one, so a large text with thousands of matches took minutes. 2.2.365 replaced both with sorted sweeps upstream, so there's nothing to patch on our side.
  • Mask client: 3 attempts per chunk instead of 8, one budget for every retryable failure (the separate 500 budget is gone). A retried chunk re-runs its whole NER pass, and a chunk that timed out on its own input fails the same way each time, so 8 attempts multiplied the work and delayed the fail-safe.
  • Log chars= on redact_batch so a slow request can be tied to its size.

Measurements (real docker/pii.Dockerfile image, 2 CPUs, same corpus through both versions)

shape 2.2.362 2.2.366
2,000 × 120-char batch 2.85s 2.92s
prose 1M chars 20.1s 11.2s
CSV 200k chars (21k matches) 108.5s 8.3s
pretty JSON 300k chars 27.4s 8.0s
minified JSON 300k chars 29.6s 4.6s
HTML 300k chars 21.8s 2.9s
one text, 15,000 emails 79.8s 1.3s

Output across the corpus: no character the old version masked is left unmasked (0 positions lost). Differences are all more masking: IPv6 addresses are masked whole (the old version left the last group visible), a UUID is masked as one <UUID> under detect-all instead of three mislabeled pieces, and valid Polish PESEL numbers are detected. Where several recognizers match the same span, the chosen label can differ (<US_SSN> vs <US_BANK_NUMBER>), with the same characters masked.

Type of Change

  • Improvement

Testing

  • New e2e check in test-pii-redaction-e2e.ts: one text with 15,000 emails must be masked exactly within 30s. The same request takes 79.8s on 2.2.362 (red) and 1.3s on 2.2.366 (green); the old cost grows quadratically (1.4s / 8s / 34s / 80s at 2k / 5k / 10k / 15k).
  • Full test-pii-redaction-e2e.ts against a local app and the 2.2.366 Presidio image: see comment below.
  • mask-client.test.ts retry-budget case updated to 3 attempts; biome on changed files.

Checklist

  • Code follows project style guidelines
  • Self-reviewed my changes
  • Tests added/updated and passing (new tests pass the test-audit authoring gate)
  • No new warnings introduced
  • I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

…ry failed chunks three times

Presidio before 2.2.365 compared every match with every other one when removing duplicates and resolving overlaps, so a large text with thousands of matches took minutes. 2.2.366 replaces both with sorted sweeps. The mask client now makes three attempts per chunk instead of eight, since a retried chunk re-runs its whole NER pass.
@vercel

vercel Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated
docs Skipped Skipped Nov 18, 2026 5:19am UTC

Request Review

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 5 files

Reply with feedback, questions, or to request a fix.

Turn on auto-fix | Re-trigger cubic

Comment thread apps/sim/lib/guardrails/mask-client.ts Resolved
@greptile-apps

greptile-apps Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 5/5

[Medium impact] The PR appears safe to merge; no actionable issues were found.

Summary

Updates Presidio to speed up masking texts with many matches and limits each failed chunk to three attempts.

  • Presidio masks texts with many matches much faster.
  • Failed masking chunks stop retrying after three attempts.
  • Batch masking logs include the total input character count.

No actionable issues found.

Reviews (1) · Last reviewed commit: "improvement(pii): mask texts with many m..." · Reviewed by Greptile

@waleedlatif1
waleedlatif1 merged commit d712f94 into staging Nov 18, 2026
81 checks passed
@waleedlatif1
waleedlatif1 deleted the improvement/pii-large-text-speed branch November 18, 2026 05:58

This branch was previously deployed

1 inactive deployment
Preview — 5bc7312d Deployed Nov 18, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant