Repository navigation
improvement(pii): mask texts with many matches in linear time and retry failed chunks three times - #10151
Merged
Conversation
…ry failed chunks three times Presidio before 2.2.365 compared every match with every other one when removing duplicates and resolving overlaps, so a large text with thousands of matches took minutes. 2.2.366 replaces both with sorted sweeps. The mask client now makes three attempts per chunk instead of eight, since a retried chunk re-runs its whole NER pass.
|
The latest updates on your projects. Learn more about Vercel for GitHub. |
Contributor
There was a problem hiding this comment.
All reported issues were addressed across 5 files
Reply with feedback, questions, or to request a fix.
Turn on auto-fix | Re-trigger cubic
Contributor
[Medium impact] The PR appears safe to merge; no actionable issues were found. SummaryUpdates Presidio to speed up masking texts with many matches and limits each failed chunk to three attempts.
No actionable issues found. Reviews (1) · Last reviewed commit: "improvement(pii): mask texts with many m..." · Reviewed by Greptile |
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
presidio-analyzer/presidio-anonymizer2.2.362 → 2.2.366. Before 2.2.365, the analyzer'sremove_duplicatesand the anonymizer's overlap resolution compared every match with every other one, so a large text with thousands of matches took minutes. 2.2.365 replaced both with sorted sweeps upstream, so there's nothing to patch on our side.chars=onredact_batchso a slow request can be tied to its size.Measurements (real
docker/pii.Dockerfileimage, 2 CPUs, same corpus through both versions)Output across the corpus: no character the old version masked is left unmasked (0 positions lost). Differences are all more masking: IPv6 addresses are masked whole (the old version left the last group visible), a UUID is masked as one
<UUID>under detect-all instead of three mislabeled pieces, and valid Polish PESEL numbers are detected. Where several recognizers match the same span, the chosen label can differ (<US_SSN>vs<US_BANK_NUMBER>), with the same characters masked.Type of Change
Testing
test-pii-redaction-e2e.ts: one text with 15,000 emails must be masked exactly within 30s. The same request takes 79.8s on 2.2.362 (red) and 1.3s on 2.2.366 (green); the old cost grows quadratically (1.4s / 8s / 34s / 80s at 2k / 5k / 10k / 15k).test-pii-redaction-e2e.tsagainst a local app and the 2.2.366 Presidio image: see comment below.mask-client.test.tsretry-budget case updated to 3 attempts; biome on changed files.Checklist
test-auditauthoring gate)