Skip to content

Switch from codepoint-based character classification to Grapheme Cluster-based classification when a character >= 0x100 is encountered #37

Open

Description

@sttk

Overview

The current implementation performs character classification at the Codepoint level. However, for strings containing Unicode combining characters, Codepoint-level processing alone cannot always correctly distinguish non-ASCII letters and digits from other characters.

For example, in e◌̈, the e itself is an ASCII character, while e◌̈ as a whole forms a single Grapheme Cluster.

Therefore, simply classifying each Codepoint independently may incorrectly classify e as an ASCII alphabetic character even though it is part of a non-ASCII grapheme.

Proposed Change

For strings consisting of characters in the 0x00–0xff range, use a fast path that performs character classification and conversion directly.

Process the string from the beginning, handling characters in the 0x00–0xff range using this fast path.

When a character in the 0x100 or higher range is encountered, switch from Codepoint-based processing to a Unicode-aware mode that converts the entire string into Grapheme Clusters starting from the beginning.

This is necessary because a character in the 0x100 or higher range may form the same Grapheme Cluster as a preceding character in the 0x00–0xff range, as in e◌̈.

Optimization of Reprocessing

Even when a character in the 0x100 or higher range is detected, it is not necessary to re-run the classification logic for the entire string.

After converting the string into Grapheme Clusters, the results already obtained by the fast path can be reused for the consecutive 0x00–0xff range at the beginning of the string.

The processing flow is therefore:

  1. Classify and convert characters in the 0x00–0xff range using the fast path.
  2. When a character in the 0x100 or higher range is encountered, switch to Unicode-aware processing.
  3. Convert the entire string into Grapheme Clusters.
  4. Reuse the classification and conversion results already produced by the fast path for the consecutive 0x00–0xff range at the beginning of the string.
  5. Process the remaining characters using Grapheme Cluster-based classification and conversion.

This preserves the performance of the fast path for ASCII/Latin-1-range input while providing correct classification for strings containing Unicode combining characters.

Activity

  1. sttk commented on Sep 20, 2026

    @sttk
    OwnerAuthor

    If a character that combines with and modifies the preceding character—primarily Unicode General Categories Mn, Mc, and Me—appears immediately after an ASCII alphanumeric character, we may be able to remove both the ASCII alphanumeric character and its following combining characters. This could eliminate the need to introduce a library that carries a large Unicode character table.

    To determine exactly which characters should be handled, we could examine an existing library that implements grapheme segmentation and extract the relevant Unicode properties or character-code ranges that it uses to identify characters that continue the same grapheme cluster after a base character. We could then create a smaller, application-specific character-code table containing only those entries and use it for the detection logic.

    If this resulting partial character-code table is still too large, we could further reduce it by removing ranges that are clearly intended for base characters other than ASCII alphanumeric characters, preferably removing entire ranges where possible. This would allow us to keep the runtime table as small as reasonably possible while retaining the behavior needed for our specific use case.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions