Open
Labels
interpreter-core(Objects, Python, Grammar, and Parser dirs)(Objects, Python, Grammar, and Parser dirs)performancePerformance or resource usagePerformance or resource usagetype-featureA feature request or enhancementA feature request or enhancement
Description
Feature or enhancement
asciilib was introduced in c3cec78 as a performance feature for ascii strings.
11 years have passed since then.
While working on #98025 we with @vstinner experimented on unicode_count with and without asciilib_count calls.
Results were clear: it does not bring any benefits on our benchmarks.
And this commit was made: df3a6d9
Later, while working on #98228 I've noticed that asciilib_rsplit_whitespace also does not provide significant performance gains on my platform and my simple data input.
So, maybe this should be analyzed deeply?
Pitch
- I think that these calls should be further investigated: do we really need this?
- We should come up with better data to make a final decision, including:
-
- What pytohn methods / c-api functions are affected?
-
- Short and long ascii / non-ascii strings (because this check slows down all non-ascii strings by calling extra
PyUnicode_IS_ASCII(str)
- Short and long ascii / non-ascii strings (because this check slows down all non-ascii strings by calling extra
-
- Windows / MacOS / Linux platforms, maybe the results will be different
I don't have access to Windows, but I can do the research for other platforms.
If it does not provide any performance benefits, it should be removed.
Metadata
Metadata
Assignees
Labels
interpreter-core(Objects, Python, Grammar, and Parser dirs)(Objects, Python, Grammar, and Parser dirs)performancePerformance or resource usagePerformance or resource usagetype-featureA feature request or enhancementA feature request or enhancement
Activity
mdboom commentedon Oct 13, 2022
I'd generally be in favor of removing almost-duplicate code if there is no performance benefit.
Have you done any analysis with the pyperformance benchmarks?
vstinner commentedon Oct 13, 2022
ASCII-only strings have multiple efficient properties. Examples:
Some optimizations are implemented outside stringlib. Like:
sobolevn commentedon Oct 13, 2022
Only very limited ones in #98025 (comment) and #98228 (comment)
serhiy-storchaka commentedon Oct 14, 2022
Please perform tests also with longer strings (100, 1000, 10000 characters or more). Also test with strings containing non-ASCII whitespace characters
\x85and\xa0.rhpvorderman commentedon Jan 24, 2023
If you want to work with some real-world ASCII data you can download a FASTQ file named SRR15096500_1.fastq.gz here https://www.ebi.ac.uk/ena/browser/view/SRR15096500.
FASTQ is a format to store DNA (or RNA) sequencing reads. You can find a description of the format here: https://en.wikipedia.org/wiki/FASTQ_format.
The data can be quite big. Whole genome sequencing can yield FASTQ files that are 100GB when compressed with gzip.
@marcelm has created a nice library called dnaio for parsing the format. To add to @vstinner ASCII has some really interesting other properties to (to complement @vstinner):
Anyway. I am curious to see if asciilib works well when using real-world data such as posted above. A lot of formats in my field are ASCII, so Python's performance in this area matters to me.
vstinner commentedon Apr 5, 2023
The utf8_decode() function of the Python "stringlib" internal library uses this property to check if a string is ASCII-only in an efficient way. It even works on size_t words (ex: 8 bytes on 64-bit systems) per loop iteration. See: Objects/stringlib/codecs.h.
rhpvorderman commentedon Apr 7, 2023
@vstinner I am aware that the current ASCII check exists and is quite efficient. However, it can be more efficient: #101289
The current ASCII check is per string and involves a lot of code. A quick check on the input buffer is almost cost-free. The copying can then proceed using only PyUnicode_New(127) and memcpy. That is much faster. Although it will require some reworking of the ASCII/UTF-8 code.