Repository navigation
Conversation
SwiftSyntax → native Rust: 15-project production benchmarkThe fixed CI candidate
Summed project medians: extraction 452.10 → 77.52 s (5.83×), full database creation 560.35 → 192.55 s (2.91×). These weighted totals are distinct from the equal-project geometric means. Largest project-median extractor RSS fell from 3,646 to 1,990 MiB; every project used less extractor RSS. Method: Apple M3 Max, two workers; two repetitions per backend/project, alternating backend order and reversing it for repetition two; two discarded warmups; warm source caches but a fresh database for every run. Production extractor timings include parsing, AST mapping/desugaring, TRAP compression and source archival; full creation additionally includes CodeQL/JVM orchestration, import and finalization. This compares parser backends, not just the later parser-internal optimizations. All 60 measured databases finalized and passed dataset/coverage checks, with zero diagnostics. All 18,026 selected Swift inputs were covered; 72,104 archived-input hash checks passed. These checks establish database health and input coverage, not independent AST equality. All pre-fix measurements were excluded and the full paired trial restarted. Limitations: shared workstation, only two observations per condition, substantial background-noise variance. Small projects were essentially unchanged end-to-end; JBird and DVIA were slightly slower on the native median. This is a directional result, not a precise universal speedup. The uncached, tuple-counting, QL-submodule-override DCA was started after CI passed: github/codeql-dca-main#40671. DCA results remain pending. |
Introduce the reusable parser and bottom-up syntax sink interfaces, with a temporary unimplemented parser. Wire the sink into yeast and expose explicit Cargo and Bazel backend selection while retaining SwiftSyntax as the default. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: c3453b85-5da4-4c2f-9e1e-87bb48fe348a
Add the final raw-syntax and trivia comparison harness, pinned corpus, reference fixtures, validity metadata, and coverage tooling before implementing the parser. Cache the oracle adapter schema and keep corpus selection, invalid-source handling, and failure limits explicit. Reference-only runs are usable with --generate-only --check-json. Enable the Cargo fixture comparison target in the implementation commit, so this infrastructure commit does not require the unimplemented native parser to pass fixtures. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: c3453b85-5da4-4c2f-9e1e-87bb48fe348a
Add generated syntax kinds, UTF-8 lexing, recursive-descent grammar, operator folding, recovery, and nesting limits. Emit directly into the syntax sink and reuse parser token storage between files. Enable the Cargo fixture comparison target and cover the initial grammar with 20 oracle-backed fixtures, including iterative else-if parsing. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: c3453b85-5da4-4c2f-9e1e-87bb48fe348a
Extend declarations, types, attributes, literals, patterns, macros, and contextual lookahead to match the pinned SwiftSyntax oracle. Preserve closure scopes after initialized bindings before enabling the native backend by default. Keep parser behavior and focused oracle fixtures together; all 85 grammar fixtures pass against live SwiftSyntax. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: c3453b85-5da4-4c2f-9e1e-87bb48fe348a
Match pound-directive names, Unicode operator continuations and identifier priority, period punctuation, and comment-adjacent operator fixity. Preserve nested-string, escape, binding, and operator boundary regressions in oracle-backed fixtures. Keep these correctness changes independent of allocation and dispatch optimizations. All 91 parser fixtures match live SwiftSyntax. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: c3453b85-5da4-4c2f-9e1e-87bb48fe348a
Add an in-memory parser benchmark with a fixed-memory checksum sink and a helper for direct extractor comparisons. Keep benchmark infrastructure separate from parser correctness and optimizations. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: c3453b85-5da4-4c2f-9e1e-87bb48fe348a
Recycle production child buffers, avoid expression stacks when no operator is present, and fast-path ASCII lexing and literal token checks. Cache the immutable standard operator table and clone it only for possible source-declared operators. Preserve the sink contract and syntax output. All 91 parser fixtures pass, and the focused benchmark retains the same node counts and checksum. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: c3453b85-5da4-4c2f-9e1e-87bb48fe348a
Select Rust by default in Cargo and Bazel while preserving explicit SwiftSyntax fallback and Rust precedence when both Cargo features are enabled. Forward pack-builder Cargo options and keep backend-selection guidance with the rollout. Retain mapped-AST corpus checks for both backends. Check SwiftSyntax-specific raw snapshots and regenerate shared references only with the SwiftSyntax backend. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: c3453b85-5da4-4c2f-9e1e-87bb48fe348a
f098d1b to
70d9130
Compare
Make native Swift parsing unconditional and remove backend features, Swift FFI bindings, and their Cargo and Bazel build infrastructure. Keep the reference parser as a standalone Swift executable and move the JSON-to-yeast adapter into differential-test infrastructure. Generate and check raw corpus snapshots with the native parser, preserving all mapped AST expectations. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: c3453b85-5da4-4c2f-9e1e-87bb48fe348a
5aecb33 to
d143478
Compare
This PR is a PoC that replaces the use of swift-syntax in the Unified extractor with a custom Swift parser implemented in Rust.
Idea: We have swift-syntax as a reference implementation/truth oracle for how a Swift parser should behave. We can validate correctness of a Rust implementation of a Swift parser by comparing it against swift-syntax on a large amount of Swift code. Then generating an implementation is simply a matter of asking an agent to keep working until all tests are green.
Benefits:
codeql database createis around 1.8x faster in average.This PR is just an experiment and definitely not ready for merging. It's very vibe coded and would need cleanup.