Skip to content

Unified: Replace swift-syntax with Swift parser in Rust - #23140

Draft
paldepind wants to merge 9 commits into
mainfrom
unified/swift-parser-port
Draft

paldepind wants to merge 9 commits into
mainfrom
unified/swift-parser-port

Conversation

@paldepind

@paldepind paldepind commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

This PR is a PoC that replaces the use of swift-syntax in the Unified extractor with a custom Swift parser implemented in Rust.

Idea: We have swift-syntax as a reference implementation/truth oracle for how a Swift parser should behave. We can validate correctness of a Rust implementation of a Swift parser by comparing it against swift-syntax on a large amount of Swift code. Then generating an implementation is simply a matter of asking an agent to keep working until all tests are green.

Benefits:

  • Building the Unified extractor does not require building any Swift! We don't need to build Swift neither through Bazel nor Cargo and we don't need the FFI thing where we link Rust with Swift. We only need to build a separate Swift projects when testing the Rust Swift parser.
  • This makes the Unified+Swift extractor 5 times faster. Probably because the whole JSON bridge is avoided. A full codeql database create is around 1.8x faster in average.

This PR is just an experiment and definitely not ready for merging. It's very vibe coded and would need cleanup.

@paldepind

Copy link
Copy Markdown
Contributor Author

SwiftSyntax → native Rust: 15-project production benchmark

The fixed CI candidate f098d1b gives 5.00× faster actual extraction and 1.80× faster complete database creation, geometric means across 15 pinned projects from the Unified DCA nightly suite.

Project Extractor speedup Full DB speedup
frzi/swiftui-router 2.72× 1.07×
djyde/WebShell 5.55× 1.06×
vsanthanam/JBird 3.30× 0.98×
prateek147/DVIA-v2 3.55× 1.00×
Alamofire/Alamofire 3.97× 1.24×
dheerajghub/Dovve 5.40× 1.48×
AndrewBennet/ReadingListV1 4.17× 1.79×
utmapp/UTM 6.13× 1.70×
krzysztofzablocki/Sourcery 7.47× 2.13×
JohanDegraeve/xdripswift 7.37× 2.76×
mozilla-mobile/firefox-ios 6.54× 2.64×
soto-project/soto 9.09× 4.76×
ProtonMail/ios-mail 4.16× 1.92×
kickstarter/ios-oss 5.84× 2.98×
manaflow-ai/cmux 4.01× 2.53×

Summed project medians: extraction 452.10 → 77.52 s (5.83×), full database creation 560.35 → 192.55 s (2.91×). These weighted totals are distinct from the equal-project geometric means. Largest project-median extractor RSS fell from 3,646 to 1,990 MiB; every project used less extractor RSS.

Method: Apple M3 Max, two workers; two repetitions per backend/project, alternating backend order and reversing it for repetition two; two discarded warmups; warm source caches but a fresh database for every run. Production extractor timings include parsing, AST mapping/desugaring, TRAP compression and source archival; full creation additionally includes CodeQL/JVM orchestration, import and finalization. This compares parser backends, not just the later parser-internal optimizations.

All 60 measured databases finalized and passed dataset/coverage checks, with zero diagnostics. All 18,026 selected Swift inputs were covered; 72,104 archived-input hash checks passed. These checks establish database health and input coverage, not independent AST equality. All pre-fix measurements were excluded and the full paired trial restarted.

Limitations: shared workstation, only two observations per condition, substantial background-noise variance. Small projects were essentially unchanged end-to-end; JBird and DVIA were slightly slower on the native median. This is a directional result, not a precise universal speedup.

The uncached, tuple-counting, QL-submodule-override DCA was started after CI passed: github/codeql-dca-main#40671. DCA results remain pending.

@paldepind paldepind changed the title Unified: Replace SwiftSyntax with the native Rust Swift parser Unified: Replace swift-syntax with Rust Swift parser Nov 16, 2026
paldepind and others added 8 commits November 17, 2026 09:16
Introduce the reusable parser and bottom-up syntax sink interfaces, with a temporary unimplemented parser. Wire the sink into yeast and expose explicit Cargo and Bazel backend selection while retaining SwiftSyntax as the default.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: c3453b85-5da4-4c2f-9e1e-87bb48fe348a
Add the final raw-syntax and trivia comparison harness, pinned corpus, reference fixtures, validity metadata, and coverage tooling before implementing the parser. Cache the oracle adapter schema and keep corpus selection, invalid-source handling, and failure limits explicit.

Reference-only runs are usable with --generate-only --check-json. Enable the Cargo fixture comparison target in the implementation commit, so this infrastructure commit does not require the unimplemented native parser to pass fixtures.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: c3453b85-5da4-4c2f-9e1e-87bb48fe348a
Add generated syntax kinds, UTF-8 lexing, recursive-descent grammar, operator folding, recovery, and nesting limits. Emit directly into the syntax sink and reuse parser token storage between files.

Enable the Cargo fixture comparison target and cover the initial grammar with 20 oracle-backed fixtures, including iterative else-if parsing.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: c3453b85-5da4-4c2f-9e1e-87bb48fe348a
Extend declarations, types, attributes, literals, patterns, macros, and contextual lookahead to match the pinned SwiftSyntax oracle. Preserve closure scopes after initialized bindings before enabling the native backend by default.

Keep parser behavior and focused oracle fixtures together; all 85 grammar fixtures pass against live SwiftSyntax.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: c3453b85-5da4-4c2f-9e1e-87bb48fe348a
Match pound-directive names, Unicode operator continuations and identifier priority, period punctuation, and comment-adjacent operator fixity. Preserve nested-string, escape, binding, and operator boundary regressions in oracle-backed fixtures.

Keep these correctness changes independent of allocation and dispatch optimizations. All 91 parser fixtures match live SwiftSyntax.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: c3453b85-5da4-4c2f-9e1e-87bb48fe348a
Add an in-memory parser benchmark with a fixed-memory checksum sink and a helper for direct extractor comparisons. Keep benchmark infrastructure separate from parser correctness and optimizations.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: c3453b85-5da4-4c2f-9e1e-87bb48fe348a
Recycle production child buffers, avoid expression stacks when no operator is present, and fast-path ASCII lexing and literal token checks. Cache the immutable standard operator table and clone it only for possible source-declared operators.

Preserve the sink contract and syntax output. All 91 parser fixtures pass, and the focused benchmark retains the same node counts and checksum.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: c3453b85-5da4-4c2f-9e1e-87bb48fe348a
Select Rust by default in Cargo and Bazel while preserving explicit SwiftSyntax fallback and Rust precedence when both Cargo features are enabled. Forward pack-builder Cargo options and keep backend-selection guidance with the rollout.

Retain mapped-AST corpus checks for both backends. Check SwiftSyntax-specific raw snapshots and regenerate shared references only with the SwiftSyntax backend.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: c3453b85-5da4-4c2f-9e1e-87bb48fe348a
@paldepind
paldepind force-pushed the unified/swift-parser-port branch from f098d1b to 70d9130 Compare November 17, 2026 08:23
Make native Swift parsing unconditional and remove backend features,
Swift FFI bindings, and their Cargo and Bazel build infrastructure.

Keep the reference parser as a standalone Swift executable and move the
JSON-to-yeast adapter into differential-test infrastructure. Generate and
check raw corpus snapshots with the native parser, preserving all mapped
AST expectations.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: c3453b85-5da4-4c2f-9e1e-87bb48fe348a
@paldepind
paldepind force-pushed the unified/swift-parser-port branch from 5aecb33 to d143478 Compare November 17, 2026 09:23
@paldepind paldepind changed the title Unified: Replace swift-syntax with Rust Swift parser Unified: Replace swift-syntax with Swift parser in Rust Nov 17, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant