Repository navigation
chore(dev): run local dev, integration and e2e on DocumentDB - #3375
Conversation
…of MongoDB Local development and CI now use open-source DocumentDB (Postgres plus a Mongo wire-protocol gateway, MIT) in place of MongoDB 5.0 (SSPL, EOL). The API still talks to it through Mongoose, so this is the first step towards dropping MongoDB from what we ship, tested daily by every developer and CI run before users see it. HDX-5709. - One image, built from source in docker/documentdb and shared by the dev, integration and e2e stacks; it handles root-owned volumes, interrupted first boots and clean shutdown, and pins the upstream commit. - e2e database helpers use the Node driver instead of mongosh, which the image doesn't ship; CI builds the image with the GHA cache and waits on its healthcheck. - Test teardown retries DocumentDB's intermittent dropDatabase failure. - Dev data moves to .volumes/docdb_dev_<slot>; old MongoDB volumes are not migrated.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
🟡 Tier 3 — StandardIntroduces new logic, modifies core functionality, or touches areas with non-trivial risk. Why this tier:
Additional context: touches background tasks or the delivery pipeline lightly (10 lines, under the 30-line bar for Tier 4) Review process: Full human review — logic, architecture, edge cases. Stats
|
Duplicate code analysisNo new duplicate code introduced by this PR. jscpd compared this branch against |
PR Review
2 finding(s): 🔴 0 P0/P1 · 🟡 2 P2 · 🔵 0 P3 None could be anchored to changed lines; they are listed below. Findings outside the changed lines
Priority comes from a separate grading pass over the finished review. P3 is folded here instead of posted inline. No finding blocks merge automatically; a maintainer decides. |
E2E Test Results✅ All tests passed • 469 passed • 1 skipped • 1079s
Tests ran across 4 shards in parallel. |
[Medium impact] The PR appears safe to merge; no new actionable issues were found. SummaryMoves development, integration tests, and E2E tests to a source-built DocumentDB image. Shipped deployment images remain unchanged.
|
Deep Review✅ No critical issues found. This is a careful, well-commented infrastructure migration (MongoDB → source-built DocumentDB) scoped to local dev, integration, and e2e. The API, shipped images, and the user-facing 🟡 P2 -- recommended
🔵 P3 nitpicks (3)
Reviewers (8): correctness, reliability, adversarial, maintainability, project-standards, testing, kieran-typescript, previous-comments. Testing gaps:
Dropped during re-grade: three reported No finding blocks merge automatically. A maintainer will expect P0/P1 findings in code this PR changes to be fixed; P2/P3 are your call -- fix or reply. Never fix findings about surrounding code here; reply instead. Do not widen the PR. How to respond |
…s start errors CI e2e setup failed on every shard: `buildx bake` on the e2e compose file resolves its out-of-tree build context as a filesystem entitlement and refuses to run. Build docker/documentdb directly with the same GHA cache. A failed Postgres start now prints the tail of postgresql.log to the container logs, so `docker compose logs db` shows the cause.
|
The integration job's `up --wait` failed before any test ran: the OTel collector's healthcheck needs an OpAMP server the job doesn't run, so it never reports healthy. Start every service as before, then wait on db.
0006 records DocumentDB 1.0 replacing FerretDB, following ClickHouse/ClickStack#3375. 0001 keeps its text and gains a status line pointing at it, and the index marks it superseded.
Integration shards were rebuilding DocumentDB from source and timing out. The bake shared one GHA cache scope between the collector and db images, and a scope holds a single manifest, so each export overwrote the other. db now uses the documentdb scope the e2e job already fills.
…cumentdb-local-dev
- Build DocumentDB v1.2-0 (pinned to its commit) instead of v1.0-RC2. - Publish DocumentDB on loopback only in the dev, integration and e2e compose files; the credentials are well known. - Name the shared DocumentDB credentials and TLS options in the e2e helpers and the alert seed script, and type the drop retry's error. - Keep the entrypoint's TERM trap from aborting before Postgres stops. - Update the e2e README and the AGENTS.md data-flow line for DocumentDB.
| docker buildx create --use --driver=docker-container | ||
| docker buildx bake -f ./docker-compose.ci.yml --set *.cache-to="type=gha,mode=max,scope=ci-integration" --set *.cache-from="type=gha,scope=ci-integration" --load | ||
| # One cache scope per image (a scope holds one manifest); db reuses the e2e job's documentdb scope. | ||
| docker buildx bake -f ./docker-compose.ci.yml \ |
There was a problem hiding this comment.
🟡 P2 — Building the DocumentDB image from source on a cache miss probably won't fit the 16/15-minute CI job timeouts
The Dockerfile compiles the C extensions serially (make install-documentdb, no -j) and runs a release cargo build of the gateway. Any change to docker/documentdb/**, or the GHA cache evicting the large documentdb scope (mode=max keeps the Rust and C builder layers), makes all 4 integration shards (timeout-minutes: 16) and all 4 e2e shards (.github/workflows/e2e-tests.yml, timeout-minutes: 15) cold-build at once before tests start. Fix: build the image once in a dedicated upstream job (or push it to GHCR) that the shards needs:/pull, or raise the timeouts to allow for a cold build.
Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond
| 'common-utils', | ||
| 'clickhouse', | ||
| 'mongodb', | ||
| 'documentdb', |
There was a problem hiding this comment.
This might be another portal running which picked up on this. This should be formatted properly, but the portal will use whatever worktree process is currently occupying the port.
| - Docker Compose manages ClickHouse, MongoDB, OTel Collector | ||
| - Docker Compose manages ClickHouse, DocumentDB, OTel Collector | ||
| - The metadata store is [DocumentDB](https://github.com/documentdb/documentdb) | ||
| (Postgres plus a Mongo wire-protocol gateway), built from |
There was a problem hiding this comment.
I was trying to connect via postgres directly as well, but we dont expose the port. do you think its worth exposing or later problem?
There was a problem hiding this comment.
Hmm, I'd say that we probably don't need the PG port exposed for now. Added a docker exec command to development.md if we need to jump into PG to debug anything.
| ### Default: Full-Stack Mode | ||
|
|
||
| By default, `make e2e` runs tests in **full-stack mode** with MongoDB + API + | ||
| By default, `make e2e` runs tests in **full-stack mode** with DocumentDB + API + |
There was a problem hiding this comment.
Should update (they describe the local stack)
| File | Line | Current text | Change |
|---|---|---|---|
CONTRIBUTING.md |
240 | "reset ClickHouse and MongoDB storage" | Say DocumentDB, and mention .volumes/docdb_dev_<slot> |
CONTRIBUTING.md |
260 | "E2E tests run against a full local stack (MongoDB + ClickHouse + API)" | DocumentDB |
CONTRIBUTING.md |
201–202 | Lists collector + ClickHouse in Docker, leaves out the db | Add DocumentDB, and optionally link to the Compass connection string in development.md |
agent_docs/evals.md |
31, 78 | "Docker containers (ClickHouse, MongoDB)" | DocumentDB |
agent_docs/evals.md |
64 | Diagram label Mongo:30498 |
Fine as-is (the port variable is still HDX_DEV_MONGO_PORT), or rename to DocDB:30498 |
agent_docs/evals.md |
323 | "default ports (8123, 27017)" | Still accurate: the compose default is still 27017, now mapped to gateway port 10260. No change needed |
agent_docs/evals.md |
329 | "When MongoDB is recreated…" | DocumentDB |
packages/hdx-eval/README.md |
16, 60 | "ClickHouse, MongoDB" | DocumentDB |
.claude/skills/playwright/SKILL.md |
81 | "@full-stack — tests requiring MongoDB + API" |
DocumentDB |
.claude/agents/playwright-test-generator.md |
61 | Same | DocumentDB |
.claude/agents/playwright-test-planner.md |
66 | "the test runner clears MongoDB before each run" | DocumentDB |
brandon-pereira
left a comment
There was a problem hiding this comment.
Overall works great - couple minor nits for doc improvements (maybe a port exposed) but lets ship it!
Contributing, eval and Playwright docs still said the local dev, e2e and eval stacks run MongoDB. Also document a psql one-liner for reaching the Postgres under DocumentDB.
| docker buildx create --use --driver=docker-container | ||
| docker buildx bake -f ./docker-compose.ci.yml --set *.cache-to="type=gha,mode=max,scope=ci-integration" --set *.cache-from="type=gha,scope=ci-integration" --load | ||
| # One cache scope per image (a scope holds one manifest); db reuses the e2e job's documentdb scope. | ||
| docker buildx bake -f ./docker-compose.ci.yml \ |
There was a problem hiding this comment.
🟡 P2 — CI compiles DocumentDB from source on a cold cache inside 15–16 minute job limits
On a cache miss (first run of this PR, eviction from the 10 GB GHA cache, or any change under docker/documentdb), each of the 4 integration shards (timeout-minutes: 16) and the 4 e2e shards (timeout-minutes: 15 in .github/workflows/e2e-tests.yml) builds the Dockerfile in parallel. That means libbson, pcre2, the Intel decimal library, the C extensions and the Rust gateway (cargo build --release), and the tests still have to run inside the same budget. Build the image once, either in a single upstream job that the shards needs: or by pushing it to a registry pinned by digest, and have the shards only pull it.
Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond
| @bash scripts/ensure-dev-portal.sh | ||
| docker compose -p $(HDX_CI_PROJECT) -f ./docker-compose.ci.yml up -d | ||
| docker compose -p $(HDX_CI_PROJECT) -f ./docker-compose.ci.yml up -d --build | ||
| docker compose -p $(HDX_CI_PROJECT) -f ./docker-compose.ci.yml up -d --wait --wait-timeout 300 db |
There was a problem hiding this comment.
🟡 P2 — A failed DocumentDB health wait in dev-int skips the down -v cleanup
The new up -d --wait ... db runs as its own recipe line, so when DocumentDB never becomes healthy, make stops there. The teardown on the following line never runs, and ClickHouse, the collector and db stay up on the slot. The same applies to dev-int-common-utils and ci-int. Fold the wait into the same shell line as the test run so down -v always runs, e.g. docker compose ... up -d --wait ... db && bash -c '...'; ret=$$?; docker compose ... down -v; ....
Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

Summary
Local development, integration tests and e2e tests now run against open-source DocumentDB (Postgres extensions plus a Mongo wire-protocol gateway, MIT) instead of MongoDB 5.0 (SSPL, end of life). The API is unchanged and still uses Mongoose. Running DocumentDB everywhere internally first surfaces compatibility problems before anything we ship to users changes.
What changed
yarn dev,make dev-int,make dev-e2eand the CI integration and e2e jobs all use one DocumentDB image built fromdocker/documentdb. It is 176 MB, against about 700 MB formongo:5.0.32-focal.docker exec mongosh, because the image doesn't shipmongosh.dropDatabasefailure (codes 112 and 1, while index builds are running).Key decisions
HDX_*_MONGO_PORT) and thedbservice name are unchanged, to limit churn across scripts, slots and the dev portal.Impact
.volumes/docdb_dev_<slot>. Existing local MongoDB data isn't migrated, so the dev stack starts empty after this lands.yarn devbuilds the image, which takes a few minutes. It is cached after that.mongosh, use the connection string inagent_docs/development.md. Everything works except Compass's Performance tab.docker-compose.ymlare unchanged.Implementation detail
tar, serialmake, and a one-line patch to an upstream symbol check that miscounts musl's_init/_fini.lost+founddocker stopmake dev-intpasses 2273/2273.mainwith MongoDB: 380 passed, 9 failed, 28 flaky.dropDatabaserace: fails in 3 to 5% of drops under a stress test, the same on PG17, PG18 and the Alpine build. HyperDX never drops databases outside tests.How to reproduce
Starting point: local
yarn dev(needs the API and the metadata database).On
main:yarn dev.docker psshows thedbcontainer runningmongo:5.0.32-focal.On this branch:
yarn dev. The first run buildshyperdx/documentdb:local.docker psshows thedbcontainer runninghyperdx/documentdb:localashealthy, and the startup banner lists DocumentDB.make dev-int. All suites pass.How to test on Vercel preview
N/A, non-UI change.
References