Skip to content

chore(dev): run local dev, integration and e2e on DocumentDB - #3375

Merged
jordan-simonovski merged 10 commits into
mainfrom
jordansimonovski/documentdb-local-dev
Nov 19, 2026
Merged

jordan-simonovski merged 10 commits into
mainfrom
jordansimonovski/documentdb-local-dev

Conversation

@jordan-simonovski

@jordan-simonovski jordan-simonovski commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Local development, integration tests and e2e tests now run against open-source DocumentDB (Postgres extensions plus a Mongo wire-protocol gateway, MIT) instead of MongoDB 5.0 (SSPL, end of life). The API is unchanged and still uses Mongoose. Running DocumentDB everywhere internally first surfaces compatibility problems before anything we ship to users changes.

What changed

  • yarn dev, make dev-int, make dev-e2e and the CI integration and e2e jobs all use one DocumentDB image built from docker/documentdb. It is 176 MB, against about 700 MB for mongo:5.0.32-focal.
  • The e2e database helpers use the Node driver instead of docker exec mongosh, because the image doesn't ship mongosh.
  • CI builds the image with the GHA cache and waits on its healthcheck before tests start.
  • Integration teardown retries DocumentDB's intermittent dropDatabase failure (codes 112 and 1, while index builds are running).

Key decisions

  • The image is built from source at DocumentDB v1.2-0, pinned to its commit, because upstream only ships Ubuntu and EL9 packages. Alpine 3.24 matches the all-in-one image's base, so the same build can be reused there later.
  • The env var names (HDX_*_MONGO_PORT) and the db service name are unchanged, to limit churn across scripts, slots and the dev portal.

Impact

  • Dev data moves to .volumes/docdb_dev_<slot>. Existing local MongoDB data isn't migrated, so the dev stack starts empty after this lands.
  • The first yarn dev builds the image, which takes a few minutes. It is cached after that.
  • To browse the dev database with Compass or mongosh, use the connection string in agent_docs/development.md. Everything works except Compass's Performance tab.
  • Shipped images and the user-facing docker-compose.yml are unchanged.
Implementation detail
  • Image. Postgres 18 with the DocumentDB extensions and gateway. It copies only what those link against: no GDAL, JIT, or PostGIS raster or topology.
  • Alpine build fixes: GNU tar, serial make, and a one-line patch to an upstream symbol check that miscounts musl's _init/_fini.
  • Entrypoint behaviour:
    • it fixes ownership on root-owned volumes, and works on volumes containing lost+found
    • it re-initialises after an interrupted first boot
    • it stops Postgres cleanly on docker stop
    • it exits non-zero if the gateway crashes
  • Healthcheck: probes both the gateway port and Postgres.
  • Testing:
    • make dev-int passes 2273/2273.
    • Full e2e on DocumentDB: 368 passed, 13 failed, 32 flaky. The same suite on main with MongoDB: 380 passed, 9 failed, 28 flaky.
    • On both databases, the failures are Next.js dev-mode page compiles (18 to 30 s) outlasting test timeouts, not database errors. The API logged no database errors.
    • Every test that failed only on DocumentDB passed or was flaky when re-run.
  • API compatibility: checked separately against DocumentDB, including a restore of real MongoDB 4.0.5 data with all 19 indexes (HDX-5709).
  • dropDatabase race: fails in 3 to 5% of drops under a stress test, the same on PG17, PG18 and the Alpine build. HyperDX never drops databases outside tests.

How to reproduce

Starting point: local yarn dev (needs the API and the metadata database).

On main:

  1. Run yarn dev.
  2. docker ps shows the db container running mongo:5.0.32-focal.

On this branch:

  1. Run yarn dev. The first run builds hyperdx/documentdb:local.
  2. docker ps shows the db container running hyperdx/documentdb:local as healthy, and the startup banner lists DocumentDB.
  3. Open the app, register an account, then create a saved search, a dashboard and an alert. All of them persist across a reload.
  4. Run make dev-int. All suites pass.

How to test on Vercel preview

N/A, non-UI change.

References

  • Linear Issue: HDX-5709 (parent HDX-5707)
  • Related PRs:

…of MongoDB

Local development and CI now use open-source DocumentDB (Postgres plus a
Mongo wire-protocol gateway, MIT) in place of MongoDB 5.0 (SSPL, EOL). The
API still talks to it through Mongoose, so this is the first step towards
dropping MongoDB from what we ship, tested daily by every developer and CI
run before users see it. HDX-5709.

- One image, built from source in docker/documentdb and shared by the dev,
  integration and e2e stacks; it handles root-owned volumes, interrupted
  first boots and clean shutdown, and pins the upstream commit.
- e2e database helpers use the Node driver instead of mongosh, which the
  image doesn't ship; CI builds the image with the GHA cache and waits on
  its healthcheck.
- Test teardown retries DocumentDB's intermittent dropDatabase failure.
- Dev data moves to .volumes/docdb_dev_<slot>; old MongoDB volumes are not
  migrated.
@vercel

vercel Bot commented Oct 15, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
hyperdx-oss Ready Ready Preview Nov 19, 2026 10:32pm UTC
hyperdx-storybook Ready Ready Preview Nov 19, 2026 10:32pm UTC

Request Review

@github-actions github-actions Bot added the review/tier-3 Standard — full human review required label Oct 15, 2026
@github-actions

github-actions Bot commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

🟡 Tier 3 — Standard

Introduces new logic, modifies core functionality, or touches areas with non-trivial risk.

Why this tier:

  • Diff size: 282 production lines changed (Tier 2 max: < 250)
  • Cross-layer change: touches frontend (packages/app) + backend (packages/api)

Additional context: touches background tasks or the delivery pipeline lightly (10 lines, under the 30-line bar for Tier 4)

Review process: Full human review — logic, architecture, edge cases.
SLA: First-pass feedback within 1 business day.

Stats
  • Production files changed: 16
  • Production lines changed: 282 (+ 347 in test files, excluded from tier calculation)
  • Branch: jordansimonovski/documentdb-local-dev
  • Author: jordan-simonovski

To override this classification, remove the review/tier-3 label and apply a different review/tier-* label. Manual overrides are preserved on subsequent pushes.

@github-actions

github-actions Bot commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

Duplicate code analysis

No new duplicate code introduced by this PR.


jscpd compared this branch against origin/main
across app, api and common-utils. Only new clones are listed; the codebase
currently holds 108 in total (1.13%).
Run yarn dupes locally to reproduce, or yarn dupes:all for the full picture.

@github-actions

github-actions Bot commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

PR Review

If you are a coding agent acting for the author, read this first.
Nothing below blocks merge automatically; a maintainer decides. What they will expect:

  • 🔴 P0/P1 in code this PR adds or changes: fix before asking for review.
  • 🟡 P2 in code this PR changes: your call. Fix if small, otherwise reply.
  • 🔵 P3 is collapsed at the bottom: nobody needs to act on it before merge.
  • Anything about surrounding code, or asking you to widen the change (hoist a helper,
    dedupe with another file, fix other call sites): never fix here, whatever the
    priority. Reply with one sentence; if it is P0, say so plainly so a human sees it.

One commit per review round. After two rounds, stop and ask a maintainer to review
scope rather than addressing more automated findings. Full rule: AGENTS.md.

2 finding(s): 🔴 0 P0/P1 · 🟡 2 P2 · 🔵 0 P3

None could be anchored to changed lines; they are listed below.

Findings outside the changed lines

  • 🟡 P2 docker/documentdb/entrypoint.sh:506 — createUser JSON is built with printf, so the user and password are not escaped → The username and password are pasted straight into a JSON string, with no escaping. A DOCDB_PASSWORD containing " or \ breaks first boot. A crafted value can also add its own roles. The PR says this image is meant to be reused in the all-in-one image, where these values would be user-configurable. Fix: pass the user and password as separate psql variables and build the document in SQL, e.g. json_build_object('createUser', :'user', 'pwd', :'pwd', 'roles', json_build_array(...))::text. Or escape the values with jq -n --arg.
  • 🟡 P2 packages/app/tests/e2e/utils/db-helpers.ts:149 — The DocumentDB credentials and TLS options are now hand-copied in about 10 places → The hyperdx:hyperdx credentials and the tls=true&tlsAllowInvalidCertificates=true&authMechanism=SCRAM-SHA-256 options appear in three compose files and three .env.* files. They also appear in .github/actions/e2e-setup/action.yml, packages/api/scripts/seed-alerts.ts and here. Only code comments keep them in sync, so changing the password or auth mechanism breaks whichever copy is missed. Fix: keep one shared copy of the auth and options strings for the scripts, and drop the MONGO_URI export from e2e-setup. packages/api/.env.e2e already builds the same URI from HDX_E2E_MONGO_PORT.

Priority comes from a separate grading pass over the finished review. P3 is folded here instead of posted inline. No finding blocks merge automatically; a maintainer decides.

@github-actions

github-actions Bot commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

E2E Test Results

✅ All tests passed • 469 passed • 1 skipped • 1079s

Status Count
✅ Passed 469
❌ Failed 0
⚠️ Flaky 13
⏭️ Skipped 1

Tests ran across 4 shards in parallel.

View full report →

@greptile-apps

greptile-apps Bot commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 5/5

[Medium impact] The PR appears safe to merge; no new actionable issues were found.

Summary

Moves development, integration tests, and E2E tests to a source-built DocumentDB image. Shipped deployment images remain unchanged.

  • Local development runs on a shared DocumentDB image.
  • Integration tests now start and wait for DocumentDB.
  • Full-stack E2E tests use DocumentDB through the Node driver.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  Dev[Local development] --> Gateway[DocumentDB gateway]
  Int[Integration tests] --> Gateway
  E2E[E2E API and database helpers] --> Gateway
  Gateway --> PG[(Postgres with DocumentDB extensions)]
Loading

Reviews (4) · Last reviewed commit: "Merge branch 'main' into jordansimonovsk..." · Reviewed by Greptile

Comment thread docker/documentdb/entrypoint.sh Resolved
@github-actions

github-actions Bot commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

Deep Review

✅ No critical issues found. This is a careful, well-commented infrastructure migration (MongoDB → source-built DocumentDB) scoped to local dev, integration, and e2e. The API, shipped images, and the user-facing docker-compose.yml are untouched. Verification against the diff confirms the high-risk paths hold up: withRetry extracts the numeric error.code (1/112) and the redirect/4xx guards leave those codes retryable; the removed runMongoshScript/mongosh helpers have no dangling references; the e2e DB port binds moved to 127.0.0.1 (a hardening over the old all-interfaces MongoDB bind); and the --wait healthcheck gating correctly waits on db only where the collector healthcheck can't pass. The recommendations below are non-blocking.

🟡 P2 -- recommended

  • packages/app/tests/e2e/utils/db-helpers.ts:35 -- the new transient-drop retry and Node-driver helpers (dropE2eDatabase, withE2eDb, setTeamFlag) and the fixtures.ts withRetry wrapper on dropDatabase are new behavior exercised only in the full-stack e2e job, so the code-coercion and transient-retry branches have no lightweight coverage and a regression surfaces only when the DocumentDB stack is up.
    • Fix: Add a focused test that drives dropE2eDatabase/withRetry through a mocked MongoServerError with codes 1, 112, and a non-transient code to lock the retry-vs-rethrow boundaries.
    • testing, kieran-typescript
🔵 P3 nitpicks (3)
  • docker/documentdb/entrypoint.sh:52 -- the TERM/INT trap runs pg_ctl ... stop unguarded under set -euo pipefail, so if that stop returns non-zero the shell exits with that code and never reaches the trailing exit 0, inconsistent with the guarded || true stop on the normal exit path at line 56.
    • Fix: Append || true to the pg_ctl stop inside the trap so the handler reliably reaches exit 0.
  • packages/app/tests/e2e/utils/db-helpers.ts:12 -- the DocumentDB credential plus the tls=true&tlsAllowInvalidCertificates=true&authMechanism=SCRAM-SHA-256 fragment is duplicated as DOCDB_AUTH/DOCDB_OPTIONS here and as DEV_DOCDB_AUTH/DEV_DOCDB_OPTIONS in packages/api/scripts/seed-alerts.ts, across separate packages where it can drift into a runtime auth/TLS failure.
    • Fix: If a shared test-util module becomes available, hoist the fragment there; otherwise keep the existing cross-reference comments.
    • maintainability
  • packages/app/tests/e2e/utils/db-helpers.ts:16 -- the transient dropDatabase codes [1, 112] and the retry policy are encoded twice: a hand-rolled loop here and withRetry(..., retryOnlyOnStatus: [1, 112]) in fixtures.ts, so a new transient code must be updated in lockstep in two places.
    • Fix: Add a comment in each site pointing at the other, or centralize the code list if the packages can share it later.
    • maintainability

Reviewers (8): correctness, reliability, adversarial, maintainability, project-standards, testing, kieran-typescript, previous-comments.

Testing gaps:

  • The driver-migration helpers run only in the full-stack e2e job; non-full-stack CI never executes them, so the DocumentDB path is validated only when the stack is up.
  • clearDatabase() in global-setup-fullstack.ts still only warns on a drop failure (pre-existing), but a genuinely failed drop now fails loudly downstream via the No sources found check rather than false-passing.

Dropped during re-grade: three reported .claude/* "MongoDB regressions" are uncommitted working-tree edits from the review environment, not part of this PR's committed diff (which correctly updated those files) — out of scope, not PR findings.

No finding blocks merge automatically. A maintainer will expect P0/P1 findings in code this PR changes to be fixed; P2/P3 are your call -- fix or reply. Never fix findings about surrounding code here; reply instead. Do not widen the PR. How to respond

…s start errors

CI e2e setup failed on every shard: `buildx bake` on the e2e compose file
resolves its out-of-tree build context as a filesystem entitlement and
refuses to run. Build docker/documentdb directly with the same GHA cache.

A failed Postgres start now prints the tail of postgresql.log to the
container logs, so `docker compose logs db` shows the cause.
@changeset-bot

changeset-bot Bot commented Oct 15, 2026 •

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: 03bb6b7

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

The integration job's `up --wait` failed before any test ran: the OTel
collector's healthcheck needs an OpAMP server the job doesn't run, so it
never reports healthy. Start every service as before, then wait on db.
Comment thread packages/api/scripts/seed-alerts.ts Resolved
catinspace-au added a commit to hyperi-io/dfe-hyperdx that referenced this pull request Oct 22, 2026
0006 records DocumentDB 1.0 replacing FerretDB, following ClickHouse/ClickStack#3375. 0001 keeps its text and gains a status line pointing at it, and the index marks it superseded.
Integration shards were rebuilding DocumentDB from source and timing out.
The bake shared one GHA cache scope between the collector and db images,
and a scope holds a single manifest, so each export overwrote the other.
db now uses the documentdb scope the e2e job already fills.
@jordan-simonovski
jordan-simonovski requested a review from a team as a code owner November 5, 2026 05:27
- Build DocumentDB v1.2-0 (pinned to its commit) instead of v1.0-RC2.
- Publish DocumentDB on loopback only in the dev, integration and e2e
  compose files; the credentials are well known.
- Name the shared DocumentDB credentials and TLS options in the e2e
  helpers and the alert seed script, and type the drop retry's error.
- Keep the entrypoint's TERM trap from aborting before Postgres stops.
- Update the e2e README and the AGENTS.md data-flow line for DocumentDB.
docker buildx create --use --driver=docker-container
docker buildx bake -f ./docker-compose.ci.yml --set *.cache-to="type=gha,mode=max,scope=ci-integration" --set *.cache-from="type=gha,scope=ci-integration" --load
# One cache scope per image (a scope holds one manifest); db reuses the e2e job's documentdb scope.
docker buildx bake -f ./docker-compose.ci.yml \

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 P2 — Building the DocumentDB image from source on a cache miss probably won't fit the 16/15-minute CI job timeouts

The Dockerfile compiles the C extensions serially (make install-documentdb, no -j) and runs a release cargo build of the gateway. Any change to docker/documentdb/**, or the GHA cache evicting the large documentdb scope (mode=max keeps the Rust and C builder layers), makes all 4 integration shards (timeout-minutes: 16) and all 4 e2e shards (.github/workflows/e2e-tests.yml, timeout-minutes: 15) cold-build at once before tests start. Fix: build the image once in a dedicated upstream job (or push it to GHCR) that the shards needs:/pull, or raise the timeouts to allow for a cold build.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

'common-utils',
'clickhouse',
'mongodb',
'documentdb',

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Screenshot 2026-11-19 at 12 48 48 PM

minor nit: we should make documentdb formatting better

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This might be another portal running which picked up on this. This should be formatted properly, but the portal will use whatever worktree process is currently occupying the port.

Comment thread agent_docs/development.md
- Docker Compose manages ClickHouse, MongoDB, OTel Collector
- Docker Compose manages ClickHouse, DocumentDB, OTel Collector
- The metadata store is [DocumentDB](https://github.com/documentdb/documentdb)
(Postgres plus a Mongo wire-protocol gateway), built from

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was trying to connect via postgres directly as well, but we dont expose the port. do you think its worth exposing or later problem?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, I'd say that we probably don't need the PG port exposed for now. Added a docker exec command to development.md if we need to jump into PG to debug anything.

### Default: Full-Stack Mode

By default, `make e2e` runs tests in **full-stack mode** with MongoDB + API +
By default, `make e2e` runs tests in **full-stack mode** with DocumentDB + API +

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should update (they describe the local stack)

File Line Current text Change
CONTRIBUTING.md 240 "reset ClickHouse and MongoDB storage" Say DocumentDB, and mention .volumes/docdb_dev_<slot>
CONTRIBUTING.md 260 "E2E tests run against a full local stack (MongoDB + ClickHouse + API)" DocumentDB
CONTRIBUTING.md 201–202 Lists collector + ClickHouse in Docker, leaves out the db Add DocumentDB, and optionally link to the Compass connection string in development.md
agent_docs/evals.md 31, 78 "Docker containers (ClickHouse, MongoDB)" DocumentDB
agent_docs/evals.md 64 Diagram label Mongo:30498 Fine as-is (the port variable is still HDX_DEV_MONGO_PORT), or rename to DocDB:30498
agent_docs/evals.md 323 "default ports (8123, 27017)" Still accurate: the compose default is still 27017, now mapped to gateway port 10260. No change needed
agent_docs/evals.md 329 "When MongoDB is recreated…" DocumentDB
packages/hdx-eval/README.md 16, 60 "ClickHouse, MongoDB" DocumentDB
.claude/skills/playwright/SKILL.md 81 "@full-stack — tests requiring MongoDB + API" DocumentDB
.claude/agents/playwright-test-generator.md 61 Same DocumentDB
.claude/agents/playwright-test-planner.md 66 "the test runner clears MongoDB before each run" DocumentDB

@brandon-pereira brandon-pereira left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall works great - couple minor nits for doc improvements (maybe a port exposed) but lets ship it!

Contributing, eval and Playwright docs still said the local dev, e2e and
eval stacks run MongoDB. Also document a psql one-liner for reaching the
Postgres under DocumentDB.
docker buildx create --use --driver=docker-container
docker buildx bake -f ./docker-compose.ci.yml --set *.cache-to="type=gha,mode=max,scope=ci-integration" --set *.cache-from="type=gha,scope=ci-integration" --load
# One cache scope per image (a scope holds one manifest); db reuses the e2e job's documentdb scope.
docker buildx bake -f ./docker-compose.ci.yml \

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 P2 — CI compiles DocumentDB from source on a cold cache inside 15–16 minute job limits

On a cache miss (first run of this PR, eviction from the 10 GB GHA cache, or any change under docker/documentdb), each of the 4 integration shards (timeout-minutes: 16) and the 4 e2e shards (timeout-minutes: 15 in .github/workflows/e2e-tests.yml) builds the Dockerfile in parallel. That means libbson, pcre2, the Intel decimal library, the C extensions and the Rust gateway (cargo build --release), and the tests still have to run inside the same budget. Build the image once, either in a single upstream job that the shards needs: or by pushing it to a registry pinned by digest, and have the shards only pull it.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

Comment thread Makefile
@bash scripts/ensure-dev-portal.sh
docker compose -p $(HDX_CI_PROJECT) -f ./docker-compose.ci.yml up -d
docker compose -p $(HDX_CI_PROJECT) -f ./docker-compose.ci.yml up -d --build
docker compose -p $(HDX_CI_PROJECT) -f ./docker-compose.ci.yml up -d --wait --wait-timeout 300 db

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 P2 — A failed DocumentDB health wait in dev-int skips the down -v cleanup

The new up -d --wait ... db runs as its own recipe line, so when DocumentDB never becomes healthy, make stops there. The teardown on the following line never runs, and ClickHouse, the collector and db stay up on the slot. The same applies to dev-int-common-utils and ci-int. Fold the wait into the same shell line as the test run so down -v always runs, e.g. docker compose ... up -d --wait ... db && bash -c '...'; ret=$$?; docker compose ... down -v; ....

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

This branch was successfully deployed

2 active deployments
Preview – hyperdx-oss — 03bb6b70 Deployed Nov 19, 2026 by vercel[bot]
Preview – hyperdx-storybook — 03bb6b70 Deployed Nov 19, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

review/tier-3 Standard — full human review required

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants