Skip to content

feat(docker): run the all-in-one image on DocumentDB instead of MongoDB 4.0.5 - #3734

Open
jordan-simonovski wants to merge 1 commit into
mainfrom
jordansimonovski/documentdb-aio
Open

jordan-simonovski wants to merge 1 commit into
mainfrom
jordansimonovski/documentdb-aio

Conversation

@jordan-simonovski

Copy link
Copy Markdown
Contributor

Summary

The all-in-one image now runs DocumentDB (MIT, Postgres 18) in place of MongoDB 4.0.5. That removes the SSPL licence and the four critical OpenSSL 1.1.1k CVEs that came with the Alpine 3.9 MongoDB packages. It uses the same DocumentDB build that local dev, integration and e2e have run on since #3375.

  • Metadata stays on the /data/db mount the docs tell people to use, now under /data/db/pgdata.
  • The gateway listens on localhost only, because its hyperdx:hyperdx credentials are fixed. The old mongod was also localhost-only.
  • Image size is about the same: 2.47 GB, against 2.49 GB with MongoDB.

Breaking changes

  • Existing all-in-one users start with an empty metadata database after upgrading: users, sources, dashboards, alerts and webhooks. The old MongoDB files stay in /data/db and are not read. The container logs a warning on first boot when it finds them.
  • The changeset carries the one-off migration steps: mongodump with the 4.0 tools, then mongorestore into the new container. They were tested from an image built on main to this branch, and every document and index came across.
  • Helm and Docker Compose deployments are unaffected; they bring their own MongoDB.
Implementation detail
  • The DocumentDB build moves from docker/documentdb/Dockerfile into stages of docker/hyperdx/Dockerfile. The compose db service builds the documentdb target and the all-in-one copies from the same builder, so the two can't drift.
  • Shared libraries are installed with apk from a package list the builder writes, instead of being copied. Copying overwrote the ClickHouse base's musl and libssl with files from another Alpine point release, the same bug as fix(docker): stop clobbering the base image's /usr/lib, and build images on main #3157. apk audit --system is clean in both images.
  • The build runs ldd over Postgres, the gateway and the extensions, and fails if the base image's Alpine release is missing a library.
  • The postgres user is pinned to UID/GID 70, so files on a volume keep the right owner across upgrades.
  • The MongoDB VEX file is gone, since nothing it covered ships any more.
  • Tested locally on arm64: scripts/ci/smoke-all-in-one.sh passes, and the migration was run end to end. amd64 and the auth image were not built locally.

How to reproduce

Starting point: a checkout of this branch with Docker.

On main:

  1. docker run --rm --entrypoint mongod docker.clickhouse.com/clickhouse/clickstack-all-in-one:2 --version prints MongoDB 4.0.5.

On this branch:

  1. Build the image: docker buildx build . -f docker/hyperdx/Dockerfile --target all-in-one-noauth --build-context clickhouse=./docker/clickhouse --build-context otel-collector=./docker/otel-collector --build-context hyperdx=./docker/hyperdx --build-context api=./packages/api --build-context app=./packages/app -t aio:local --load
  2. Run scripts/ci/smoke-all-in-one.sh aio:local. It should print canary ingested end-to-end.
  3. To test the upgrade, run the main image with -v vol:/data/db, then follow the steps in .changeset/aio-documentdb.md.

References

…DB 4.0.5

MongoDB 4.0.5 is EOL, SSPL licensed, and comes from Alpine 3.9 packages
that carry OpenSSL 1.1.1k critical CVEs. The all-in-one image now runs
DocumentDB v1.2-0 (MIT, Postgres 18) from the same Dockerfile stage the
dev, integration and e2e stacks use. Metadata stays on the /data/db mount
under /data/db/pgdata; the changeset documents moving old MongoDB data.
@jordan-simonovski
jordan-simonovski requested a review from a team as a code owner November 19, 2026 23:03
@vercel

vercel Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
hyperdx-oss Ready Ready Preview Nov 19, 2026 11:06pm UTC
hyperdx-storybook Ready Ready Preview Nov 19, 2026 11:06pm UTC

Request Review

@changeset-bot

changeset-bot Bot commented Nov 19, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 6ff89f8

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 3 packages
Name Type
@hyperdx/api Minor
@hyperdx/app Minor
@hyperdx/otel-collector Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@github-actions github-actions Bot added the review/tier-4 Critical — deep review + domain expert sign-off label Nov 19, 2026
@github-actions

Copy link
Copy Markdown
Contributor

🔴 Tier 4 — Critical

Touches authentication, tenancy data models, the public API or shipped database config — or substantially changes the query rendering engine, background tasks, the OTel pipeline, image build, or release CI.

Why this tier:

  • Background tasks or delivery pipeline substantially modified — 98 lines (bar: 30):
    • docker/hyperdx/Dockerfile
    • docker/hyperdx/entry.local.base.sh

Review process: Deep review from a domain expert. Synchronous walkthrough may be required.
SLA: Schedule synchronous review within 2 business days.

Stats
  • Production files changed: 7
  • Production lines changed: 308 (+ 5 in test files, excluded from tier calculation)
  • Critical-path lines changed: 98
  • Branch: jordansimonovski/documentdb-aio
  • Author: jordan-simonovski

To override this classification, remove the review/tier-4 label and apply a different review/tier-* label. Manual overrides are preserved on subsequent pushes.

@github-actions

Copy link
Copy Markdown
Contributor

Duplicate code analysis

No new duplicate code introduced by this PR.


jscpd compared this branch against origin/main
across app, api and common-utils. Only new clones are listed; the codebase
currently holds 108 in total (1.13%).
Run yarn dupes locally to reproduce, or yarn dupes:all for the full picture.

Comment thread docker/hyperdx/Dockerfile
FROM alpine:3.24 AS documentdb
COPY --from=documentdb-builder /out/ /
RUN apk add --no-cache bash openssl su-exec $(cat /etc/documentdb/apk-packages) \
&& addgroup -S -g 70 postgres && adduser -S -u 70 -G postgres -h /var/lib/postgresql postgres \

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 P1 — Pinning postgres to UID/GID 70 breaks existing dev DocumentDB volumes

The old docker/documentdb/Dockerfile ran addgroup -S postgres && adduser -S ..., which on alpine:3.24 picks the first free system ID (100 or above), not 70. docker-compose.dev.yml:21 keeps /data/docdb on a host bind mount (.volumes/docdb_dev_<slot>), so a dev who already ran #3375 has a PGDATA full of files owned by the old UID. docker/documentdb/entrypoint.sh:9 (install -d -o postgres ...) only re-owns the top directory. The .documentdb-initialised marker skips initdb, so pg_ctl running as UID 70 gets permission denied on PG_VERSION/global/* and the db container never becomes healthy. Fix: in the root branch of the entrypoint, re-own the contents when the owner differs, e.g. [ "$(stat -c %u "$PGDATA")" = "$(id -u postgres)" ] || chown -R postgres:postgres "$PGDATA", run before the install -d.

Nothing blocks merge automatically, but a maintainer will expect this fixed if it is a real defect in code this PR changes. If it is about surrounding code, reply and say so instead of patching. Do not widen the PR. How to respond

fi

# Start DocumentDB (Postgres + Mongo wire-protocol gateway on :10260)
DOCDB_USER=hyperdx DOCDB_PASSWORD=hyperdx PGDATA=/data/db/pgdata \

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 P2 — In the all-in-one image the DocumentDB entrypoint's clean-shutdown trap never fires, so Postgres is SIGKILLed on every docker stop

docker/documentdb/entrypoint.sh:51-52 relies on receiving SIGTERM to run pg_ctl stop ("so persistent volumes skip crash recovery"). In the all-in-one image, PID 1 is sh /etc/local/entry.sh, which sets no trap and does not forward signals. docker stop therefore waits out its timeout and SIGKILLs everything, and Postgres runs crash recovery on each boot. Fix: add a trap in entry.local.base.sh that forwards TERM/INT to the background DocumentDB PID (and the others) and waits for them, or run the services under a small init such as tini.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

@github-actions

Copy link
Copy Markdown
Contributor

PR Review

If you are a coding agent acting for the author, read this first.
Nothing below blocks merge automatically; a maintainer decides. What they will expect:

  • 🔴 P0/P1 in code this PR adds or changes: fix before asking for review.
  • 🟡 P2 in code this PR changes: your call. Fix if small, otherwise reply.
  • 🔵 P3 is collapsed at the bottom: nobody needs to act on it before merge.
  • Anything about surrounding code, or asking you to widen the change (hoist a helper,
    dedupe with another file, fix other call sites): never fix here, whatever the
    priority. Reply with one sentence; if it is P0, say so plainly so a human sees it.

One commit per review round. After two rounds, stop and ask a maintainer to review
scope rather than addressing more automated findings. Full rule: AGENTS.md.

4 finding(s): 🔴 1 P0/P1 · 🟡 1 P2 · 🔵 2 P3

2 posted as inline comment(s) on the changed lines. 2 P3 folded at the bottom.

🔵 2 P3 — no action needed before merge
  • 🔵 P3 docker/hyperdx/Dockerfile:266 — ldd compatibility check only catches missing libraries, not symbol/version mismatches → The guard greps only Error loading shared library. If the ClickHouse base installs an older point release of a library with the same filename (libssl3, libxml2, etc.), musl's ldd instead prints Error relocating ...: symbol not found. The build then passes and Postgres or the gateway fails at runtime, which is exactly the fix(docker): stop clobbering the base image's /usr/lib, and build images on main #3157 class of bug the comment says this guards against. Also, psql/pg_ctl/initdb are not checked. Fix: match Error (loading|relocating) (or any Error), and include $PG/psql $PG/pg_ctl $PG/initdb in the ldd list.
  • 🔵 P3 scripts/ci/smoke-all-in-one.sh:57 — No test exercises the new persistence/restart path (/data/db/pgdata, marker file, UID 70 ownership) → The smoke test boots a fresh container with no volume and never restarts it. A broken .documentdb-initialised marker check, wrong PGDATA ownership, or a PGDATA path outside /data/db would all still print canary ingested end-to-end. Fix: add a mode that runs with -v <vol>:/data/db, writes a document (or checks the default team or sources), restarts the container, and asserts that the data and API health survive the restart.

Priority comes from a separate grading pass over the finished review. P3 is folded here instead of posted inline. No finding blocks merge automatically; a maintainer decides.

@github-actions

Copy link
Copy Markdown
Contributor

Deep Review

✅ No critical issues found.

This change swaps bundled MongoDB 4.0.5 for DocumentDB (Postgres 18 + Mongo wire gateway) in the all-in-one image. The high-risk paths hold up: the first-boot init deletes only /data/db/pgdata (find "$PGDATA" -mindepth 1 -delete), so pre-existing MongoDB files in /data/db survive — the documented empty-start is the only metadata loss, and it is warned on boot and in release notes. The gateway is pinned localhost-only in the all-in-one via the UseLocalHost sed plus a grep -q assertion, compose ports publish to 127.0.0.1 only, and the .documentdb-initialised marker correctly gates both init idempotency and the leftover-MongoDB warning.

🟡 P2 -- recommended

  • docker/hyperdx/entry.local.base.sh:70 -- DocumentDB is backgrounded and the all-in-one entrypoint does not forward container SIGTERM to it, so the clean-shutdown trap in docker/documentdb/entrypoint.sh never fires here; on docker stop Postgres is killed abruptly and runs WAL crash recovery on the next boot, which defeats the entrypoint's stated clean-shutdown goal (that trap only works in the standalone compose container where the entrypoint is PID 1).
    • Fix: install a trap in entry.local.base.sh that relays TERM/INT to the backgrounded DocumentDB PID and waits for it before the shell exits.
🔵 P3 nitpicks (3)
  • docker/documentdb/entrypoint.sh:1 -- after the DocumentDB Dockerfile moved into docker/hyperdx/Dockerfile stages, docker/documentdb/ holds only entrypoint.sh, so the build logic and its entrypoint now live in separate trees with nothing signposting the split.
    • Fix: add a short pointer comment (or README) in docker/documentdb/ noting the build stages now live in docker/hyperdx/Dockerfile.
  • docker/hyperdx/Dockerfile:463 -- the runtime setup (apk add bash openssl su-exec $(cat .../apk-packages), create postgres UID/GID 70, copy the entrypoint) is duplicated between the standalone documentdb stage and the all-in-one-base RUN, so a future change to DocumentDB's runtime deps must be made in two places.
    • Fix: note the intentional duplication with a comment, or factor the shared runtime setup so the two targets cannot drift.
  • docker/hyperdx/entry.local.base.sh:70 -- the API is not gated on DocumentDB readiness (only ClickHouse is waited on), and DocumentDB's first boot runs initdb plus extension creation, a materially longer window than the prior mongod start even though both are un-gated.
    • Fix: add a lightweight readiness wait (e.g., nc -z 127.0.0.1 10260 loop) before relying on the metadata store, mirroring the ClickHouse wait.

Reviewers (10): correctness, security, adversarial, reliability, data-migrations, maintainability, testing, project-standards, learnings-researcher, previous-comments.

Testing gaps:

  • The upgrade path (old MongoDB volume → empty DocumentDB plus the leftover-data warning) is covered only by the manual mongodump/mongorestore procedure in the changeset; no automated test exercises it or asserts the /data/db/WiredTiger warning fires.
  • Per the PR, only arm64 was built locally; the amd64 and auth-image builds (and the ldd/apk-packages build-time guards) are first exercised in CI.
  • Prior PR comments were Vercel/changeset-bot automated notices only — no substantive human feedback to verify.

No finding blocks merge automatically. A maintainer will expect P0/P1 findings in code this PR changes to be fixed; P2/P3 are your call -- fix or reply. Never fix findings about surrounding code here; reply instead. Do not widen the PR. How to respond

@github-actions

Copy link
Copy Markdown
Contributor

E2E Test Results

✅ All tests passed • 474 passed • 1 skipped • 1107s

Status Count
✅ Passed 474
❌ Failed 0
⚠️ Flaky 8
⏭️ Skipped 1

Tests ran across 4 shards in parallel.

View full report →

This branch was successfully deployed

2 active deployments
Preview – hyperdx-oss — 6ff89f8f Deployed Nov 19, 2026 by vercel[bot]
Preview – hyperdx-storybook — 6ff89f8f Deployed Nov 19, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

review/tier-4 Critical — deep review + domain expert sign-off

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants