Repository navigation
Description
Command Code: Reliability, Java Engineering Quality, and Developer-Experience Roadmap
GitHub issue / discussion proposal for Command Code
Evidence: Independent TableNotifier — Java 25 coding-agent benchmark, 15 November 2026
Priority: Correctness first; then test effectiveness, simplicity, responsiveness, and cost
Audience: Command Code maintainers, harness engineers, and contributors
Status: External engineering feedback. Not a claim that every weakness in generated code is a defect in the harness.
Executive summary
Command Code produced the strongest original TableNotifier implementation in our independent comparison. It passed 13/14 shared source-derived behavioral checks, versus 11/14 for OpenCode V2 and 9/14 for DeepSeek Harness (DSH). It demonstrated particularly strong handling of Spring transaction completion, worker ownership, callback isolation, and concurrency races.
That is a meaningful result, but not a definitive general-purpose harness ranking. The original Java 25/real-dependency suites were not independently rerun in the audit environment; the independent checks used Java 21-compiled, source-derived implementations and controlled NATS/Spring/Micrometer stand-ins. Another participant, AIR (Codex), was examined with an expanded and different 22-check suite, so its score is not numerically comparable. The harness versions, model conditions, instruction sets, and full time/cost traces were not controlled identically.
Our proposal: Turn Command Code's demonstrated correctness advantage into a more consistently verifiable, efficient, explainable, and maintainable development workflow. The highest-value improvements are not “add more AI agents” or “produce longer plans”; they are:
- Risk-driven verification: translate contracts into adversarial tests, then deliberately challenge assumptions the planner and implementation share.
- A production-failure scenario library: lifecycle, transactions, NATS/network outages, exceptions, races, failure atomicity, and thread-context isolation.
- Better code-shape discipline: achieve equivalent guarantees with fewer states, fewer defensive layers, and less incidental complexity.
- Transparent execution: show live phase, progress, verification evidence, and the reasons for delay or additional tool calls.
- Repeatable engineering benchmarks: compare correctness, severity of defects, planning/implementation time, tokens, cache use, review effectiveness, and maintainability with identical models and test suites.
Request to maintainers: Please consider this as a prioritized collection of engineering quality improvements and a proposal for a reproducible benchmark suite, not a request to implement every feature by default.
1. Benchmark context and evidence standard
Task
Build TableNotifier, a small production-oriented Java 25 library that uses NATS Core table-change hints to trigger polling, with independent periodic polling as a reliability fallback. The requirements emphasized:
- An injected, application-owned
io.nats.client.Connection; never close or reconfigure it. - Subject validation and multicast delivery (no queue groups).
- Independent listeners, no overlapping callback execution, burst coalescing, and follow-up polling when a notification arrives during a poll.
- Spring transaction integration: publish after successful commit, suppress on rollback, and deduplicate per transaction.
- Clean lifecycle; no dedicated platform thread per listener; no unbounded per-event queues.
- Micrometer metrics, manual NATS CLI documentation, and strong deterministic tests.
- Java 25, Gradle, Lombok, jnats, Spring TX, JUnit 5, and Google Truth.
Participants:
| Participant | Harness | Observed workflow difference |
|---|---|---|
Command Code (cmdc) |
Command Code | Original Terra/DeepSeek task; planning and review artifacts present |
| OpenCode V2 | OpenCode | Original Terra/DeepSeek task |
| DSH | DeepSeek Harness | Original Terra/DeepSeek task; user reported a particularly fast/low-cost experience |
| AIR | Codex | Newer AGENTS.md; review by GPT-6.1 Sol, not Terra; user reported unusually long planning and repeated requests for progress |
Interpretation: The benchmark evaluates end-to-end outputs of differently configured workflows, not isolated intrinsic abilities of each harness. A defect in a generated Java library may reflect a model decision, misleading plan, missing test, execution policy, or harness/tooling design. Establish root cause with controlled ablations before attributing it to Command Code.
Verification evidence and limitations
| Evidence | Status | What it establishes |
|---|---|---|
| Source inspection of four submission archives | Performed | Architecture, implementation patterns, submitted tests, and declared design decisions |
| Same 14 source-derived behavioral probes on DSH/OpenCode/Command Code | Executed | Comparable isolated behavior for those three implementations |
| AIR's 22 isolated probes | Executed separately | Additional AIR-specific failure modes and strengths; not a common denominator |
| Java 25 original Gradle/JUnit suite independently rerun | Not performed | Original integration correctness remains unverified by this independent audit |
| Genuine Spring, Micrometer, jnats and broker restart tests by independent evaluator | Not performed | Stand-ins cannot validate every library-specific lifecycle or reconnection behavior |
| Command Code's own claimed 91 tests, one optional real-NATS round trip | Submission report, not independently reproduced | Evidence of engineering effort and recorded self-verification, not an independent pass certificate |
| Measured same-model time, tokens and cost across all harnesses | Unavailable | Do not claim a measured speed or price ranking from this benchmark |
The benchmark intentionally did not penalize any participant for missing Gradle Wrapper binaries: the user had separately instructed participants to use installed Gradle 9.9.1 and minimize archive size.
Original shared 14-check results
| Independent scenario | Command Code | OpenCode V2 | DSH |
|---|---|---|---|
| Basic multi-listener delivery + fallback | PASS | PASS | PASS |
| 1,000 notification burst; callback non-overlap | PASS | PASS | PASS |
| Missed notification repaired by periodic polling | PASS | PASS | PASS |
| Notifier shutdown releases owned resources; borrowed NATS stays open | PASS | PASS | FAIL |
| Spring commit/rollback/same-table deduplication | PASS | PASS | PASS |
Additional notification registered from another afterCommit callback |
PASS | FAIL | PASS |
Callback AssertionError does not prevent later polls |
PASS | PASS | PASS |
| Bound resource without actual Spring transaction | PASS | PASS | FAIL |
AssertionError counted as poll failure |
PASS | PASS | FAIL |
Worker avoids inheriting request InheritableThreadLocal |
PASS | FAIL | FAIL |
createDispatcher() fails, but fallback-only service starts |
FAIL | FAIL | FAIL |
| 12 concurrent clients Ă— 150 operations | PASS | PASS | PASS |
| Notifier closed from within its own callback | PASS | PASS | PASS |
| Subscription closed during long-running callback | PASS | PASS | PASS |
| Total | 13/14 | 11/14 | 9/14 |
Severity matters: A numerical “14-check score” is only a useful first indicator. These tests vary in business impact, and the strict dispatcher-failure check is partly a requirement-interpretation question: does “poll during NATS outage” also mandate successful construction if creation of the jnats dispatcher itself fails? The desired contract must be made explicit and tested against real jnats.
AIR / Codex: separate expanded evaluation
AIR passed 15/22 isolated probes and 4/4 main concurrent-stress scenarios; seven probe failures reflected six distinct concerns, including two manifestations of one Spring error. Its confirmed concerns were: false Spring transaction detection when a resource is bound without a transaction; inherited request thread context; callback self-interruption; blocked publish holding a lifecycle lock; dispatcher-startup failure; and a missing notifyAfterCommit method on the public interface. This cannot be subtracted from Command Code's 13/14 or used to declare a seven-defect difference.
AIR had extensive planning and test documentation but still missed fundamental assumptions. The lesson for all harnesses is that plan length and test count do not substitute for independent risk verification.
Code-shape and self-reported testing comparison
| Measure | Command Code | OpenCode V2 | DSH | AIR |
|---|---|---|---|---|
| Production Java files | 8 | 5 | 7 | 15 (incl. two package-info.java) |
| Production Java lines, incl. comments/blank lines | 1,297 | 904 | 1,008 | 1,330 |
| Largest production Java class | 336 | 557 | 267 | 547 |
| Separate core and Spring modules | Yes | Yes | No | Yes |
| Tests claimed in final report | 91 (one skipped) | 98 | 48 (47 pass, one skipped) | 103 deterministic |
Line counts measure size, not quality. Command Code's larger solution avoided several defects that the smaller solutions exhibited; its next opportunity is to retain those invariants with less incidental complexity. Similarly, 91 passing tests do not establish a low defect rate unless the tests probe the correct failure modes.
2. What Command Code did exceptionally well — preserve these strengths
- Correct resource ownership. It separates borrowed NATS
Connection/MeterRegistryfrom owned dispatcher, scheduler and callback executor. Tests exercise teardown and leave caller-owned resources alive. - Spring transaction reasoning. It uses a transaction synchronization, publishes in
afterCompletion(STATUS_COMMITTED), handlesREQUIRES_NEW, avoids interpreting an arbitrary bound resource as an active transaction, and prevents a callback failure from undoing a committed DB transaction. - Thread-context isolation. Worker creation explicitly disables inheritable thread-local capture. This prevents a leak independently detected in OpenCode, DSH, and AIR.
- Callback reentrancy/lifecycle. Self-close and closure during active processing passed the shared suite. This is exactly the sort of fault that ordinary happy-path tests miss.
- Bounded notification state. Notifications are coalesced into fixed per-listener state rather than queued one by one.
- Meaningful initial tests. Concurrency, error recovery, transaction boundaries and resource cleanup were addressed in submitted tests; the plan anticipated several defects competitors later exhibited.
- Appropriately qualified release reporting. The submission disclosed reconnect and soak-test gaps, a gauge warning, and compatibility limits instead of claiming unconditional production readiness.
Guardrail: Any simplification, faster execution strategy, model routing, or new automatic repair feature must preserve these properties. Do not exchange strong failure semantics for prettier code or lower latency.
3. Concrete weaknesses in Command Code's generated TableNotifier
Each item distinguishes direct evidence from a proposal. File paths are relative to the archived cmdc/ submission. They describe the generated project, not necessarily defects in Command Code's own repository.
C1 — Dispatcher creation failure prevents fallback-only startup
- Evidence: Independently reproduced failure in the shared 14-check audit (1 of 14 failures for Command Code).
- Code:
table-notifier-core/src/main/java/io/tablenotifier/NatsTableNotifier.java, constructor around lines 93–117. It callsconnection.createDispatcher(...)and rethrows onRuntimeException, shutting down its owned executors. - Impact: Under the strict “database polling continues despite unavailable NATS” interpretation, an unavailable NATS dispatcher at startup prevents the notifier from existing and starting its polling listeners.
- Important qualification: The fail-fast constructor is a defensible API choice if dispatcher-creation failure is explicitly fatal. Disconnect after successful construction is a different state. Do not silently hide a broken connection or security/permission error as mere unavailability.
- Recommendation: Specify a decision table: disconnected existing connection; dispatcher creation failure; subscription failure; reconnection; permission denial; permanent misconfiguration. Offer an opt-in degraded/fallback-only startup policy if the contract demands it, with bounded recovery, visible health/metrics and no busy waiting. Alternatively document fail-fast as the supported contract and update acceptance requirements accordingly.
- Acceptance criteria: Independent initial/fallback polls proceed during an authorized transient startup failure; NATS subscription recovers when possible; close releases every owned resource; permanent errors remain observable; real jnats semantics tested.
C2 — Duplicate Micrometer gauge registration warning
- Evidence: The submission's own
docs/FINAL_REPORT.mdexplicitly reportsWARNING: This Gauge has been already registeredonce per registry. This is self-reported, not separately reproduced. - Code:
table-notifier-core/src/main/java/io/tablenotifier/NotifierMetrics.java,activeState(...), around lines 62–80, registers aGaugeon each new metrics facade, with registry-specific weak-map bookkeeping and a strong gauge state reference. - Risk: Noisy logs, unnecessary registry bookkeeping, complicated ownership semantics under multiple notifiers sharing one registry; possible ambiguity if registries/meters are removed/reused.
- Recommendation: Define gauge identity, aggregation and lifetime explicitly. Prefer deterministic, registry-scoped registration with no duplicate warnings; use registry APIs consistently and test multi-notifier add/remove/GC/registry lifecycle. Do not add a home-grown metrics framework.
- Acceptance criteria: 1, 2, and many concurrent notifier instances sharing a registry show the correct listener total; no duplicate registration warnings; no stale counts after close; bounded memory retention.
C3 — Spring adapter has an easy-to-misuse constructor
- Evidence: Declared by the submission's own final report; confirmed in source.
- Code:
table-notifier-spring/src/main/java/io/tablenotifier/spring/SpringTableNotifier.java, constructors around 38–56. The two-argument overload rejects anAbstractPlatformTransactionManagerwithSYNCHRONIZATION_NEVER; the one-argument overload cannot validate this configuration. - Impact: A consumer can choose the less-safe constructor and unknowingly get post-commit semantics that are impossible to enforce with synchronization disabled. The implementation currently logs and skips publication when an actual transaction has no synchronization: this preserves transaction safety but makes the missing hint an operational concern.
- Recommendation: Prefer one clearly safe documented bean-wiring path; expose explicit validation/fail-fast policy for supported transaction managers. If retaining the one-argument path, make its limitation unavoidable in API documentation and diagnostics. Avoid global static transaction-state heuristics.
- Acceptance criteria: A runnable Spring bean example rejects unsupported settings before use; correct behavior for no transaction, bound-resource/no transaction, sync-only scope, rollback, commit,
REQUIRES_NEW, and active transaction with sync disabled.
C4 — Reconnection, broker restart and sustained load remain unverified
- Evidence: Explicit gap in
docs/FINAL_REPORT.md: no broker-restart recovery test, multi-instance soak, or production fleet validation. The optional live-NATS round trip claimed by the submission is only one scenario. - Risk: A local fake connection cannot prove real dispatcher/subscription recovery after reconnect, authentication failure or connection churn. Likewise a single 1,000-message burst cannot establish sustained polling-capacity or jitter behavior.
- Recommendation: Add reproducible, separate opt-in integration tests with a disposable local NATS server: disconnect/reconnect, restart, listener added during outage, publish failures, simultaneous instance fanout, and observable fallback polling. Keep the default unit suite free of broker dependencies.
- Acceptance criteria: Broker restart tests reliably reproduce and assert recovery; no accidental closure of the injected connection; fallback continues throughout outage; test results and versions are archived.
C5 — Potential unbounded cross-listener callback concurrency
- Evidence: Design risk, not a reproduced defect.
NatsTableNotifier.newWorkerExecutor()usesExecutors.newThreadPerTaskExecutor(Thread.ofVirtual()...)(around 129–134). Per-listener non-overlap is enforced byPollingSubscription, but a large number of different listeners may poll simultaneously. Virtual threads reduce thread cost, not database capacity use. - Impact: Database connection-pool contention, callback latency inflation, and missed nominal fallback cadence under many slow listeners. No benchmark demonstrated safe capacity for thousands of independent listeners.
- Recommendation: First measure workload-specific capacity: listener count Ă— poll duration / target interval, real DB pool size, and notification bursts. If necessary introduce optional bounded global admission with fairness and explicit saturation metrics. Preserve the no-lost-follow-up and no-overlap guarantees. Do not insert a fixed small pool without capacity analysis.
- Acceptance criteria: Repeatable workload curves for active listeners, simultaneous callbacks, poll latency, delayed fallback, memory and cleanup; no unbounded callback backlog; no starvation; minimal overhead at ordinary load.
C6 — Defensive state-machine complexity can be reduced carefully
- Evidence:
NatsTableNotifier.javais 336 lines;PollingSubscription.javais 324 lines, with multiple synchronized flags (closed,running,admissionRequested,pendingReasons,notificationPending) and three scheduled-future fields. This is a maintainability concern, not evidence the code is incorrect. - Impact: Future contributors must reason about a fairly large product of possible states. Over-simplification could reintroduce lost wakeups or concurrency bugs.
- Recommendation: Document the finite state transitions and invariants first; write state-transition and fault-injection tests; then remove only state shown redundant by a proof or executable reference model. Keep cohesive classes and avoid an extra abstraction for every field.
- Acceptance criteria: Same or better adversarial pass rate; fewer independent mutable states or less control-flow complexity; no measurable throughput regression; maintainers can explain the state machine in one short diagram.
C7 — Exception reporting may be unnecessarily noisy
- Evidence: The earlier independent audit notes an
AssertionErrorcorrectly counted in failure metrics, with recovery of later polls, but possibly rethrown as an uncaught virtual-thread error. This is an observed/logging concern, not a demonstrated availability failure. - Recommendation: Distinguish cancellation/interruption, ordinary callback failure, and JVM-level
Error; decide a consistent logging policy that avoids both silent failures and duplicate stack traces. PreserveErrormetrics and future fallback polls; never silently swallow fatal JVM failures merely for a green test. - Acceptance criteria: Every error path is visible exactly as intended;
AssertionErrortests show recovery and counted failures; no excessive duplicate logs under repeated failures.
C8 — Compatibility, packaging and reproducible delivery need hardening
- Evidence: Self-reported: only Spring 7.0.10 verified; no Spring 6/Boot 3 run; no CI, sources/Javadoc JAR packaging, license, or publishing metadata in the submission. These were not all mandatory to satisfy the original benchmark.
- Recommendation: Add a small supported-version compatibility matrix, CI verification on Java 25 and supported Spring generations, publishing metadata and reproducible artifact evidence. Keep third-party infrastructure opt-in.
- Acceptance criteria: Clearly stated compatibility; green documented CI matrix; generated release artifacts with sources/Javadoc; no surprise dependency inflation.
4. Command Code harness roadmap (proposals, not confirmed product defects)
Current Command Code documentation already describes Plan Mode and review/approval, planning/implementation model lanes, task ledgers/background agents, hooks, MCP, session checkpoints, context compaction, headless JSON output, and Taste learning. This roadmap should build on those primitives, not propose them again as missing features. See the official Plan Mode, Background Tasks, Context, Hooks, Sessions, and Headless references.
H1 — Risk-to-test traceability as an optional quality mode [P0]
Problem: Agents can claim “comprehensive tests” while never testing the exact assumption that will fail in production. AIR's incorrect Spring transaction classifier illustrates this failure mode; DSH's executor ownership gap demonstrates the same general problem. Command Code did better here, but can systematize it.
Proposal: For substantial tasks, maintain a concise machine-readable or Markdown risk ledger:
| Field | Example |
|---|---|
| Requirement/invariant | No publishing before successful Spring commit |
| Negative scenario | Resource bound, no real transaction |
| Failure injection | Synchronization unavailable; registration throws |
| Proof/test | Named test + outcome + actual command |
| Confidence | Real dependency / controlled fake / static inspection / unverified |
| Owner and status | Planner / implementer / reviewer, open or verified |
The harness should prevent accidental omission in its final report, not prevent a developer from making partial progress. It should surface uncovered high-risk rows instead of padding tests with low-value cases.
Acceptance: In a seeded set of Java tasks, the risk ledger increases the proportion of independently found critical defects and shrinks “claimed verified / actually untested” gaps, with acceptable token/time overhead. Keep it off for trivial edits.
H2 — Independent adversarial review that challenges the plan itself [P0]
Problem: If the same plan informs implementation and review, an incorrect assumption may receive repeated approval. Plan compliance is not requirement correctness.
Proposal: A review mode that reads (a) the original user task, (b) current source/diff, and (c) executed test evidence, then challenges the plan rather than using it as the oracle. Ideally place a reviewer in a fresh context or different model for correctness-critical work; selectively withhold plan commentary until after the reviewer has independently identified the likely failure modes.
Require explicit risk checks for transaction lifecycle, resource ownership, callback failures, concurrency, reentrancy, and shutdown whenever those features exist. Review should produce reproducible counterexamples (failing tests), not merely checklist scores or assertions that code looks good.
Acceptance: Blind reviewer finds seeded bugs that plan-following reviewers miss; false-positive rates and review cost are measured. No mandatory second agent for low-risk changes. Respect permissions for any executed tests.
H3 — A Java reliability scenario pack, not a bulky global prompt [P0]
Deliver a versioned, on-demand Java verification skill containing minimal reproducible scenarios for:
- Atomic updates with injected
hashCode/equals/callback failures. AutoCloseableownership and partial initialization failure.- Callback-triggered shutdown, external shutdown, races with subscribe/unsubscribe.
- Event arriving during active processing, burst coalescing, fairness, bounded queues.
InheritableThreadLocaland request/MDC context isolation.- Spring commit/rollback,
REQUIRES_NEW, synchronization-only scope, synchronization disabled, and lateafterCommitregistrations. - NATS disconnect/reconnect, dispatcher registration failures, lost messages, shared connection ownership.
- Metrics consistency,
ErrorversusException, duplicate meter registration. - Deterministic seeded stateful/reference-model tests for tricky state machines.
Implementation choice: Skills or verified test templates (JUnit + Truth + Mockito as appropriate). Do not load the entire scenario catalog into every prompt, require Docker by default, or prescribe one state-machine implementation.
Acceptance: The pack detects all known TableNotifier defects across all four archived submissions with original dependencies, with per-case failure logs. Avoid changing production code merely to satisfy the pack.
H4 — Verified evidence, not “tests passed” prose [P0]
Use existing execution and headless JSON capabilities to produce a minimal verification receipt for important tasks:
# Illustrative output schema, not a documented Command Code API
checks:
- command: "gradle test"
exit_code: 0
duration_ms: 32100
result: "91 tests; 1 skipped"
evidence: "build/test-results/test/"
- command: "gradle integrationTest"
status: "not_run"
reason: "No authorized broker available"
known_risks:
- "NATS broker-restart behavior not verified"Important: the system should read actual process exit codes and test XML, flag skipped or filtered tests, and distinguish test claims from observed execution. If a test is not runnable, it should say unverified rather than silently count it as passed.
Acceptance: A machine-readable receipt can be reconstructed from the session log; false success claims are flagged; receipts remain concise and redact credentials/hostnames.
H5 — Preserve quality while reducing implementation complexity [P1]
Problem: High-correctness implementations can accumulate defensive bookkeeping, duplicate adapters, and large control-flow graphs. Command Code's TableNotifier is more complex than DSH's, though observably more correct.
Proposal: Add a single bounded simplification pass after correctness verification. Ask:
- Which state variables are necessary for the stated invariants?
- Which abstractions exist only for tests rather than useful seams?
- Can existing Java/JDK/library APIs safely replace custom mechanisms?
- Is a class combining unrelated responsibilities?
- Do simplifications preserve exception atomicity and resource ownership?
Use changes in branch complexity, state variables, reviewer comprehension, and performance—not arbitrary line limits—as optimization evidence. Do not refactor passing code solely to hit a line-count target.
Acceptance: No regression in adversarial tests; demonstrably lower complexity on a curated set; changes remain scoped and reversible.
H6 — Better planning efficiency and a first-class “good enough to build” gate [P1]
Problem: Long plans often repeat the prompt, overexplore low-risk details, or postpone implementation. In this benchmark AIR's prolonged planning did not improve correctness; Command Code's own plan was more concise, but hard measured latency comparisons are unavailable.
Proposal: Extend the existing plan/review process with a risk-proportional plan budget and an explicit exit criterion:
- For a tiny fix: no long plan, just the affected invariant/test.
- For moderate work: short design with choices, risks, and acceptance tests.
- For complex systems: detailed ownership/concurrency/failure model, but stop expanding when decisions and testable contracts are clear.
Show a visible reason if planning continues: unresolved API, ambiguous requirement, failed experiment, or evidence gap. Prefer targeted exploration by a cheaper model only when the handoff saves more than it costs. Never trade accuracy for an arbitrary token cap.
Acceptance: Reduce median and P95 “task received → approved plan” latency without worsening critical-defect detection; capture wall time and per-phase token/cost baselines.
H7 — Progress visibility and interruptibility [P1]
Problem: Users can perceive deep planning or review as a stalled agent when they receive no useful status. Existing background panels and task ledgers already expose many details, but the user-facing story can still be improved.
Proposal: Build on existing tools to emit concise milestone-based updates for substantive tasks: current phase, completed evidence, remaining uncertainty, and next verification step. Expose “last useful progress” and the reason for extended analysis. Allow the user to ask for status without disrupting work or restarting exploration.
Avoid raw stream-of-consciousness, repetitive “still thinking” messages, and arbitrary per-second updates. The goal is trustworthy state, not constant chatter.
Acceptance: User can tell whether an agent is exploring, planning, implementing, running tests, or blocked. Status inquiries do not discard work; time-to-first-useful-update and interruption recovery are measured.
H8 — Repository intelligence without repeated full-codebase loading [P1]
Command Code supports MCP, memory and sessions already. The proposed improvement is selective, freshness-aware reuse of verified repository facts:
- Stable module and dependency graph, indexed Java symbols, endpoint → service → DB call chains.
- Provenance: file path, symbol, revision, timestamp/commit, and confidence.
- Automatic invalidation after edits or branch changes, plus cheap partial refresh.
- Source/IDE verification of graph findings before correctness-critical changes.
- Cached exploration summary/handoff shared across sessions and subagents, scoped by repository and revision.
Avoid blindly persisting conjectures as facts, or assuming a graph index resolves Spring reflection/proxy wiring perfectly. Prefer existing IDE/Codebase Memory MCP integrations when available instead of building a new Java parser in the harness.
Acceptance: Repeat queries on large multi-repository Java workspaces use fewer fresh tool calls and fewer input tokens without increasing stale-answer failures. Report cache hit/miss and invalidation accuracy.
H9 — Cache, tool-output and model-routing optimization with quality gates [P1]
Command Code advertises strong provider-side prompt caching and documents context/compaction; these are vendor-reported capabilities, not measured by this benchmark. We recommend exposing auditable per-phase costs: wall time, input/output/cached tokens, number of tool calls, repeated reads, compaction events, and model switches.
Consider cheap exploration + strong planning + fast implementation + independent review, but select it based on actual quality/cost evidence. Stable prompts and resumable sessions should be reused when safe. Avoid re-ingesting entire documentation sets and large test logs; filter output without hiding failing assertions or stack traces.
Acceptance: Benchmarks show equal or better defect detection and lower total cost/latency, not just a superficially high caching percentage. Record whether the provider exposed cached-token counts; do not invent metrics.
H10 — Taste learning must not override reliability contracts [P1]
Command Code's Taste is a differentiator. It should treat style preferences differently from correctness constraints:
- Learn formatting, API idioms, test naming, package organization, and explicit user review preferences.
- Never learn “skip failure tests,” “swallow exceptions,” “ignore shutdown,” or “silence failing checks” as a positive coding taste merely because a user accepted a patch.
- Keep security/ownership/transaction invariants higher priority than learned style.
- Expose why a learned preference was applied, and allow a user to correct or disable it.
Acceptance: Adversarial preference-training scenarios do not weaken previously specified engineering invariants; precedence is deterministic and explainable.
H11 — Controlled quality benchmark program [P0/P1]
The largest improvement opportunity is a repeatable measurement discipline:
- Same model and effort across harnesses for harness-only comparisons (e.g., same DeepSeek release, inference endpoint, reasoning effort, temperature if applicable). Separately benchmark the best model/harness pairing.
- Same frozen task, source snapshot, runtime, permissions and tool access. Track
AGENTS.mddifferences explicitly. - Same hidden evaluator suite with genuine Java 25, JUnit, Spring and jnats; add isolated synthetic fault injection for rare races.
- Multiple independent runs and varied random seeds to estimate variance, not merely the best cherry-picked result.
- Independently written oracles, seeded reference-model testing, targeted PIT mutations, and production-code-only fault injection.
- Separate plan, implementation, and review scores: which bugs are prevented in the plan, introduced in code, detected in review, or fixed after feedback?
- Track actual work: duration by phase, retries, tools, cached/uncached tokens, expense, user interventions, final code complexity and maintainer comprehension.
- Publish failure traces and exact reproduction steps with proprietary hosts, tokens, and paths redacted.
Suggested scoring (proposal, not a historical calculated score)
| Measure | Example operational definition |
|---|---|
| Critical defect rate | Independently reproduced safety/data-loss/transaction/lifecycle bugs, weighted by severity |
| Correctness coverage | Weighted pass rate across the same frozen independent acceptance suite |
| Review effectiveness | Confirmed defects discovered after implementation / defects present before review |
| Test quality | Known mutation kills, risk coverage, deterministic replay, and independent reference-model agreement |
| Maintainability | Reviewers' blinded comprehension tasks, cohesion, mutation-state burden, dependency/runtime footprint |
| Efficiency | Wall-clock P50/P95, total cost, token usage, tool count, user interruptions |
| Honesty of evidence | Claimed-pass versus independently verified-pass discrepancy |
| Reliability | Task completion rate over repeated seeds, crashes, deadlocks, tool-loop stalls and false successes |
A proposed composite score could place 50% on weighted correctness, 20% on independent test effectiveness, 15% on maintainability/simplicity, 10% on time/cost, and 5% on evidence honesty and user visibility. Keep raw dimensions visible; do not let a composite conceal a single critical correctness failure. Treat this weighting as a team decision, not an objective law.
Acceptance: Any “best harness” claim can point to exact task/model/version, immutable evidence, variance, known limitations, and reproducible evaluation commands.
5. Suggested issue / PR breakdown and priorities
| Priority | Candidate issue | Deliverable | Exit condition |
|---|---|---|---|
| P0 | benchmark: common Java 25 harness acceptance suite |
Frozen task + real-dependency tests + independent fault injection | All contenders tested against the same cases; results reproducible |
| P0 | quality: risk-driven verification artifacts |
Lightweight risk ledger + evidence mapping | Critical invariant cannot be silently reported verified without evidence |
| P0 | review: blind adversarial verification |
Fresh-context/different-model review option using original requirements | Seeded planner-assumption defects independently detected |
| P0 | skills: Java lifecycle/transaction/concurrency regression pack |
On-demand tests + fault-injection templates | Known defects in all four submissions detected |
| P1 | planning: budget and exit criterion |
Proportional plan length and phase telemetry | Lower planning latency without lower correctness |
| P1 | ux: progress and blocker visibility |
Milestone state integrated with existing task ledger | User sees meaningful progress and can interrupt/resume |
| P1 | quality: simplify only after passing adversarial checks |
Optional measured simplification pass | Reduced complexity with invariants and performance preserved |
| P1 | context: repository index provenance and invalidation |
Revision-aware facts / MCP tool routing | Fewer repeat reads and no worse stale answers |
| P1 | telemetry: measured cached/uncached tokens and model routes |
Per-phase benchmark export, no undocumented guesses | Repeatable accuracy/cost tradeoff reports |
| P1 | taste: preference vs correctness separation |
Priority/validation rules and regression scenarios | Learned style cannot defeat explicit safety contracts |
| P2 | java: optional JaCoCo/PIT/JFR adapters |
Opt-in existing-tool integration; no JMH requirement | Useful coverage/mutation/performance evidence, modest overhead |
| P2 | docs: reproducible Java benchmark walkthrough |
GitHub-ready instructions and sanitized fixture | Outside maintainer can independently reproduce results |
Recommended implementation sequence
Milestone A — Make the evidence comparable. Freeze the task and evaluator, record harness/model settings, rerun the same acceptance suite across participants, and publish a baseline. Before changing behavior, prove what actually fails under real Java dependencies.
Milestone B — Eliminate blind spots. Add risk-to-test mapping, independent review, and a small library of high-value failure scenarios. Use the discovered cases to measure effectiveness instead of requiring blanket extra agents.
Milestone C — Reduce friction. Improve planning exit criteria, progress transparency, file/context reuse, and concise verified reports. Measure engineering speed and cost after controlling for correctness.
Milestone D — Refine developer experience. Optional simplification, traceable Taste rules and Java-specific tooling; prove they improve outcomes before making them default.
6. What not to do
To preserve Command Code's strengths, please avoid these anti-patterns:
- Do not add mandatory multi-agent approval loops to every change. A typo should stay a typo-sized task.
- Do not maximize unit-test count or JaCoCo percentage at the expense of strong negative scenarios and failure injection.
- Do not treat generated plans or the model's own assertions as an independent truth oracle.
- Do not implement a generic AI “reliability layer” that swallows exceptions, retries unsafe operations, or obscures transaction semantics.
- Do not mandate fixed classes-per-module, lines-of-code, or global worker limits without evidence.
- Do not disable security prompts, weaken permission checks, install unreviewed hooks, or log secrets to improve benchmark throughput.
- Do not confuse a provider's cache-hit marketing metric with measured total engineering cost.
- Do not take AIR's 15/22 score as directly worse than Command Code's 13/14; the suites differ.
- Do not publish client-specific broker addresses, credentials or private code from benchmark archives without permission.
7. Proposed reproducible benchmark protocol
A maintainers' reproducibility package should contain:
benchmark/
README.md # scope, known limitations, use/consent
task.md # frozen original TableNotifier specification
environment.md # versions, models, effort, tools, permissions
fixtures/
java25-spring-jnats/ # sanitized, reproducible fixture (as permitted)
evaluation/
acceptance/ # common hidden/held-out Java tests
fault-injection/ # lifecycle/transaction/concurrency scenarios
reference-models/ # deterministic stateful oracles
runs/
command-code/
plan.md
build-transcript-redacted.jsonl
run-metadata.json
test-results/
reviewed-patch.diff
opencode/
dsh/
codex/
results/
comparative-matrix.md
defects.md
timing-and-cost.csv
For fairness, version the evaluator separately from the task. Pin dependencies; do not let harness-generated tests modify the independent acceptance suite; use a clean workspace per run. Record tool restrictions and human interventions. Ensure a reviewer can reproduce each claimed bug from a focused test without needing confidential production systems.
Representative hidden tests to add
- Spring transaction resource bound but not active (two variants).
afterCommitobserver registers another table notification.- Entire registration fails on a partially created dispatcher; verify ownership and specified fallback policy.
- Worker created in a request with
InheritableThreadLocal/ security context. close()from inside a poll; from another thread while poll blocks; repeated close.Connection.publish()blocks or throws while subscribe/close happen concurrently.- Worker rejects execution mid-registration or mid-notification; listener remains recoverable.
- Burst of ≥1,000 messages during a blocked poll; exactly one necessary subsequent poll, never overlap.
- Many distinct listeners Ă— long callback duration; DB-concurrency safety and fair fallback.
- MeterRegistry shared by many notifier instances; gauge identity and teardown.
- Broker restart/reconnect, permission error, subscribe/unsubscribe after recovery.
ErrorversusRuntimeExceptioncallback failures, with metrics and subsequent polling.- Positive and negative stateful randomized scenarios with a fixed seed and independent reference oracle.
- Tests intentionally mutated to drop a listener, leak a worker, duplicate a publish, or publish before commit; verify the tests fail.
An evaluator should not count a test as a pass solely because an agent can explain why it would work. It must execute or be marked unverified.
8. Questions for Command Code maintainers
- Would you be interested in integrating an independent Java 25 reliability benchmark into the public harness evaluation program?
- Can the existing plan/review system support an optional fresh-context adversarial pass that does not inherit the planner's assumptions?
- Could existing session/task/headless telemetry expose a clean per-phase evidence receipt for builds/tests and the real cached-token/cost breakdown?
- Would you accept a small, on-demand Java fault-injection scenario pack instead of growing global instructions?
- Which current mechanisms best support revision-aware multi-repository knowledge reuse without duplicating MCP or IDE indexing capabilities?
- Is there interest in a quality-constrained simplification pass, where complexity is optimized only after invariant-preserving tests pass?
Final recommendation
Command Code already showed the strongest observable correctness among the first three TableNotifier submissions. Its competitive advantage is not that it writes the least code or always responds fastest; it is that the generated solution survived more important negative scenarios.
To become a demonstrably stronger harness, focus on independent defect discovery, reproducible verification, controlled execution time/cost, and understandable code. The best next step is a common real-dependency acceptance suite and a few small, measured improvements—not a mandatory new multi-agent bureaucracy.
A successful result would be: at least the same correctness, fewer untested production assumptions, better review detection, simpler implementations, more visible progress, and lower measured end-to-end development cost.
How important is this to you?
Important for my workflow
Activity
[-]Reliability, Java Engineering Quality, and Developer-Experience Roadmap[/-][+]Command Code Benchmark Results: Strengths, Weaknesses & Recommendations🙏[/+]magicprinc commentedon Nov 15, 2026
TableNotifier — Java 25 Coding Harness Benchmark
You are a team of two AI agents:
Your work will be independently evaluated by another AI architect using additional tests that are not available to you.
Goal: Build a small, production-quality Java library that replaces wasteful database polling with NATS Core change notifications while preserving periodic polling as a reliability fallback.
Prioritize correctness, simplicity, low resource consumption and maintainability. Do not overengineer.
1. Technology
Required:
io.nats:jnats)spring-tx).propertiesconfiguration where configuration is neededUse modern Java features when appropriate.
No Spring Boot dependency, Docker, embedded NATS broker, external database, Netty, JetStream or additional messaging framework.
The library must work in Spring Boot applications and also be usable without Spring.
2. Existing NATS connection
Applications already maintain their own NATS connections.
TableNotifier must accept
io.nats.client.Connectionas a constructor argument.It must:
Accept a Micrometer
MeterRegistrythrough constructor injection. Do not create a global metric registry.3. NATS subjects
Subject format:
TableNotifier.<database>.<schema>.<table>Example:
TableNotifier.Gate.dbo.ApplicationPropertiesThe Java API accepts table names without the
TableNotifier.prefix.Example:
Gate.dbo.ApplicationPropertiesRequirements:
TableNotifier.A notification indicates that the corresponding database table may have changed. It is not a database event containing changed rows.
4. Public Java API
Design a small, clean API supporting operations equivalent to:
The precise interfaces and implementation classes are Terra's responsibility.
Requirements:
AutoCloseablelifecycle contracts.5. Spring integration
Applications register TableNotifier as a singleton
@Beanand inject it into their DAOs.A typical application already exposes:
TableNotifier must use that existing bean.
The planner must supply a working example of a TableNotifier Spring bean and DAO integration.
Transaction behavior:
Keep Spring integration isolated from the core NATS implementation. Prefer a small optional Spring adapter or decorator.
Do not rely on Spring Boot autoconfiguration.
6. Subscriber polling
Every subscription must perform database polling through its supplied callback.
Rules:
Allow polling interval and debounce delay to be configured. Use 20 seconds as the default fallback interval.
Optional: add a small configurable jitter to reduce simultaneous database queries across application instances.
Do not assume an unlimited number of notifications can be buffered.
7. Failure behavior
Core NATS is at-most-once and does not persist notifications.
The library must remain useful when:
The database remains the source of truth.
A notification failure must not invalidate an already committed database transaction.
Do not introduce JetStream, persistent queues or distributed coordination.
8. Micrometer metrics
Implement appropriate metrics using the provided
MeterRegistry.Include at least:
table_notifier_published_totaltable_notifier_publish_errors_totaltable_notifier_received_totaltable_notifier_polls_totaltable_notifier_poll_durationtable_notifier_poll_errors_totaltable_notifier_active_subscriptionstable_notifier_coalesced_totalUse bounded, low-cardinality tags such as
reason=notification|fallback|initial.Avoid unbounded tags containing arbitrary table names, connection addresses, user IDs or exception messages.
Avoid unnecessary per-event allocations and duplicate timers/counters.
Do not close the caller-provided
MeterRegistry.Document the semantics of each metric.
9. Manual notifications for DevOps
Mandatory: Provide
docs/MANUAL_NOTIFICATIONS.md.Our DevOps engineers sometimes modify database tables manually through SQL.
They must be able to notify applications afterward using the official NATS CLI.
The documentation must include:
Gate.dbo.ApplicationProperties.The documented CLI notification must be compatible with the Java subscriber without requiring a custom payload format.
Optional bonus: Python utility
Implement a small
send_notification.pyusing the official Python NATS client.Example interface:
It should support external NATS connection configuration and proper error handling.
This Python utility is optional. It must not delay completion of the Java implementation.
10. Testing
Use JUnit 5 and Google Truth.
Write deterministic, meaningful tests.
Mandatory scenarios:
Use controlled clocks, executors, latches or other deterministic techniques where practical.
Tests must not require Docker, a running corporate NATS server or access to a real database.
An optional separate integration-test suite against a local NATS server is welcome.
Do not modify tests just to hide production defects.
11. Architecture constraints
Prefer proven library APIs to handwritten infrastructure.
12. Two-phase agent workflow
Phase A — Terra: Planning
First inspect the workspace and verify the relevant library APIs and compatibility.
Create:
docs/PLAN.md— architecture, modules, public API, concurrency model, lifecycle and error handling.docs/TEST_PLAN.md— tests, invariants and difficult edge cases.docs/MANUAL_NOTIFICATIONS.md— exact DevOps CLI procedure, with independently checked command syntax.Identify the critical correctness risks before implementation.
Choose the simplest architecture that fulfills the requirements.
The planning phase must not produce the production implementation.
Phase B — DeepSeek: Implementation
Read Terra's documents before writing production code.
Then:
FINAL_REPORT.md.The final report must contain actual commands executed, test results, architectural decisions, known limitations and incomplete requirements.
Do not claim tests passed unless they were executed.
Do not silently omit requirements.
If a design change is necessary, document its reason and update the corresponding plan.
13. Deliverables
@Beanand DAO usage examples.14. Evaluation
The evaluator will independently inspect the implementation and run additional tests.
Your own test results are not sufficient evidence of correctness.
Evaluation priorities:
Finish a working, verifiable implementation rather than adding unnecessary features.
Proceed autonomously using Terra for planning and DeepSeek for implementation.
magicprinc commentedon Nov 15, 2026
It was interesting to see, that OpenAI Codex was ~ Nr.3
GPT-6.1 Sol XHIGH has selected Command Code as harness Nr.1
TL;DR
Feature Request: Make Command Code More Reliable at Building Correct Software
Quality Mode: Independent Verification, Risk-Driven Tests, and Evidence-Based DeliverySummary
Command Code performed well in our independent coding-agent exercise. We would like to help it become more consistently reliable on stateful, asynchronous, failure-prone software, regardless of programming language or framework.
Our main request is an optional, end-to-end correctness workflow that makes agents actively challenge their own design and implementation, rather than merely verifying that the generated code passes the tests they also wrote.
This is a request for harness orchestration and verification behavior, not for Command Code maintainers to fix bugs in generated application code.
Evidence and limitations
We asked coding agents to implement TableNotifier: a relatively small component involving asynchronous notifications, transactions, retries/fallback behavior, callbacks, concurrent state changes, and resource cleanup. All these are common systems-engineering problems, not Java-only problems.
In our original 14 shared, source-derived independent behavioral checks, submissions scored:
Command Code was the strongest in this particular exercise, but this does not establish that the harness itself is intrinsically superior: the evaluation was not fully controlled for configurations, models, prompts, or review workflows; the original full Java 25/real-service suites were not independently reproduced in this audit. A fourth participant (Codex/AIR) was evaluated with a different 22-check suite, so its score is not directly comparable.
The most useful observation is not the ranking: agents can generate a substantial test suite and still miss high-impact failure paths. This points to an opportunity for better harness-level verification.
Requested improvements (prioritized)
P0 — 1. Optional independent adversarial reviewer
Request: Add or strengthen a verification mode in which a reviewer operates with a fresh context (and optionally a different model) after implementation.
The reviewer should:
Why: A planner, implementer, and reviewer sharing one incorrect assumption can reinforce each other's mistake. An independent review should find errors the original plan never anticipated.
Acceptance: In a seeded-defect evaluation, a fresh reviewer detects more genuine defects than a plan-following review, with tracked false positives, runtime, and token cost. It remains opt-in for routine edits.
P0 — 2. Requirements → invariants → negative tests → evidence
Request: For high-risk tasks, maintain a lightweight, structured contract/verification map:
Requirement → invariant → likely failure scenario → test/proof → observed result → confidenceFor example:
The harness should automatically surface unverified critical invariants before declaring completion; it should not silently call a requirement “covered” merely because some test passed.
Why: Counting test cases or lines of test code is a weak proxy for correctness. We need risk coverage, not test volume.
Acceptance: Critical uncovered invariants are visible in the final report; a test result is attached to each verified claim, or the item is explicitly marked unverified.
P0 — 3. Risk-based failure and concurrency scenario generation
Request: Give the reviewer and test-writing agent a small, language-neutral set of failure-pattern prompts or skills, selected only when relevant to the code being changed:
These should become concrete executable tests where possible, using the project's existing test stack. Do not load an enormous checklist into every prompt.
Why: Most expensive production bugs occur outside the happy path; the harness can systematically ask “What happens if this operation fails here?”
Acceptance: Fault-pattern checks catch seeded errors across at least two languages/platforms without imposing substantial overhead on small tasks.
P0 — 4. Evidence-based completion, not self-reported success
Request: Provide a concise verification receipt automatically assembled from tool results:
The receipt should detect suspicious cases such as “all tests passed” despite filtered-out tests, a missing test task, ignored exit codes, or an unexecuted integration suite.
Why: An agent-written summary is not independent evidence. Developers need to trust the observed execution, not the prose.
Acceptance: The final status can be reconstructed from recorded tool outputs and does not silently overstate test coverage.
P1 — 5. A verification loop with a clear stop condition
Request: Build on Command Code's existing planning, agents, hooks and task tracking to support an opt-in sequence:
Plan → Implement → Test → Independent risk review → Reproduce → Fix → Re-test → Evidence reportRequirements:
Why: The value is in closing the feedback loop—not in adding more agents or lengthy reviews by default.
Acceptance: Higher critical-defect detection with measured, bounded cost; no uncontrolled agent loops.
P1 — 6. Simplicity and maintainability checks after correctness
Request: Add an optional architecture-simplification review once correctness checks pass. Look for:
Do not optimize for line count alone. A shorter solution that loses a wakeup or mishandles cleanup is worse.
Why: Strong agents may handle edge cases correctly by accumulating complex state machines. We want the simplest implementation that preserves all invariants.
Acceptance: Simplification reduces objectively inspectable complexity on selected tasks while all previously passing adversarial tests remain green.
P1 — 7. Risk-proportional planning and visible progress
Request: Improve the existing planning workflow rather than add a new planning system:
Why: More planning tokens do not necessarily improve correctness. Long silent planning and repeated exploration increase cost and reduce confidence.
Acceptance: Reduced time-to-first-useful-result and planning latency, with no increase in critical defects; status inquiries do not reset or duplicate work.
P1 — 8. Freshness-aware repository understanding across sessions and agents
Request: Preserve verified, reusable repository knowledge without forcing every new session to reread a large codebase:
Why: Repeated cold-start exploration wastes tokens; stale project “memory” can be even worse.
Acceptance: Fewer repeated reads and lower input cost on multi-repository tasks, with measured stale-answer rate no worse than baseline.
P2 — 9. Cost and quality telemetry, including model handoffs
Request: Export per-phase execution metrics: wall time, tool calls, repeated reads, tokens (including provider-reported cached tokens where available), compaction/model switches, test runs, and review findings.
Why: We want to optimize cost per verified correct result, not tokens alone. Cheap implementation followed by expensive debugging may cost more overall.
Acceptance: Users can compare workflows and model-routing strategies without inventing caching or timing estimates.
P2 — 10. A reproducible harness-quality benchmark
Request: Maintain a small public fixture suite that evaluates harness behaviors separately from model ability:
Why: A single complex submission proves that one workflow produced one result; it cannot isolate which harness feature contributed to success or failure.
Acceptance: Each claimed improvement is backed by reproducible, comparable results, not only a vendor benchmark or passing agent-generated tests.
How these requests generalize beyond Java
The benchmark's interesting problems are universal:
These apply to TypeScript services, Go daemons, Rust infrastructure, Python workers, distributed systems and desktop applications just as much as to Java.
Recommended minimal first increment
If the team can implement only one improvement, start with a configurable Independent Verification quality mode using existing Command Code agent infrastructure:
Keep it opt-in and bounded. This is likely to offer more practical value than introducing another generic coding agent or expanding global prompts.
What we are not requesting
Success criterion: The same models, given the same requirements and budget, should produce fewer independently reproducible severe defects with transparent evidence and acceptable runtime/cost.
vipulgupta2048 commentedon Nov 17, 2026
Thanks @magicprinc for benchmarking Command Code. Will review the report and figure out what improvements we can add in our roadmap.