Skip to content

Command Code Benchmark Results: Strengths, Weaknesses & Recommendations🙏 #1146

Description

@magicprinc

Command Code: Reliability, Java Engineering Quality, and Developer-Experience Roadmap

GitHub issue / discussion proposal for Command Code
Evidence: Independent TableNotifier — Java 25 coding-agent benchmark, 15 November 2026
Priority: Correctness first; then test effectiveness, simplicity, responsiveness, and cost
Audience: Command Code maintainers, harness engineers, and contributors
Status: External engineering feedback. Not a claim that every weakness in generated code is a defect in the harness.

Executive summary

Command Code produced the strongest original TableNotifier implementation in our independent comparison. It passed 13/14 shared source-derived behavioral checks, versus 11/14 for OpenCode V2 and 9/14 for DeepSeek Harness (DSH). It demonstrated particularly strong handling of Spring transaction completion, worker ownership, callback isolation, and concurrency races.

That is a meaningful result, but not a definitive general-purpose harness ranking. The original Java 25/real-dependency suites were not independently rerun in the audit environment; the independent checks used Java 21-compiled, source-derived implementations and controlled NATS/Spring/Micrometer stand-ins. Another participant, AIR (Codex), was examined with an expanded and different 22-check suite, so its score is not numerically comparable. The harness versions, model conditions, instruction sets, and full time/cost traces were not controlled identically.

Our proposal: Turn Command Code's demonstrated correctness advantage into a more consistently verifiable, efficient, explainable, and maintainable development workflow. The highest-value improvements are not “add more AI agents” or “produce longer plans”; they are:

  1. Risk-driven verification: translate contracts into adversarial tests, then deliberately challenge assumptions the planner and implementation share.
  2. A production-failure scenario library: lifecycle, transactions, NATS/network outages, exceptions, races, failure atomicity, and thread-context isolation.
  3. Better code-shape discipline: achieve equivalent guarantees with fewer states, fewer defensive layers, and less incidental complexity.
  4. Transparent execution: show live phase, progress, verification evidence, and the reasons for delay or additional tool calls.
  5. Repeatable engineering benchmarks: compare correctness, severity of defects, planning/implementation time, tokens, cache use, review effectiveness, and maintainability with identical models and test suites.

Request to maintainers: Please consider this as a prioritized collection of engineering quality improvements and a proposal for a reproducible benchmark suite, not a request to implement every feature by default.


1. Benchmark context and evidence standard

Task

Build TableNotifier, a small production-oriented Java 25 library that uses NATS Core table-change hints to trigger polling, with independent periodic polling as a reliability fallback. The requirements emphasized:

  • An injected, application-owned io.nats.client.Connection; never close or reconfigure it.
  • Subject validation and multicast delivery (no queue groups).
  • Independent listeners, no overlapping callback execution, burst coalescing, and follow-up polling when a notification arrives during a poll.
  • Spring transaction integration: publish after successful commit, suppress on rollback, and deduplicate per transaction.
  • Clean lifecycle; no dedicated platform thread per listener; no unbounded per-event queues.
  • Micrometer metrics, manual NATS CLI documentation, and strong deterministic tests.
  • Java 25, Gradle, Lombok, jnats, Spring TX, JUnit 5, and Google Truth.

Participants:

Participant Harness Observed workflow difference
Command Code (cmdc) Command Code Original Terra/DeepSeek task; planning and review artifacts present
OpenCode V2 OpenCode Original Terra/DeepSeek task
DSH DeepSeek Harness Original Terra/DeepSeek task; user reported a particularly fast/low-cost experience
AIR Codex Newer AGENTS.md; review by GPT-6.1 Sol, not Terra; user reported unusually long planning and repeated requests for progress

Interpretation: The benchmark evaluates end-to-end outputs of differently configured workflows, not isolated intrinsic abilities of each harness. A defect in a generated Java library may reflect a model decision, misleading plan, missing test, execution policy, or harness/tooling design. Establish root cause with controlled ablations before attributing it to Command Code.

Verification evidence and limitations

Evidence Status What it establishes
Source inspection of four submission archives Performed Architecture, implementation patterns, submitted tests, and declared design decisions
Same 14 source-derived behavioral probes on DSH/OpenCode/Command Code Executed Comparable isolated behavior for those three implementations
AIR's 22 isolated probes Executed separately Additional AIR-specific failure modes and strengths; not a common denominator
Java 25 original Gradle/JUnit suite independently rerun Not performed Original integration correctness remains unverified by this independent audit
Genuine Spring, Micrometer, jnats and broker restart tests by independent evaluator Not performed Stand-ins cannot validate every library-specific lifecycle or reconnection behavior
Command Code's own claimed 91 tests, one optional real-NATS round trip Submission report, not independently reproduced Evidence of engineering effort and recorded self-verification, not an independent pass certificate
Measured same-model time, tokens and cost across all harnesses Unavailable Do not claim a measured speed or price ranking from this benchmark

The benchmark intentionally did not penalize any participant for missing Gradle Wrapper binaries: the user had separately instructed participants to use installed Gradle 9.9.1 and minimize archive size.

Original shared 14-check results

Independent scenario Command Code OpenCode V2 DSH
Basic multi-listener delivery + fallback PASS PASS PASS
1,000 notification burst; callback non-overlap PASS PASS PASS
Missed notification repaired by periodic polling PASS PASS PASS
Notifier shutdown releases owned resources; borrowed NATS stays open PASS PASS FAIL
Spring commit/rollback/same-table deduplication PASS PASS PASS
Additional notification registered from another afterCommit callback PASS FAIL PASS
Callback AssertionError does not prevent later polls PASS PASS PASS
Bound resource without actual Spring transaction PASS PASS FAIL
AssertionError counted as poll failure PASS PASS FAIL
Worker avoids inheriting request InheritableThreadLocal PASS FAIL FAIL
createDispatcher() fails, but fallback-only service starts FAIL FAIL FAIL
12 concurrent clients Ă— 150 operations PASS PASS PASS
Notifier closed from within its own callback PASS PASS PASS
Subscription closed during long-running callback PASS PASS PASS
Total 13/14 11/14 9/14

Severity matters: A numerical “14-check score” is only a useful first indicator. These tests vary in business impact, and the strict dispatcher-failure check is partly a requirement-interpretation question: does “poll during NATS outage” also mandate successful construction if creation of the jnats dispatcher itself fails? The desired contract must be made explicit and tested against real jnats.

AIR / Codex: separate expanded evaluation

AIR passed 15/22 isolated probes and 4/4 main concurrent-stress scenarios; seven probe failures reflected six distinct concerns, including two manifestations of one Spring error. Its confirmed concerns were: false Spring transaction detection when a resource is bound without a transaction; inherited request thread context; callback self-interruption; blocked publish holding a lifecycle lock; dispatcher-startup failure; and a missing notifyAfterCommit method on the public interface. This cannot be subtracted from Command Code's 13/14 or used to declare a seven-defect difference.

AIR had extensive planning and test documentation but still missed fundamental assumptions. The lesson for all harnesses is that plan length and test count do not substitute for independent risk verification.

Code-shape and self-reported testing comparison

Measure Command Code OpenCode V2 DSH AIR
Production Java files 8 5 7 15 (incl. two package-info.java)
Production Java lines, incl. comments/blank lines 1,297 904 1,008 1,330
Largest production Java class 336 557 267 547
Separate core and Spring modules Yes Yes No Yes
Tests claimed in final report 91 (one skipped) 98 48 (47 pass, one skipped) 103 deterministic

Line counts measure size, not quality. Command Code's larger solution avoided several defects that the smaller solutions exhibited; its next opportunity is to retain those invariants with less incidental complexity. Similarly, 91 passing tests do not establish a low defect rate unless the tests probe the correct failure modes.


2. What Command Code did exceptionally well — preserve these strengths

  1. Correct resource ownership. It separates borrowed NATS Connection/MeterRegistry from owned dispatcher, scheduler and callback executor. Tests exercise teardown and leave caller-owned resources alive.
  2. Spring transaction reasoning. It uses a transaction synchronization, publishes in afterCompletion(STATUS_COMMITTED), handles REQUIRES_NEW, avoids interpreting an arbitrary bound resource as an active transaction, and prevents a callback failure from undoing a committed DB transaction.
  3. Thread-context isolation. Worker creation explicitly disables inheritable thread-local capture. This prevents a leak independently detected in OpenCode, DSH, and AIR.
  4. Callback reentrancy/lifecycle. Self-close and closure during active processing passed the shared suite. This is exactly the sort of fault that ordinary happy-path tests miss.
  5. Bounded notification state. Notifications are coalesced into fixed per-listener state rather than queued one by one.
  6. Meaningful initial tests. Concurrency, error recovery, transaction boundaries and resource cleanup were addressed in submitted tests; the plan anticipated several defects competitors later exhibited.
  7. Appropriately qualified release reporting. The submission disclosed reconnect and soak-test gaps, a gauge warning, and compatibility limits instead of claiming unconditional production readiness.

Guardrail: Any simplification, faster execution strategy, model routing, or new automatic repair feature must preserve these properties. Do not exchange strong failure semantics for prettier code or lower latency.


3. Concrete weaknesses in Command Code's generated TableNotifier

Each item distinguishes direct evidence from a proposal. File paths are relative to the archived cmdc/ submission. They describe the generated project, not necessarily defects in Command Code's own repository.

C1 — Dispatcher creation failure prevents fallback-only startup

  • Evidence: Independently reproduced failure in the shared 14-check audit (1 of 14 failures for Command Code).
  • Code: table-notifier-core/src/main/java/io/tablenotifier/NatsTableNotifier.java, constructor around lines 93–117. It calls connection.createDispatcher(...) and rethrows on RuntimeException, shutting down its owned executors.
  • Impact: Under the strict “database polling continues despite unavailable NATS” interpretation, an unavailable NATS dispatcher at startup prevents the notifier from existing and starting its polling listeners.
  • Important qualification: The fail-fast constructor is a defensible API choice if dispatcher-creation failure is explicitly fatal. Disconnect after successful construction is a different state. Do not silently hide a broken connection or security/permission error as mere unavailability.
  • Recommendation: Specify a decision table: disconnected existing connection; dispatcher creation failure; subscription failure; reconnection; permission denial; permanent misconfiguration. Offer an opt-in degraded/fallback-only startup policy if the contract demands it, with bounded recovery, visible health/metrics and no busy waiting. Alternatively document fail-fast as the supported contract and update acceptance requirements accordingly.
  • Acceptance criteria: Independent initial/fallback polls proceed during an authorized transient startup failure; NATS subscription recovers when possible; close releases every owned resource; permanent errors remain observable; real jnats semantics tested.

C2 — Duplicate Micrometer gauge registration warning

  • Evidence: The submission's own docs/FINAL_REPORT.md explicitly reports WARNING: This Gauge has been already registered once per registry. This is self-reported, not separately reproduced.
  • Code: table-notifier-core/src/main/java/io/tablenotifier/NotifierMetrics.java, activeState(...), around lines 62–80, registers a Gauge on each new metrics facade, with registry-specific weak-map bookkeeping and a strong gauge state reference.
  • Risk: Noisy logs, unnecessary registry bookkeeping, complicated ownership semantics under multiple notifiers sharing one registry; possible ambiguity if registries/meters are removed/reused.
  • Recommendation: Define gauge identity, aggregation and lifetime explicitly. Prefer deterministic, registry-scoped registration with no duplicate warnings; use registry APIs consistently and test multi-notifier add/remove/GC/registry lifecycle. Do not add a home-grown metrics framework.
  • Acceptance criteria: 1, 2, and many concurrent notifier instances sharing a registry show the correct listener total; no duplicate registration warnings; no stale counts after close; bounded memory retention.

C3 — Spring adapter has an easy-to-misuse constructor

  • Evidence: Declared by the submission's own final report; confirmed in source.
  • Code: table-notifier-spring/src/main/java/io/tablenotifier/spring/SpringTableNotifier.java, constructors around 38–56. The two-argument overload rejects an AbstractPlatformTransactionManager with SYNCHRONIZATION_NEVER; the one-argument overload cannot validate this configuration.
  • Impact: A consumer can choose the less-safe constructor and unknowingly get post-commit semantics that are impossible to enforce with synchronization disabled. The implementation currently logs and skips publication when an actual transaction has no synchronization: this preserves transaction safety but makes the missing hint an operational concern.
  • Recommendation: Prefer one clearly safe documented bean-wiring path; expose explicit validation/fail-fast policy for supported transaction managers. If retaining the one-argument path, make its limitation unavoidable in API documentation and diagnostics. Avoid global static transaction-state heuristics.
  • Acceptance criteria: A runnable Spring bean example rejects unsupported settings before use; correct behavior for no transaction, bound-resource/no transaction, sync-only scope, rollback, commit, REQUIRES_NEW, and active transaction with sync disabled.

C4 — Reconnection, broker restart and sustained load remain unverified

  • Evidence: Explicit gap in docs/FINAL_REPORT.md: no broker-restart recovery test, multi-instance soak, or production fleet validation. The optional live-NATS round trip claimed by the submission is only one scenario.
  • Risk: A local fake connection cannot prove real dispatcher/subscription recovery after reconnect, authentication failure or connection churn. Likewise a single 1,000-message burst cannot establish sustained polling-capacity or jitter behavior.
  • Recommendation: Add reproducible, separate opt-in integration tests with a disposable local NATS server: disconnect/reconnect, restart, listener added during outage, publish failures, simultaneous instance fanout, and observable fallback polling. Keep the default unit suite free of broker dependencies.
  • Acceptance criteria: Broker restart tests reliably reproduce and assert recovery; no accidental closure of the injected connection; fallback continues throughout outage; test results and versions are archived.

C5 — Potential unbounded cross-listener callback concurrency

  • Evidence: Design risk, not a reproduced defect. NatsTableNotifier.newWorkerExecutor() uses Executors.newThreadPerTaskExecutor(Thread.ofVirtual()...) (around 129–134). Per-listener non-overlap is enforced by PollingSubscription, but a large number of different listeners may poll simultaneously. Virtual threads reduce thread cost, not database capacity use.
  • Impact: Database connection-pool contention, callback latency inflation, and missed nominal fallback cadence under many slow listeners. No benchmark demonstrated safe capacity for thousands of independent listeners.
  • Recommendation: First measure workload-specific capacity: listener count Ă— poll duration / target interval, real DB pool size, and notification bursts. If necessary introduce optional bounded global admission with fairness and explicit saturation metrics. Preserve the no-lost-follow-up and no-overlap guarantees. Do not insert a fixed small pool without capacity analysis.
  • Acceptance criteria: Repeatable workload curves for active listeners, simultaneous callbacks, poll latency, delayed fallback, memory and cleanup; no unbounded callback backlog; no starvation; minimal overhead at ordinary load.

C6 — Defensive state-machine complexity can be reduced carefully

  • Evidence: NatsTableNotifier.java is 336 lines; PollingSubscription.java is 324 lines, with multiple synchronized flags (closed, running, admissionRequested, pendingReasons, notificationPending) and three scheduled-future fields. This is a maintainability concern, not evidence the code is incorrect.
  • Impact: Future contributors must reason about a fairly large product of possible states. Over-simplification could reintroduce lost wakeups or concurrency bugs.
  • Recommendation: Document the finite state transitions and invariants first; write state-transition and fault-injection tests; then remove only state shown redundant by a proof or executable reference model. Keep cohesive classes and avoid an extra abstraction for every field.
  • Acceptance criteria: Same or better adversarial pass rate; fewer independent mutable states or less control-flow complexity; no measurable throughput regression; maintainers can explain the state machine in one short diagram.

C7 — Exception reporting may be unnecessarily noisy

  • Evidence: The earlier independent audit notes an AssertionError correctly counted in failure metrics, with recovery of later polls, but possibly rethrown as an uncaught virtual-thread error. This is an observed/logging concern, not a demonstrated availability failure.
  • Recommendation: Distinguish cancellation/interruption, ordinary callback failure, and JVM-level Error; decide a consistent logging policy that avoids both silent failures and duplicate stack traces. Preserve Error metrics and future fallback polls; never silently swallow fatal JVM failures merely for a green test.
  • Acceptance criteria: Every error path is visible exactly as intended; AssertionError tests show recovery and counted failures; no excessive duplicate logs under repeated failures.

C8 — Compatibility, packaging and reproducible delivery need hardening

  • Evidence: Self-reported: only Spring 7.0.10 verified; no Spring 6/Boot 3 run; no CI, sources/Javadoc JAR packaging, license, or publishing metadata in the submission. These were not all mandatory to satisfy the original benchmark.
  • Recommendation: Add a small supported-version compatibility matrix, CI verification on Java 25 and supported Spring generations, publishing metadata and reproducible artifact evidence. Keep third-party infrastructure opt-in.
  • Acceptance criteria: Clearly stated compatibility; green documented CI matrix; generated release artifacts with sources/Javadoc; no surprise dependency inflation.

4. Command Code harness roadmap (proposals, not confirmed product defects)

Current Command Code documentation already describes Plan Mode and review/approval, planning/implementation model lanes, task ledgers/background agents, hooks, MCP, session checkpoints, context compaction, headless JSON output, and Taste learning. This roadmap should build on those primitives, not propose them again as missing features. See the official Plan Mode, Background Tasks, Context, Hooks, Sessions, and Headless references.

H1 — Risk-to-test traceability as an optional quality mode [P0]

Problem: Agents can claim “comprehensive tests” while never testing the exact assumption that will fail in production. AIR's incorrect Spring transaction classifier illustrates this failure mode; DSH's executor ownership gap demonstrates the same general problem. Command Code did better here, but can systematize it.

Proposal: For substantial tasks, maintain a concise machine-readable or Markdown risk ledger:

Field Example
Requirement/invariant No publishing before successful Spring commit
Negative scenario Resource bound, no real transaction
Failure injection Synchronization unavailable; registration throws
Proof/test Named test + outcome + actual command
Confidence Real dependency / controlled fake / static inspection / unverified
Owner and status Planner / implementer / reviewer, open or verified

The harness should prevent accidental omission in its final report, not prevent a developer from making partial progress. It should surface uncovered high-risk rows instead of padding tests with low-value cases.

Acceptance: In a seeded set of Java tasks, the risk ledger increases the proportion of independently found critical defects and shrinks “claimed verified / actually untested” gaps, with acceptable token/time overhead. Keep it off for trivial edits.

H2 — Independent adversarial review that challenges the plan itself [P0]

Problem: If the same plan informs implementation and review, an incorrect assumption may receive repeated approval. Plan compliance is not requirement correctness.

Proposal: A review mode that reads (a) the original user task, (b) current source/diff, and (c) executed test evidence, then challenges the plan rather than using it as the oracle. Ideally place a reviewer in a fresh context or different model for correctness-critical work; selectively withhold plan commentary until after the reviewer has independently identified the likely failure modes.

Require explicit risk checks for transaction lifecycle, resource ownership, callback failures, concurrency, reentrancy, and shutdown whenever those features exist. Review should produce reproducible counterexamples (failing tests), not merely checklist scores or assertions that code looks good.

Acceptance: Blind reviewer finds seeded bugs that plan-following reviewers miss; false-positive rates and review cost are measured. No mandatory second agent for low-risk changes. Respect permissions for any executed tests.

H3 — A Java reliability scenario pack, not a bulky global prompt [P0]

Deliver a versioned, on-demand Java verification skill containing minimal reproducible scenarios for:

  • Atomic updates with injected hashCode/equals/callback failures.
  • AutoCloseable ownership and partial initialization failure.
  • Callback-triggered shutdown, external shutdown, races with subscribe/unsubscribe.
  • Event arriving during active processing, burst coalescing, fairness, bounded queues.
  • InheritableThreadLocal and request/MDC context isolation.
  • Spring commit/rollback, REQUIRES_NEW, synchronization-only scope, synchronization disabled, and late afterCommit registrations.
  • NATS disconnect/reconnect, dispatcher registration failures, lost messages, shared connection ownership.
  • Metrics consistency, Error versus Exception, duplicate meter registration.
  • Deterministic seeded stateful/reference-model tests for tricky state machines.

Implementation choice: Skills or verified test templates (JUnit + Truth + Mockito as appropriate). Do not load the entire scenario catalog into every prompt, require Docker by default, or prescribe one state-machine implementation.

Acceptance: The pack detects all known TableNotifier defects across all four archived submissions with original dependencies, with per-case failure logs. Avoid changing production code merely to satisfy the pack.

H4 — Verified evidence, not “tests passed” prose [P0]

Use existing execution and headless JSON capabilities to produce a minimal verification receipt for important tasks:

# Illustrative output schema, not a documented Command Code API
checks:
  - command: "gradle test"
    exit_code: 0
    duration_ms: 32100
    result: "91 tests; 1 skipped"
    evidence: "build/test-results/test/"
  - command: "gradle integrationTest"
    status: "not_run"
    reason: "No authorized broker available"
known_risks:
  - "NATS broker-restart behavior not verified"

Important: the system should read actual process exit codes and test XML, flag skipped or filtered tests, and distinguish test claims from observed execution. If a test is not runnable, it should say unverified rather than silently count it as passed.

Acceptance: A machine-readable receipt can be reconstructed from the session log; false success claims are flagged; receipts remain concise and redact credentials/hostnames.

H5 — Preserve quality while reducing implementation complexity [P1]

Problem: High-correctness implementations can accumulate defensive bookkeeping, duplicate adapters, and large control-flow graphs. Command Code's TableNotifier is more complex than DSH's, though observably more correct.

Proposal: Add a single bounded simplification pass after correctness verification. Ask:

  • Which state variables are necessary for the stated invariants?
  • Which abstractions exist only for tests rather than useful seams?
  • Can existing Java/JDK/library APIs safely replace custom mechanisms?
  • Is a class combining unrelated responsibilities?
  • Do simplifications preserve exception atomicity and resource ownership?

Use changes in branch complexity, state variables, reviewer comprehension, and performance—not arbitrary line limits—as optimization evidence. Do not refactor passing code solely to hit a line-count target.

Acceptance: No regression in adversarial tests; demonstrably lower complexity on a curated set; changes remain scoped and reversible.

H6 — Better planning efficiency and a first-class “good enough to build” gate [P1]

Problem: Long plans often repeat the prompt, overexplore low-risk details, or postpone implementation. In this benchmark AIR's prolonged planning did not improve correctness; Command Code's own plan was more concise, but hard measured latency comparisons are unavailable.

Proposal: Extend the existing plan/review process with a risk-proportional plan budget and an explicit exit criterion:

  • For a tiny fix: no long plan, just the affected invariant/test.
  • For moderate work: short design with choices, risks, and acceptance tests.
  • For complex systems: detailed ownership/concurrency/failure model, but stop expanding when decisions and testable contracts are clear.

Show a visible reason if planning continues: unresolved API, ambiguous requirement, failed experiment, or evidence gap. Prefer targeted exploration by a cheaper model only when the handoff saves more than it costs. Never trade accuracy for an arbitrary token cap.

Acceptance: Reduce median and P95 “task received → approved plan” latency without worsening critical-defect detection; capture wall time and per-phase token/cost baselines.

H7 — Progress visibility and interruptibility [P1]

Problem: Users can perceive deep planning or review as a stalled agent when they receive no useful status. Existing background panels and task ledgers already expose many details, but the user-facing story can still be improved.

Proposal: Build on existing tools to emit concise milestone-based updates for substantive tasks: current phase, completed evidence, remaining uncertainty, and next verification step. Expose “last useful progress” and the reason for extended analysis. Allow the user to ask for status without disrupting work or restarting exploration.

Avoid raw stream-of-consciousness, repetitive “still thinking” messages, and arbitrary per-second updates. The goal is trustworthy state, not constant chatter.

Acceptance: User can tell whether an agent is exploring, planning, implementing, running tests, or blocked. Status inquiries do not discard work; time-to-first-useful-update and interruption recovery are measured.

H8 — Repository intelligence without repeated full-codebase loading [P1]

Command Code supports MCP, memory and sessions already. The proposed improvement is selective, freshness-aware reuse of verified repository facts:

  • Stable module and dependency graph, indexed Java symbols, endpoint → service → DB call chains.
  • Provenance: file path, symbol, revision, timestamp/commit, and confidence.
  • Automatic invalidation after edits or branch changes, plus cheap partial refresh.
  • Source/IDE verification of graph findings before correctness-critical changes.
  • Cached exploration summary/handoff shared across sessions and subagents, scoped by repository and revision.

Avoid blindly persisting conjectures as facts, or assuming a graph index resolves Spring reflection/proxy wiring perfectly. Prefer existing IDE/Codebase Memory MCP integrations when available instead of building a new Java parser in the harness.

Acceptance: Repeat queries on large multi-repository Java workspaces use fewer fresh tool calls and fewer input tokens without increasing stale-answer failures. Report cache hit/miss and invalidation accuracy.

H9 — Cache, tool-output and model-routing optimization with quality gates [P1]

Command Code advertises strong provider-side prompt caching and documents context/compaction; these are vendor-reported capabilities, not measured by this benchmark. We recommend exposing auditable per-phase costs: wall time, input/output/cached tokens, number of tool calls, repeated reads, compaction events, and model switches.

Consider cheap exploration + strong planning + fast implementation + independent review, but select it based on actual quality/cost evidence. Stable prompts and resumable sessions should be reused when safe. Avoid re-ingesting entire documentation sets and large test logs; filter output without hiding failing assertions or stack traces.

Acceptance: Benchmarks show equal or better defect detection and lower total cost/latency, not just a superficially high caching percentage. Record whether the provider exposed cached-token counts; do not invent metrics.

H10 — Taste learning must not override reliability contracts [P1]

Command Code's Taste is a differentiator. It should treat style preferences differently from correctness constraints:

  • Learn formatting, API idioms, test naming, package organization, and explicit user review preferences.
  • Never learn “skip failure tests,” “swallow exceptions,” “ignore shutdown,” or “silence failing checks” as a positive coding taste merely because a user accepted a patch.
  • Keep security/ownership/transaction invariants higher priority than learned style.
  • Expose why a learned preference was applied, and allow a user to correct or disable it.

Acceptance: Adversarial preference-training scenarios do not weaken previously specified engineering invariants; precedence is deterministic and explainable.

H11 — Controlled quality benchmark program [P0/P1]

The largest improvement opportunity is a repeatable measurement discipline:

  1. Same model and effort across harnesses for harness-only comparisons (e.g., same DeepSeek release, inference endpoint, reasoning effort, temperature if applicable). Separately benchmark the best model/harness pairing.
  2. Same frozen task, source snapshot, runtime, permissions and tool access. Track AGENTS.md differences explicitly.
  3. Same hidden evaluator suite with genuine Java 25, JUnit, Spring and jnats; add isolated synthetic fault injection for rare races.
  4. Multiple independent runs and varied random seeds to estimate variance, not merely the best cherry-picked result.
  5. Independently written oracles, seeded reference-model testing, targeted PIT mutations, and production-code-only fault injection.
  6. Separate plan, implementation, and review scores: which bugs are prevented in the plan, introduced in code, detected in review, or fixed after feedback?
  7. Track actual work: duration by phase, retries, tools, cached/uncached tokens, expense, user interventions, final code complexity and maintainer comprehension.
  8. Publish failure traces and exact reproduction steps with proprietary hosts, tokens, and paths redacted.

Suggested scoring (proposal, not a historical calculated score)

Measure Example operational definition
Critical defect rate Independently reproduced safety/data-loss/transaction/lifecycle bugs, weighted by severity
Correctness coverage Weighted pass rate across the same frozen independent acceptance suite
Review effectiveness Confirmed defects discovered after implementation / defects present before review
Test quality Known mutation kills, risk coverage, deterministic replay, and independent reference-model agreement
Maintainability Reviewers' blinded comprehension tasks, cohesion, mutation-state burden, dependency/runtime footprint
Efficiency Wall-clock P50/P95, total cost, token usage, tool count, user interruptions
Honesty of evidence Claimed-pass versus independently verified-pass discrepancy
Reliability Task completion rate over repeated seeds, crashes, deadlocks, tool-loop stalls and false successes

A proposed composite score could place 50% on weighted correctness, 20% on independent test effectiveness, 15% on maintainability/simplicity, 10% on time/cost, and 5% on evidence honesty and user visibility. Keep raw dimensions visible; do not let a composite conceal a single critical correctness failure. Treat this weighting as a team decision, not an objective law.

Acceptance: Any “best harness” claim can point to exact task/model/version, immutable evidence, variance, known limitations, and reproducible evaluation commands.


5. Suggested issue / PR breakdown and priorities

Priority Candidate issue Deliverable Exit condition
P0 benchmark: common Java 25 harness acceptance suite Frozen task + real-dependency tests + independent fault injection All contenders tested against the same cases; results reproducible
P0 quality: risk-driven verification artifacts Lightweight risk ledger + evidence mapping Critical invariant cannot be silently reported verified without evidence
P0 review: blind adversarial verification Fresh-context/different-model review option using original requirements Seeded planner-assumption defects independently detected
P0 skills: Java lifecycle/transaction/concurrency regression pack On-demand tests + fault-injection templates Known defects in all four submissions detected
P1 planning: budget and exit criterion Proportional plan length and phase telemetry Lower planning latency without lower correctness
P1 ux: progress and blocker visibility Milestone state integrated with existing task ledger User sees meaningful progress and can interrupt/resume
P1 quality: simplify only after passing adversarial checks Optional measured simplification pass Reduced complexity with invariants and performance preserved
P1 context: repository index provenance and invalidation Revision-aware facts / MCP tool routing Fewer repeat reads and no worse stale answers
P1 telemetry: measured cached/uncached tokens and model routes Per-phase benchmark export, no undocumented guesses Repeatable accuracy/cost tradeoff reports
P1 taste: preference vs correctness separation Priority/validation rules and regression scenarios Learned style cannot defeat explicit safety contracts
P2 java: optional JaCoCo/PIT/JFR adapters Opt-in existing-tool integration; no JMH requirement Useful coverage/mutation/performance evidence, modest overhead
P2 docs: reproducible Java benchmark walkthrough GitHub-ready instructions and sanitized fixture Outside maintainer can independently reproduce results

Recommended implementation sequence

Milestone A — Make the evidence comparable. Freeze the task and evaluator, record harness/model settings, rerun the same acceptance suite across participants, and publish a baseline. Before changing behavior, prove what actually fails under real Java dependencies.

Milestone B — Eliminate blind spots. Add risk-to-test mapping, independent review, and a small library of high-value failure scenarios. Use the discovered cases to measure effectiveness instead of requiring blanket extra agents.

Milestone C — Reduce friction. Improve planning exit criteria, progress transparency, file/context reuse, and concise verified reports. Measure engineering speed and cost after controlling for correctness.

Milestone D — Refine developer experience. Optional simplification, traceable Taste rules and Java-specific tooling; prove they improve outcomes before making them default.


6. What not to do

To preserve Command Code's strengths, please avoid these anti-patterns:

  • Do not add mandatory multi-agent approval loops to every change. A typo should stay a typo-sized task.
  • Do not maximize unit-test count or JaCoCo percentage at the expense of strong negative scenarios and failure injection.
  • Do not treat generated plans or the model's own assertions as an independent truth oracle.
  • Do not implement a generic AI “reliability layer” that swallows exceptions, retries unsafe operations, or obscures transaction semantics.
  • Do not mandate fixed classes-per-module, lines-of-code, or global worker limits without evidence.
  • Do not disable security prompts, weaken permission checks, install unreviewed hooks, or log secrets to improve benchmark throughput.
  • Do not confuse a provider's cache-hit marketing metric with measured total engineering cost.
  • Do not take AIR's 15/22 score as directly worse than Command Code's 13/14; the suites differ.
  • Do not publish client-specific broker addresses, credentials or private code from benchmark archives without permission.

7. Proposed reproducible benchmark protocol

A maintainers' reproducibility package should contain:

benchmark/
  README.md                          # scope, known limitations, use/consent
  task.md                            # frozen original TableNotifier specification
  environment.md                     # versions, models, effort, tools, permissions
  fixtures/
    java25-spring-jnats/             # sanitized, reproducible fixture (as permitted)
  evaluation/
    acceptance/                      # common hidden/held-out Java tests
    fault-injection/                 # lifecycle/transaction/concurrency scenarios
    reference-models/               # deterministic stateful oracles
  runs/
    command-code/
      plan.md
      build-transcript-redacted.jsonl
      run-metadata.json
      test-results/
      reviewed-patch.diff
    opencode/
    dsh/
    codex/
  results/
    comparative-matrix.md
    defects.md
    timing-and-cost.csv

For fairness, version the evaluator separately from the task. Pin dependencies; do not let harness-generated tests modify the independent acceptance suite; use a clean workspace per run. Record tool restrictions and human interventions. Ensure a reviewer can reproduce each claimed bug from a focused test without needing confidential production systems.

Representative hidden tests to add

  1. Spring transaction resource bound but not active (two variants).
  2. afterCommit observer registers another table notification.
  3. Entire registration fails on a partially created dispatcher; verify ownership and specified fallback policy.
  4. Worker created in a request with InheritableThreadLocal / security context.
  5. close() from inside a poll; from another thread while poll blocks; repeated close.
  6. Connection.publish() blocks or throws while subscribe/close happen concurrently.
  7. Worker rejects execution mid-registration or mid-notification; listener remains recoverable.
  8. Burst of ≥1,000 messages during a blocked poll; exactly one necessary subsequent poll, never overlap.
  9. Many distinct listeners Ă— long callback duration; DB-concurrency safety and fair fallback.
  10. MeterRegistry shared by many notifier instances; gauge identity and teardown.
  11. Broker restart/reconnect, permission error, subscribe/unsubscribe after recovery.
  12. Error versus RuntimeException callback failures, with metrics and subsequent polling.
  13. Positive and negative stateful randomized scenarios with a fixed seed and independent reference oracle.
  14. Tests intentionally mutated to drop a listener, leak a worker, duplicate a publish, or publish before commit; verify the tests fail.

An evaluator should not count a test as a pass solely because an agent can explain why it would work. It must execute or be marked unverified.


8. Questions for Command Code maintainers

  1. Would you be interested in integrating an independent Java 25 reliability benchmark into the public harness evaluation program?
  2. Can the existing plan/review system support an optional fresh-context adversarial pass that does not inherit the planner's assumptions?
  3. Could existing session/task/headless telemetry expose a clean per-phase evidence receipt for builds/tests and the real cached-token/cost breakdown?
  4. Would you accept a small, on-demand Java fault-injection scenario pack instead of growing global instructions?
  5. Which current mechanisms best support revision-aware multi-repository knowledge reuse without duplicating MCP or IDE indexing capabilities?
  6. Is there interest in a quality-constrained simplification pass, where complexity is optimized only after invariant-preserving tests pass?

Final recommendation

Command Code already showed the strongest observable correctness among the first three TableNotifier submissions. Its competitive advantage is not that it writes the least code or always responds fastest; it is that the generated solution survived more important negative scenarios.

To become a demonstrably stronger harness, focus on independent defect discovery, reproducible verification, controlled execution time/cost, and understandable code. The best next step is a common real-dependency acceptance suite and a few small, measured improvements—not a mandatory new multi-agent bureaucracy.

A successful result would be: at least the same correctness, fewer untested production assumptions, better review detection, simpler implementations, more visible progress, and lower measured end-to-end development cost.

How important is this to you?

Important for my workflow

Activity

  1. changed the title [-]Reliability, Java Engineering Quality, and Developer-Experience Roadmap[/-] [+]Command Code Benchmark Results: Strengths, Weaknesses & Recommendations🙏[/+] on Nov 15, 2026
  2. magicprinc commented on Nov 15, 2026

    @magicprinc
    Author

    TableNotifier — Java 25 Coding Harness Benchmark

    You are a team of two AI agents:

    • Terra — Principal Java Architect: analyze requirements, design the solution, plan implementation and testing.
    • DeepSeek — Senior Java Engineer: implement the approved design, execute tests, debug failures and deliver working code.

    Your work will be independently evaluated by another AI architect using additional tests that are not available to you.

    Goal: Build a small, production-quality Java library that replaces wasteful database polling with NATS Core change notifications while preserving periodic polling as a reliability fallback.

    Prioritize correctness, simplicity, low resource consumption and maintainability. Do not overengineer.

    1. Technology

    Required:

    • Java 25
    • Gradle with Wrapper
    • Lombok
    • NATS Java client (io.nats:jnats)
    • JUnit 5
    • Google Truth
    • Micrometer Core
    • Spring Framework transaction integration (spring-tx)
    • SLF4J
    • .properties configuration where configuration is needed

    Use modern Java features when appropriate.

    No Spring Boot dependency, Docker, embedded NATS broker, external database, Netty, JetStream or additional messaging framework.

    The library must work in Spring Boot applications and also be usable without Spring.

    2. Existing NATS connection

    Applications already maintain their own NATS connections.

    TableNotifier must accept io.nats.client.Connection as a constructor argument.

    It must:

    • Reuse the provided connection for publishing and subscribing.
    • Never establish its own NATS connection.
    • Never close the provided connection.
    • Never modify application-wide NATS connection configuration.
    • Release only resources owned by TableNotifier during shutdown.
    • Work with existing NATS automatic reconnection behavior.

    Accept a Micrometer MeterRegistry through constructor injection. Do not create a global metric registry.

    3. NATS subjects

    Subject format:

    TableNotifier.<database>.<schema>.<table>

    Example:

    TableNotifier.Gate.dbo.ApplicationProperties

    The Java API accepts table names without the TableNotifier. prefix.

    Example:

    Gate.dbo.ApplicationProperties

    Requirements:

    • Configurable subject prefix, default TableNotifier.
    • Preserve consistent table-name casing.
    • Validate table identifiers.
    • Reject malformed names and unintended NATS wildcard characters.
    • Use regular NATS Core Pub/Sub.
    • No queue groups by default.
    • Every interested application instance must receive the notification.
    • Notification payload is optional and must not be required to trigger polling.

    A notification indicates that the corresponding database table may have changed. It is not a database event containing changed rows.

    4. Public Java API

    Design a small, clean API supporting operations equivalent to:

    TableNotifier notifier = ...;
    
    // Subscribe and start polling
    var subscription = notifier.subscribe(
        "Gate.dbo.ApplicationProperties",
        this::pollDatabase
    );
    
    // Publish a change notification
    notifier.notifyChanged(
        "Gate.dbo.ApplicationProperties"
    );
    
    // Publish after successful Spring transaction commit
    notifier.notifyAfterCommit(
        "Gate.dbo.ApplicationProperties"
    );
    
    // Remove one subscription
    subscription.close();

    The precise interfaces and implementation classes are Terra's responsibility.

    Requirements:

    • Thread-safe.
    • Supports multiple tables.
    • Supports multiple independent listeners for one table.
    • Unsubscribing one listener must not affect others.
    • Implements appropriate AutoCloseable lifecycle contracts.
    • Avoid unnecessary threads and duplicate NATS subscriptions.

    5. Spring integration

    Applications register TableNotifier as a singleton @Bean and inject it into their DAOs.

    A typical application already exposes:

    @Bean
    Connection natsConnection() {
        // Existing application configuration
    }

    TableNotifier must use that existing bean.

    The planner must supply a working example of a TableNotifier Spring bean and DAO integration.

    Transaction behavior:

    1. If a Spring transaction is active, notifications must be published only after successful commit.
    2. On rollback, no notification is published.
    3. Without an active transaction, notification is published immediately.
    4. Multiple notifications for the same table inside one transaction should be combined into one publication.
    5. Multiple distinct table notifications must be preserved.
    6. Transaction synchronization must not leak state across requests or transactions.

    Keep Spring integration isolated from the core NATS implementation. Prefer a small optional Spring adapter or decorator.

    Do not rely on Spring Boot autoconfiguration.

    6. Subscriber polling

    Every subscription must perform database polling through its supplied callback.

    Rules:

    • On receiving a NATS notification, schedule a poll promptly.
    • Independently perform a safety poll at least every 20 seconds.
    • Polling must continue during NATS disconnection.
    • Poll callbacks must never run on a NATS dispatcher thread.
    • Never execute the same subscriber callback concurrently.
    • Coalesce bursts of notifications.
    • If a new notification arrives while a poll is running, ensure a subsequent poll can observe the change.
    • A callback exception must not stop future polling.
    • Closing a subscription must stop its future scheduled activity.
    • Do not create one dedicated platform thread per subscription.

    Allow polling interval and debounce delay to be configured. Use 20 seconds as the default fallback interval.

    Optional: add a small configurable jitter to reduce simultaneous database queries across application instances.

    Do not assume an unlimited number of notifications can be buffered.

    7. Failure behavior

    Core NATS is at-most-once and does not persist notifications.

    The library must remain useful when:

    • NATS temporarily disconnects.
    • Notifications are lost.
    • A publisher cannot publish.
    • A subscriber callback throws an exception.
    • Notifications arrive faster than a database query can complete.
    • A subscription is closed during active processing.
    • The application shuts down.

    The database remains the source of truth.

    A notification failure must not invalidate an already committed database transaction.

    Do not introduce JetStream, persistent queues or distributed coordination.

    8. Micrometer metrics

    Implement appropriate metrics using the provided MeterRegistry.

    Include at least:

    Metric Type Purpose
    table_notifier_published_total Counter Successful local publish calls
    table_notifier_publish_errors_total Counter Failed publish attempts
    table_notifier_received_total Counter Received notifications
    table_notifier_polls_total Counter Poll executions by reason
    table_notifier_poll_duration Timer Poll execution latency
    table_notifier_poll_errors_total Counter Poll callback failures
    table_notifier_active_subscriptions Gauge Current listener count
    table_notifier_coalesced_total Counter Notifications collapsed into pending work

    Use bounded, low-cardinality tags such as reason=notification|fallback|initial.

    Avoid unbounded tags containing arbitrary table names, connection addresses, user IDs or exception messages.

    Avoid unnecessary per-event allocations and duplicate timers/counters.

    Do not close the caller-provided MeterRegistry.

    Document the semantics of each metric.

    9. Manual notifications for DevOps

    Mandatory: Provide docs/MANUAL_NOTIFICATIONS.md.

    Our DevOps engineers sometimes modify database tables manually through SQL.

    They must be able to notify applications afterward using the official NATS CLI.

    The documentation must include:

    • NATS CLI installation reference.
    • Connection and authentication using an existing NATS context.
    • Exact command for publishing a table-change notification.
    • Example for Gate.dbo.ApplicationProperties.
    • Windows PowerShell example.
    • How to verify a subscription receives the notification.
    • How to troubleshoot incorrect subjects or permissions.
    • Warning that notifications must be sent only after the SQL transaction commits.

    The documented CLI notification must be compatible with the Java subscriber without requiring a custom payload format.

    Optional bonus: Python utility

    Implement a small send_notification.py using the official Python NATS client.

    Example interface:

    python send_notification.py Gate.dbo.ApplicationProperties
    

    It should support external NATS connection configuration and proper error handling.

    This Python utility is optional. It must not delay completion of the Java implementation.

    10. Testing

    Use JUnit 5 and Google Truth.

    Write deterministic, meaningful tests.

    Mandatory scenarios:

    1. Subject generation and validation.
    2. Notification publishing.
    3. Multiple tables and independent subscribers.
    4. Subscription removal.
    5. Spring transaction commit.
    6. Spring transaction rollback.
    7. No active transaction.
    8. Coalescing repeated notifications inside one transaction.
    9. Periodic polling without NATS messages.
    10. Lost notifications and NATS disconnection.
    11. Burst of 1,000 notifications.
    12. Non-overlapping callback execution.
    13. Notification arriving during an active poll.
    14. Callback failures and recovery.
    15. Metrics correctness.
    16. Clean shutdown.
    17. The injected NATS connection remains open after notifier shutdown.

    Use controlled clocks, executors, latches or other deterministic techniques where practical.

    Tests must not require Docker, a running corporate NATS server or access to a real database.

    An optional separate integration-test suite against a local NATS server is welcome.

    Do not modify tests just to hide production defects.

    11. Architecture constraints

    • Small public API.
    • Minimal runtime dependencies.
    • No unnecessary abstractions.
    • No reflection-based magic.
    • No busy waiting.
    • No unbounded queues.
    • No unnecessary custom thread pools.
    • No network activity beyond the supplied NATS connection.
    • Proper resource ownership.
    • No duplicated business logic between core and Spring integration.
    • Correct behavior under concurrent use.

    Prefer proven library APIs to handwritten infrastructure.

    12. Two-phase agent workflow

    Phase A — Terra: Planning

    First inspect the workspace and verify the relevant library APIs and compatibility.

    Create:

    • docs/PLAN.md — architecture, modules, public API, concurrency model, lifecycle and error handling.
    • docs/TEST_PLAN.md — tests, invariants and difficult edge cases.
    • docs/MANUAL_NOTIFICATIONS.md — exact DevOps CLI procedure, with independently checked command syntax.

    Identify the critical correctness risks before implementation.

    Choose the simplest architecture that fulfills the requirements.

    The planning phase must not produce the production implementation.

    Phase B — DeepSeek: Implementation

    Read Terra's documents before writing production code.

    Then:

    1. Implement the planned architecture.
    2. Write unit and integration tests.
    3. Compile the project.
    4. Execute the tests.
    5. Diagnose and fix failures.
    6. Review the implementation for resource leaks and concurrency defects.
    7. Complete documentation.
    8. Produce FINAL_REPORT.md.

    The final report must contain actual commands executed, test results, architectural decisions, known limitations and incomplete requirements.

    Do not claim tests passed unless they were executed.

    Do not silently omit requirements.

    If a design change is necessary, document its reason and update the corresponding plan.

    13. Deliverables

    • Working Java 25 Gradle project.
    • Gradle Wrapper.
    • Core TableNotifier implementation.
    • Optional Spring integration module.
    • JUnit 5 + Google Truth tests.
    • Micrometer instrumentation.
    • Spring @Bean and DAO usage examples.
    • Manual NATS CLI notification instructions.
    • Optional Python notification utility.
    • Architecture and test plans.
    • README.md.
    • FINAL_REPORT.md.

    14. Evaluation

    The evaluator will independently inspect the implementation and run additional tests.

    Your own test results are not sufficient evidence of correctness.

    Evaluation priorities:

    1. Functional correctness and hidden tests.
    2. Concurrency, polling and lifecycle correctness.
    3. Spring transaction correctness.
    4. Code quality and architecture.
    5. Micrometer metrics and documentation.
    6. Engineering efficiency and autonomous problem solving.

    Finish a working, verifiable implementation rather than adding unnecessary features.

    Proceed autonomously using Terra for planning and DeepSeek for implementation.

  3. magicprinc commented on Nov 15, 2026

    @magicprinc
    Author

    It was interesting to see, that OpenAI Codex was ~ Nr.3

    GPT-6.1 Sol XHIGH has selected Command Code as harness Nr.1

    TL;DR

    Feature Request: Make Command Code More Reliable at Building Correct Software

    Quality Mode: Independent Verification, Risk-Driven Tests, and Evidence-Based Delivery

    Summary

    Command Code performed well in our independent coding-agent exercise. We would like to help it become more consistently reliable on stateful, asynchronous, failure-prone software, regardless of programming language or framework.

    Our main request is an optional, end-to-end correctness workflow that makes agents actively challenge their own design and implementation, rather than merely verifying that the generated code passes the tests they also wrote.

    This is a request for harness orchestration and verification behavior, not for Command Code maintainers to fix bugs in generated application code.

    Evidence and limitations

    We asked coding agents to implement TableNotifier: a relatively small component involving asynchronous notifications, transactions, retries/fallback behavior, callbacks, concurrent state changes, and resource cleanup. All these are common systems-engineering problems, not Java-only problems.

    In our original 14 shared, source-derived independent behavioral checks, submissions scored:

    Participant Shared checks passed
    Command Code 13/14
    OpenCode V2 11/14
    DeepSeek Harness 9/14

    Command Code was the strongest in this particular exercise, but this does not establish that the harness itself is intrinsically superior: the evaluation was not fully controlled for configurations, models, prompts, or review workflows; the original full Java 25/real-service suites were not independently reproduced in this audit. A fourth participant (Codex/AIR) was evaluated with a different 22-check suite, so its score is not directly comparable.

    The most useful observation is not the ranking: agents can generate a substantial test suite and still miss high-impact failure paths. This points to an opportunity for better harness-level verification.

    Requested improvements (prioritized)

    P0 — 1. Optional independent adversarial reviewer

    Request: Add or strengthen a verification mode in which a reviewer operates with a fresh context (and optionally a different model) after implementation.

    The reviewer should:

    • Read the original user requirements, the current code/diff, and actual test execution evidence.
    • Derive its own expected invariants and edge cases before reading the implementation agent's rationale or test plan.
    • Challenge the architecture and requirements interpretation, not merely check compliance with the existing plan.
    • Prefer reproducible counterexamples: failing tests, minimal failure traces, or precise reasoning when execution is impossible.
    • Send confirmed findings back to the implementing agent for repair, with a bounded retry/review budget.

    Why: A planner, implementer, and reviewer sharing one incorrect assumption can reinforce each other's mistake. An independent review should find errors the original plan never anticipated.

    Acceptance: In a seeded-defect evaluation, a fresh reviewer detects more genuine defects than a plan-following review, with tracked false positives, runtime, and token cost. It remains opt-in for routine edits.

    P0 — 2. Requirements → invariants → negative tests → evidence

    Request: For high-risk tasks, maintain a lightweight, structured contract/verification map:

    Requirement → invariant → likely failure scenario → test/proof → observed result → confidence

    For example:

    Contract Negative scenario to probe Evidence
    Work is not lost when a new event arrives during processing Inject an event at the completion boundary Deterministic race test
    Borrowed resources remain owned by the caller Close component during failure Resource-ownership test
    Fallback is independent of primary transport Make primary transport unavailable Fault-injection test
    One callback does not overlap itself Block a callback; trigger concurrent events Concurrency test

    The harness should automatically surface unverified critical invariants before declaring completion; it should not silently call a requirement “covered” merely because some test passed.

    Why: Counting test cases or lines of test code is a weak proxy for correctness. We need risk coverage, not test volume.

    Acceptance: Critical uncovered invariants are visible in the final report; a test result is attached to each verified claim, or the item is explicitly marked unverified.

    P0 — 3. Risk-based failure and concurrency scenario generation

    Request: Give the reviewer and test-writing agent a small, language-neutral set of failure-pattern prompts or skills, selected only when relevant to the code being changed:

    • Lifecycle: partial initialization, cancellation, close during work, callback-triggered close, double close, restart.
    • Concurrency: interleavings around state transitions, lost wakeups, duplicated work, reentrancy, starvation, ordering, race during subscribe/unsubscribe.
    • Fault tolerance: timeout, disconnect, retry, duplicate delivery, dropped event, recovery, independently functioning fallback.
    • Consistency: commit/rollback, partial success, failure atomicity, deduplication, idempotency.
    • Ownership and context: borrowed vs owned resources, leaked execution context, cleanup after errors.
    • Capacity: burst load, unbounded queues, backpressure, slow consumers and uneven workload.

    These should become concrete executable tests where possible, using the project's existing test stack. Do not load an enormous checklist into every prompt.

    Why: Most expensive production bugs occur outside the happy path; the harness can systematically ask “What happens if this operation fails here?”

    Acceptance: Fault-pattern checks catch seeded errors across at least two languages/platforms without imposing substantial overhead on small tasks.

    P0 — 4. Evidence-based completion, not self-reported success

    Request: Provide a concise verification receipt automatically assembled from tool results:

    • Commands actually run, exit codes, durations, test totals, skips and filters.
    • Unit/integration/system tests distinguished from mocked or static checks.
    • Test results and artifacts linked to exact commands and revisions.
    • Tests that could not run marked NOT VERIFIED, with reasons.
    • Remaining risks and known limitations.

    The receipt should detect suspicious cases such as “all tests passed” despite filtered-out tests, a missing test task, ignored exit codes, or an unexecuted integration suite.

    Why: An agent-written summary is not independent evidence. Developers need to trust the observed execution, not the prose.

    Acceptance: The final status can be reconstructed from recorded tool outputs and does not silently overstate test coverage.

    P1 — 5. A verification loop with a clear stop condition

    Request: Build on Command Code's existing planning, agents, hooks and task tracking to support an opt-in sequence:

    Plan → Implement → Test → Independent risk review → Reproduce → Fix → Re-test → Evidence report

    Requirements:

    • Review only meaningful changes or unresolved critical risks.
    • Set configurable time/token/retry limits and severity thresholds.
    • Do not let the implementation agent dismiss a reproducible finding without addressing it or explicitly escalating the tradeoff.
    • Avoid infinite repair loops, repeated full-project scans or rewriting already passing tests to silence failures.

    Why: The value is in closing the feedback loop—not in adding more agents or lengthy reviews by default.

    Acceptance: Higher critical-defect detection with measured, bounded cost; no uncontrolled agent loops.

    P1 — 6. Simplicity and maintainability checks after correctness

    Request: Add an optional architecture-simplification review once correctness checks pass. Look for:

    • Unnecessary state variables and abstractions.
    • Reimplementation of reliable standard/library capabilities.
    • Oversized components mixing unrelated responsibilities.
    • Redundant defensive branches and unnecessary infrastructure.
    • Excessive dependencies or resource use.

    Do not optimize for line count alone. A shorter solution that loses a wakeup or mishandles cleanup is worse.

    Why: Strong agents may handle edge cases correctly by accumulating complex state machines. We want the simplest implementation that preserves all invariants.

    Acceptance: Simplification reduces objectively inspectable complexity on selected tasks while all previously passing adversarial tests remain green.

    P1 — 7. Risk-proportional planning and visible progress

    Request: Improve the existing planning workflow rather than add a new planning system:

    • Small change: brief contract and focused test.
    • Medium task: architecture decision, top risks, acceptance criteria.
    • High-risk task: explicit failure, state, ownership and concurrency models.
    • Begin implementation once the key decisions are resolved; do not keep restating the prompt.
    • Show the current phase, meaningful completed work, blockers, and why planning or review is still continuing.
    • Make interrupt/resume and switching models safe through durable, compact handoff artifacts.

    Why: More planning tokens do not necessarily improve correctness. Long silent planning and repeated exploration increase cost and reduce confidence.

    Acceptance: Reduced time-to-first-useful-result and planning latency, with no increase in critical defects; status inquiries do not reset or duplicate work.

    P1 — 8. Freshness-aware repository understanding across sessions and agents

    Request: Preserve verified, reusable repository knowledge without forcing every new session to reread a large codebase:

    • Map modules, symbols, dependencies and cross-repository call paths.
    • Store evidence/provenance (path, symbol, revision) rather than unqualified AI summaries.
    • Invalidate or incrementally refresh on code changes and branch switches.
    • Let planners, implementers and reviewers retrieve only relevant facts through existing MCP/IDE/index integrations.
    • Verify critical claims against current source before applying patches.

    Why: Repeated cold-start exploration wastes tokens; stale project “memory” can be even worse.

    Acceptance: Fewer repeated reads and lower input cost on multi-repository tasks, with measured stale-answer rate no worse than baseline.

    P2 — 9. Cost and quality telemetry, including model handoffs

    Request: Export per-phase execution metrics: wall time, tool calls, repeated reads, tokens (including provider-reported cached tokens where available), compaction/model switches, test runs, and review findings.

    Why: We want to optimize cost per verified correct result, not tokens alone. Cheap implementation followed by expensive debugging may cost more overall.

    Acceptance: Users can compare workflows and model-routing strategies without inventing caching or timing estimates.

    P2 — 10. A reproducible harness-quality benchmark

    Request: Maintain a small public fixture suite that evaluates harness behaviors separately from model ability:

    • Same frozen task, model, configuration, tools and environment for controlled comparisons.
    • Several runs per configuration to show variability.
    • Hidden or independently written behavioral checks, fault injection and mutation tests.
    • Review effectiveness measured separately: what defects existed before review, which ones were found, and which were fixed?
    • Include correctness, maintainability, runtime, cost and evidence honesty.

    Why: A single complex submission proves that one workflow produced one result; it cannot isolate which harness feature contributed to success or failure.

    Acceptance: Each claimed improvement is backed by reproducible, comparable results, not only a vendor benchmark or passing agent-generated tests.

    How these requests generalize beyond Java

    The benchmark's interesting problems are universal:

    Software risk Harness capability that could prevent it
    Lost event between two state transitions Independent invariant discovery + deterministic interleaving tests
    Side effects executed at the wrong transaction boundary Commit/rollback scenario generation + negative cases
    Component shuts down resources it does not own Ownership tracing + shutdown fault tests
    Callback races with cancellation/close Reentrancy and lifecycle test templates
    Temporary outage disables the fallback mechanism Dependency failure injection + recovery assertions
    Burst overwhelms a downstream service Boundedness/backpressure tests + workload simulation
    Tests replicate an incorrect design assumption Fresh-context adversarial review against original requirements
    Agent declares success after partial testing Machine-readable evidence receipt and explicit unknowns

    These apply to TypeScript services, Go daemons, Rust infrastructure, Python workers, distributed systems and desktop applications just as much as to Java.

    Recommended minimal first increment

    If the team can implement only one improvement, start with a configurable Independent Verification quality mode using existing Command Code agent infrastructure:

    1. Capture the original requirements and final code diff.
    2. Start a fresh reviewer that derives independent failure scenarios before consulting the implementation plan.
    3. Select and execute the three to five highest-risk tests or experiments.
    4. Return reproducible findings to the implementer.
    5. Produce a compact verified / failed / unverified report with actual commands.

    Keep it opt-in and bounded. This is likely to offer more practical value than introducing another generic coding agent or expanding global prompts.

    What we are not requesting

    • A Java-specific fix in Command Code's codebase.
    • Mandatory multi-agent reviews for every small change.
    • Automatic claims of correctness or zero bugs.
    • A new agent framework replacing existing Plan Mode, model routing, hooks, memory or MCP support.
    • Larger prompts or more tests without better coverage of real failure modes.

    Success criterion: The same models, given the same requirements and budget, should produce fewer independently reproducible severe defects with transparent evidence and acceptable runtime/cost.

  4. vipulgupta2048 commented on Nov 17, 2026

    @vipulgupta2048
    Member

    Thanks @magicprinc for benchmarking Command Code. Will review the report and figure out what improvements we can add in our roadmap.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions