Skip to content

docs(assess): align decide's needs-clarification step with the assessment guide - #5329

Open
aarjang wants to merge 5 commits into
github:mainfrom
aarjang:fix/5325-assess-decide-refine
Open

aarjang wants to merge 5 commits into
github:mainfrom
aarjang:fix/5325-assess-decide-refine

Conversation

@aarjang

@aarjang aarjang commented Nov 9, 2026 •

Copy link
Copy Markdown

Description

Fixes #5325.

docs/guides/assessment.md and the decide command's report-back disagreed on what a needs-clarification verdict means for the next step: the guide treated answering a [NEEDS CLARIFICATION] question as artifact refinement with command re-runs as the exception, while speckit.assess.decide.md told the agent to unconditionally "re-run the named stage". Tools that offer a next-step answer box had to pick a side (#5325).

Per the maintainer's direction in this PR's discussion, the adopted design — the acceptance criterion for this change — keeps refinement and reassessment separate, and all four documents now state it consistently:

  • A needs-clarification verdict means decide reports the blocking questions and stops; it does not resolve them.
  • The user supplies the missing information by refining the named artifact in place (editing the Markdown directly, or explicitly directing an agent).
  • decision.md is not revised in place: reassessment goes through re-running decide, which rebuilds the scorecard, rationale, verdict, and handoff from the updated artifacts and asks before overwriting the previous decision.
  • Re-running a stage earlier than decide stays the exception (wrong slug, discarded draft, or an answer that changes that stage's inputs), never the default next step.

Updated accordingly: speckit.assess.decide.md (opening summary and report-back step), the extension README (flowchart and "Resolving clarifications" prose), docs/guides/assessment.md (verdict table and "Resolve unknowns by refining the artifacts"), and docs/reference/agentic-assessment.md (overwrite note and "Refining an assessment"). The assess extension is patch-bumped to 1.0.3 in extension.yml, the core extensions catalog, and the assess bundle — the three places the bundled-extension-versions contract test keeps in sync.

No executable behavior changes: the diff is command/doc text, version strings, and tests.

Testing

  • test_decide_needs_clarification_step_reports_and_stops pins the adopted design: it requires report-and-stop, in-place user-driven refinement, and a decide re-run in the report-back step, rejects both the old unconditional "re-run the named stage" and the interim "revise decision.md" wordings, and rejects the stale opening summary. Verified to fail against both prior versions of the command file and pass on this branch.

  • tests/extensions/assess and tests/contract/test_bundled_extension_versions.py pass locally (.venv/bin/python -m pytest per CONTRIBUTING); a grep for the superseded phrases across extensions/ and docs/ is clean.

  • Full suite: the only failures are the pre-existing script-parity flakes on this macOS machine, identical on a clean main checkout (also noted in fix(workflows): evaluate chained comparisons pairwise like Python #5203) and untouched by this change.

  • Tested locally with uv run specify --help

  • Ran existing tests with uv sync && uv run pytest

  • Tested with a sample project (if applicable)

AI Disclosure

  • I did not use AI assistance for this contribution
  • I did use AI assistance (fill in the disclosure below)

AI disclosure: Written with Claude Code (model: Claude Fable 5, default reasoning effort, human-supervised). The agent verified the documented mismatch, implemented the maintainer-chosen separation across the four documents, wrote the contract test and synced version bumps, and ran the verification; I reviewed the diff before submitting. Commits carry a matching Assisted-by: trailer.

🤖 Generated with Claude Code

…ment guide

docs/guides/assessment.md says answering a [NEEDS CLARIFICATION] question
is artifact refinement: edit the Markdown in place or ask in free-form
chat, and re-run a stage only as an exception. The decide command's
report-back template still told the agent to re-run the named stage, so
tools offering a next step had to pick a side, and a re-run also triggers
the overwrite confirmation -- one more prompt than the guide describes.
PR github#4402 aligned the README and the guide but left this line behind.

Reword the needs-clarification next step to match the guide and the
README's own refine path: update the named artifact in place, revise
decision.md if that clears the blocker, and re-run a stage only when the
answer changes that stage's inputs. Patch-bump the assess extension to
1.0.3 in extension.yml, the core catalog, and the assess bundle, which
the bundled-extension-versions contract test keeps in sync.

Fixes github#5325

Assisted-by: Claude Code (model: Claude Fable 5, supervised)
@aarjang
aarjang requested a review from mnriem as a code owner November 9, 2026 06:32
Copilot AI adaptive review requested due to automatic review settings November 9, 2026 06:32

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The change lacks focused contract coverage for the decide report-back guidance, so existing tests could miss a regression that incorrectly instructs rerunning the named stage.

1 open issue
What changed in this PR

This pull request aligns the assess decide command’s needs-clarification guidance with the assessment guide and README, directing agents to refine artifacts in place and rerun stages only when inputs change. It also synchronizes the assess extension patch version to 1.0.3.

Changes:

  • Updated clarification guidance in the decide command.
  • Bumped the assess extension version across its manifest, catalog, and bundle pin.
  • No executable behavior changes.
File Summary
extensions/​catalog.json Updates the cataloged assess extension version to 1.0.3.
extensions/​assess/​extension.yml Bumps the assess extension manifest version to 1.0.3.
extensions/​assess/​commands/​speckit.assess.decide.md Aligns clarification guidance with the assessment documentation; follow-up coverage is requested for this agent-facing contract, and one opening description remains inconsistent.
bundles/​assess/​bundle.yml Updates the bundle’s assess extension pin; the bundle and catalog release versions should also be bumped to deliver the metadata change as a distinct release.

This pull request requires changes.

Review effort: Lite (auto)

- The next step, by verdict:
- **go** → `__SPECKIT_COMMAND_SPECIFY__` using the handoff summary as its input.
- **needs-clarification** → re-run the named stage (e.g. `__SPECKIT_COMMAND_ASSESS_RESEARCH__ slug=<ASSESS_SLUG>`).
- **needs-clarification** → refine the named artifact in place with the missing information (edit the Markdown directly, or ask in free-form chat), then revise `decision.md` if that clears the blocker. Re-run a stage (e.g. `__SPECKIT_COMMAND_ASSESS_RESEARCH__ slug=<ASSESS_SLUG>`) only when the answer changes that stage's inputs.
…ement

Review asked for focused coverage of the decide report-back contract:
the existing assess tests verify layout, catalog registration,
resolution, and installation, so they would still pass if the
needs-clarification guidance regressed to "re-run the named stage".
Pin the guide's side on the report-back arrow line: the next step names
in-place refinement, never the unconditional re-run, and any re-run
mention comes after the refinement wording and stays conditional
("only when"). The verdict definition and the decision.md output
template deliberately stay untouched, as github#4402 established.

The test fails against the pre-github#5325 wording and passes with it.

Assisted-by: Claude Code (model: Claude Fable 5, supervised)
Copilot AI adaptive review requested due to automatic review settings November 9, 2026 06:48

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues are identified, and the guidance, regression coverage, and 1.0.3 version updates are consistently reflected across the affected files.

1 open issue

Review effort: Lite (auto)

@aarjang

aarjang commented Nov 9, 2026

Copy link
Copy Markdown
Author

Addressed the Copilot review in 0e94e5f: added test_decide_needs_clarification_step_refines_in_place to the assess extension tests. It pins the report-back arrow line for needs-clarification to in-place refinement — the unconditional "re-run the named stage" wording fails it, any re-run mention must follow the refinement wording and stay conditional ("only when"). Verified the test fails against the pre-PR wording and passes with this branch; the verdict definition and the decision.md output template are deliberately not pinned, matching what #4402 established.

AI disclosure: this round's change and comment were produced with Claude Code (model: Claude Fable 5, default reasoning effort, human-supervised), posted on behalf of @aarjang, who reviewed them before posting.

@mnriem mnriem added the triage-can-wait Verdict: valid and in-scope but deprioritized; held behind the evidence gate label Nov 9, 2026
@mnriem
mnriem requested an adaptive review from Copilot November 9, 2026 14:35

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The clarification guidance is aligned, regression coverage is included, and all assess extension version references are synchronized, with no active issues recorded in the review.

1 open issue

Review effort: Lite (auto)

@mnriem

mnriem commented Nov 9, 2026

Copy link
Copy Markdown
Collaborator

I agree that the command and guide should be consistent, but this changes the process rather than just clarifying its wording. Clarifications should be supplied by the user—either through manual edits or by explicitly directing an agent—before proceeding to the next stage, which then consumes the updated artifacts. If decide identifies blocking unknowns, it should report them and stop; after supplying the missing information in the relevant artifacts, the user can explicitly request a reassessment through decide. Replacing the rerun guidance with “then revise decision.md” blurs that boundary and substitutes in-place decision editing for reassessment through the command. Could you preserve that separation in the proposed wording and coverage? Matching the guide alone is not sufficient justification if the guide also needs correcting.

AI disclosure: Drafted for @mnriem by GitHub Copilot using GPT-6.1 Sol, human-directed; PR assessment and response drafting only.

@mnriem mnriem added the author-awaiting Waiting on author response label Nov 9, 2026
Per maintainer direction on github#5329: a needs-clarification verdict reports
the blocking questions and stops; the user refines the named artifact in
place (direct edit or explicitly directing an agent) and then requests
reassessment by re-running the decide command against the updated
artifacts. Drop the interim "revise decision.md" wording that let the
agent edit the decision in place, align the README flowchart and the
assessment guide's refinement section with the same separation, and
repin the contract test to reject both the old unconditional stage
re-run and the interim in-place revision.

Assisted-by: Claude Code (model: Claude Fable 5, supervised)
Copilot AI adaptive review requested due to automatic review settings November 10, 2026 05:53
@aarjang

aarjang commented Nov 10, 2026

Copy link
Copy Markdown
Author

Reworked in c4ed11d9ea to preserve the separation you described — thanks for spelling it out.

  • speckit.assess.decide.md — the needs-clarification next step now reads: report the blocking questions and stop; the user supplies the missing information by refining the named artifact in place (direct edit, or explicitly directing an agent), then requests reassessment by re-running decide against the updated artifacts. The interim "then revise decision.md" wording is gone, so the decision is only ever rewritten through the command (which keeps the existing overwrite confirmation in the loop). The earlier-stage re-run stays the guarded exception.
  • Guide also corrected — you were right that matching the guide alone wasn't sufficient: docs/guides/assessment.md had the same blur ("revise decision.md if this resolves the blocker" in the example prompt, and "revise the downstream scorecard, verdict, or handoff" in the prose). Both now route reassessment through re-running /speckit-assess-decide, and the verdict table's "what to do" says refine, then re-run decide. The README flowchart's needs-clarification loop likewise goes back through decide.
  • Coverage — the contract test is repinned as test_decide_needs_clarification_step_reports_and_stops: it requires "report the blocking questions and stop", in-place user-driven refinement, and a decide re-run in the step, and rejects both the old unconditional "re-run the named stage" and the interim "revise decision.md" wordings (verified: the test fails against both prior versions of the file and passes on this branch; the full assess extension suite and the bundled-version contract tests pass).

AI disclosure: This comment and the commit were drafted with Claude Code (model: Claude Fable 5), human-directed and reviewed, on behalf of @aarjang.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The change leaves contradictory clarification instructions across the guide, README, and command summary, so users and agents may follow inconsistent workflows.

4 open issues

Review effort: Lite (auto)

Comment thread docs/guides/assessment.md
Comment on lines +98 to +99
`/speckit-assess-decide`: it rebuilds the scorecard, verdict, and handoff from
the updated artifacts, and asks before overwriting the existing `decision.md`.
Comment thread extensions/assess/README.md Resolved
@mnriem

mnriem commented Nov 10, 2026

Copy link
Copy Markdown
Collaborator

Please address Copilot feedback

…cide

Copilot flagged three places where the old wording survived the
report-and-stop rework: the decide command's opening summary still sent
a needs-clarification verdict back to an earlier stage, the extension
README's clarification prose still had the agent revise decision.md in
place, and the agentic-assessment reference page did the same. All
three now route reassessment through re-running decide against the
refined artifacts, with earlier-stage reruns staying the exception, and
the contract test also rejects the stale opening summary.

Assisted-by: Claude Code (model: Claude Fable 5, supervised)
Copilot AI adaptive review requested due to automatic review settings November 16, 2026 10:55

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The documented behavior conflicts with the implementation: users cannot revise decision.md in place as described and must instead rerun decide, so the PR or acceptance criteria needs alignment.

3 open issues
2 resolved since last review

Review effort: Lite (dynamic)

- The next step, by verdict:
- **go** → `__SPECKIT_COMMAND_SPECIFY__` using the handoff summary as its input.
- **needs-clarification** → re-run the named stage (e.g. `__SPECKIT_COMMAND_ASSESS_RESEARCH__ slug=<ASSESS_SLUG>`).
- **needs-clarification** → report the blocking questions and stop — do not resolve them here. The user supplies the missing information by refining the named artifact in place (editing the Markdown directly, or explicitly directing an agent), then requests reassessment by re-running `__SPECKIT_COMMAND_ASSESS_DECIDE__ slug=<ASSESS_SLUG>`, which consumes the updated artifacts. Re-run an earlier stage (e.g. `__SPECKIT_COMMAND_ASSESS_RESEARCH__ slug=<ASSESS_SLUG>`) only when the answer changes that stage's inputs.
@aarjang

aarjang commented Nov 16, 2026

Copy link
Copy Markdown
Author

Addressed all three Copilot findings in 146332c — each was a spot where the pre-rework wording survived:

  • speckit.assess.decide.md opening summary — no longer "sends it back to an earlier stage"; it now says needs-clarification reports the blocking unknowns and waits for the user to refine the artifacts and re-run decide. The contract test also rejects the stale summary now.
  • Extension README "Resolving clarifications" prose — the paragraph that had the agent revise decision.md in place now routes reassessment through re-running /speckit-assess-decide (which rebuilds the scorecard, rationale, verdict, and handoff and asks before overwriting), and the exception sentence is scoped to commands earlier than decide, matching the flowchart.
  • docs/reference/agentic-assessment.md — both spots ("refine rather than rerunning the command" and the "Refining an assessment" section's "revise decision.md as needed") now state that decision.md is not revised in place and that reassessment goes through the decide command, with earlier-stage reruns staying the exception.

Verified no stale wording remains (grep for the old phrases across extensions/ and docs/ is clean) and the assess extension and bundled-version contract suites pass.

AI disclosure: This comment and the commit were drafted with Claude Code (model: Claude Fable 5), human-directed and reviewed, on behalf of @aarjang.

@aarjang

aarjang commented Nov 16, 2026

Copy link
Copy Markdown
Author

Good catch — the description, not the change, was stale: it still described the pre-rework approach ("revise decision.md if that clears the blocker"), which the maintainer explicitly replaced in this discussion with the overwrite-confirmed decide rerun. I've updated the PR description to adopt that design as the acceptance criteria across all four documents (command, extension README, guide, reference page), matching what the branch implements and what the contract test pins. No code change in this round.

AI disclosure: This comment and the description update were drafted with Claude Code (model: Claude Fable 5), human-directed and reviewed, on behalf of @aarjang.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The README guidance is too narrow because clarifications in problem.md can leave dependent concept.md wording stale, so it should direct updates across affected artifacts while preserving decision.md as the command-owned exception.

4 open issues

Review effort: Lite (adaptive)

Comment thread extensions/assess/README.md Resolved
Copilot pointed out that limiting the review to the artifact being
edited misses downstream staleness: a clarification in problem.md can
leave concept.md wording stale, and decide reads both. The README and
the reference page now direct the user to review dependent wording
across the affected artifacts, with decision.md staying the
command-owned exception.

Assisted-by: Claude Code (model: Claude Fable 5, supervised)
Copilot AI adaptive review requested due to automatic review settings November 17, 2026 07:11
@aarjang

aarjang commented Nov 17, 2026

Copy link
Copy Markdown
Author

Addressed in 33cf0a0: the README's clarification prose now directs reviewing and updating dependent wording across the affected artifacts — with the example you gave (a problem.md clarification leaving downstream concept.md wording stale, and decide reading both) — and the reference page's "Refining an assessment" section is scoped the same way. decision.md stays the command-owned exception, rebuilt only through the decide re-run.

AI disclosure: This comment and the commit were drafted with Claude Code (model: Claude Fable 5), human-directed and reviewed, on behalf of @aarjang.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Source tests pin the wording, but automated review cannot verify how supported agents interpret the stop/refine/re-run lifecycle or the interactive overwrite flow.

3 open issues
1 resolved since last review

Review effort: Lite (adaptive)

@mnriem

mnriem commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator

Please address Copilot feedback

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

author-awaiting Waiting on author response triage-can-wait Verdict: valid and in-scope but deprioritized; held behind the evidence gate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Assess: the decide command says to re-run the stage; the assessment guide says to refine in place

3 participants