Skip to content

improvement(computer-use): name the activation step when an element refuses an advertised action - #10189

Merged
waleedlatif1 merged 1 commit into
stagingfrom
improvement/computer-use-refused-action-hint
Nov 18, 2026
Merged

waleedlatif1 merged 1 commit into
stagingfrom
improvement/computer-use-refused-action-hint

Conversation

@waleedlatif1

Copy link
Copy Markdown
Collaborator

Summary

  • Finder and file-panel cells advertise AXOpen but refuse it, and the error only said to click instead. The fallback double-click needs the app in front, so the agent's next click usually failed with "Coordinate mouse input requires the target app in front" and cost another round trip
  • The refusal now names the step: call activate_app for the app, then click it (clickCount 2 opens most files and folders)

Type of Change

  • Improvement

Testing

  • 8 Finder-heavy tasks from a native multi-app macOS suite, one run each, in a macOS VM: all passed; after a refused AXOpen, the next click failed 0/9 times (vs 3/6 with a variant that named activation only when the app looked backgrounded, and 2/7 on staging), and activate-then-click succeeded 4 times
  • An earlier variant that checked the frontmost app first still sent the agent straight into a failing click in 2 of 3 "in front" cases, so the step is always named
  • Helper compiled with swiftc -target arm64-apple-macosx14.0; no TS changes

Checklist

  • Code follows project style guidelines
  • Self-reviewed my changes
  • Tests added/updated and passing (new tests pass the test-audit authoring gate)
  • No new warnings introduced
  • I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

@vercel

vercel Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated
docs Skipped Skipped Nov 18, 2026 11:41pm UTC

Request Review

@greptile-apps

greptile-apps Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 5/5

[Low impact] This PR appears safe to merge.

Summary

The refusal message now tells the agent to call activate_app before clicking an element whose advertised action failed.

  • Refused actions now tell the agent to activate the app before clicking.

waleedlatif1 dismissed the earlier subaccount-cancel, saved-reference, and URL-whitespace findings because the branch rebuild removed those changes from this PR. The current diff confirms that only ComputerUse.swift changes.

Reviews (3) · Last reviewed commit: "improvement(computer-use): name the acti..." · Reviewed by Greptile

Comment thread apps/sim/tools/kalshi/cancel_order.ts Resolved
Comment thread apps/sim/tools/tavily/map.ts Resolved
Comment thread apps/sim/tools/url-path.ts Resolved
@greptile-apps

greptile-apps Bot commented Nov 18, 2026

Copy link
Copy Markdown
Contributor

Comments Outside Diff

These findings could not be posted inline.

  • P2 Missing job IDs hide errors apps/sim/tools/firecrawl/agent.ts:104 ▶

    Missing job IDs hide errors. Without the start-response check, a successful HTTP response containing no id is marked successful and polled at /agent/undefined. Any original data.error is then replaced by a polling error. The same regression affects extract and crawl. Restore the missing-ID checks so malformed replies fail immediately with the original error rather than sending a request for a nonexistent job.

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 32 files

Reply with feedback, questions, or to request a fix.

Turn on auto-fix | Re-trigger cubic

Comment thread apps/sim/tools/kalshi/amend_order.ts Resolved
Comment thread apps/sim/tools/tavily/map.ts Resolved
Comment thread apps/sim/tools/firecrawl/crawl.ts Resolved
Comment thread apps/sim/tools/ketch/get_consent.ts Resolved
Comment thread apps/sim/tools/kalshi/cancel_order.ts Resolved
Comment thread apps/sim/tools/firecrawl/agent.ts Resolved
Comment thread apps/sim/tools/url-path.ts Resolved
Comment thread apps/sim/tools/arxiv/get_paper.ts Resolved
@waleedlatif1
waleedlatif1 force-pushed the improvement/computer-use-refused-action-hint branch from 429160c to 911f4d9 Compare November 18, 2026 23:41
@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@greptile

@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@cubic-dev-ai review this PR

@cubic-dev-ai

cubic-dev-ai Bot commented Nov 18, 2026

Copy link
Copy Markdown
Contributor

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 1 file

Confidence score: 5/5

  • Automated review surfaced no issues in the provided summaries.
  • No files require special attention.

Turn on auto-fix | Re-trigger cubic

@waleedlatif1
waleedlatif1 merged commit 06db09b into staging Nov 18, 2026
49 checks passed
@waleedlatif1
waleedlatif1 deleted the improvement/computer-use-refused-action-hint branch November 18, 2026 23:45

This branch was previously deployed

1 inactive deployment
Preview — 911f4d92 Deployed Nov 18, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant