Skip to content

fix(computer-use): observe an app's floating and panel-level windows - #10188

Merged
waleedlatif1 merged 1 commit into
stagingfrom
fix/computer-use-panel-level-windows
Nov 18, 2026
Merged

waleedlatif1 merged 1 commit into
stagingfrom
fix/computer-use-panel-level-windows

Conversation

@waleedlatif1

Copy link
Copy Markdown
Collaborator

Summary

  • Computer use only listed an app's windows at the normal window level, so panel-level windows were never observable. Calendar's event inspector is one: the agent saw its fields in the tree but could not select its window or type into it ("The editor's window moved or is not an observed window", "Another document window has keyboard focus"), and fell back to detours through other apps to dismiss popovers
  • windowsFor now keeps every level below the Dock (normal, floating, modal panel, utility). Menus, status items and overlays stay excluded, and a window still has to match an Accessibility window to be listed

Type of Change

  • Bug fix

Testing

  • Replayed a recorded Calendar episode through the helper in a macOS VM: the inspector is an AXWindow whose window-list entry is at layer 8, and input into it failed exactly as in the recording
  • Paired benchmark on the 5 Calendar tasks of a native multi-app suite, 2 runs per task per arm, 150-call budget: 10/10 passed vs 9/10 on staging (the failure gave up with "Calendar stopped responding to my controls"); failed tool calls 43 → 23; "no accessible window", editor-not-observed and wrong-window-focus errors 19 → 0; total calls −8%, cost −6% (within noise at this sample size)
  • Helper compiled with swiftc -target arm64-apple-macosx14.0; no TS changes, so lint/audits are unaffected

Checklist

  • Code follows project style guidelines
  • Self-reviewed my changes
  • Tests added/updated and passing (new tests pass the test-audit authoring gate)
  • No new warnings introduced
  • I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

@vercel

vercel Bot commented Nov 18, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated
docs Skipped Skipped Nov 18, 2026 10:37pm UTC

Request Review

@greptile-apps

greptile-apps Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 5/5

[Medium impact] This PR appears safe to merge; no actionable issues were found.

Summary

The helper now includes an app’s floating and panel-level windows below the Dock level, while keeping the existing Accessibility matching and input checks.

  • Computer use can observe floating and panel windows below the Dock level.

Reviews (1) · Last reviewed commit: "fix(computer-use): observe an app's floa..." · Reviewed by Greptile

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 1 file

Confidence score: 5/5

  • Automated review surfaced no issues in the provided summaries.
  • No files require special attention.

Turn on auto-fix | Re-trigger cubic

@waleedlatif1
waleedlatif1 merged commit 99a6c76 into staging Nov 18, 2026
50 checks passed
@waleedlatif1
waleedlatif1 deleted the fix/computer-use-panel-level-windows branch November 18, 2026 23:06

This branch was previously deployed

1 inactive deployment
Preview — 56881e30 Deployed Nov 18, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant