Skip to content

improvement(computer-use): list only what changed in the observation after an action - #10126

Merged
waleedlatif1 merged 3 commits into
stagingfrom
improvement/computer-use-delta-observations
Nov 17, 2026
Merged

waleedlatif1 merged 3 commits into
stagingfrom
improvement/computer-use-delta-observations

Conversation

@waleedlatif1

@waleedlatif1 waleedlatif1 commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

  • After an action, the computer-use observation now lists only the accessibility rows added or changed since the snapshot the action used, plus removedElementIds and changedSince; treeFormat explains this and points to get_app_state for the whole tree
  • Built on stable element ids (improvement(computer-use): keep an element's id stable across observations #10046): the projection remembers what the model was shown per snapshot (64-entry LRU) and folds change lists into it
  • Falls back to the whole tree when the base snapshot is unknown (e.g. after reload), the app or window changed, more than half the rows changed, the change list would not fit the budget, or after 4 change lists in a row (so a condensed context recovers)
  • Only on desktops that advertise computerUse.stableElementIds (new optional bridge flag set by the preload); older shells keep whole trees
  • Explicit get_app_state reads are unchanged
  • Also sends a whole tree when an action changed nothing visible (an empty change list read as a failed action and sent the model on detours) or when a window, sheet or popover opened or closed since the base. Added after the benchmark below, from its traces: 38 of 463 change lists were empty and 37 had a different window set; the one delta-related loss (a Calendar flow, 49 calls vs 27) followed empty change lists after clicks

Type of Change

  • Improvement

Testing

  • New projection tests: whole trees without stable ids or with an unknown base; change list with changed/added rows and removed ids; folding across chained change lists; whole tree after the chain limit, on a window change, and when most of the tree changed. Each guard was reverted and its test went red
  • check:desktop-bridge passes (bridge change is backward compatible)
  • Paired benchmark, 99 multi-app macOS tasks per arm, same agent and helper, only the projection differs:
whole trees change lists
cost / episode $0.743 $0.535 (−28%)
wall / episode 65.5s 56.9s (−13%)
observation chars (median) 18,845 9,069
post-action observations sent as change lists 0% 73.5%
passed 86 82 (McNemar p=0.22)
actions on a removed id 0 0
  • Confirmation pair after adding the empty-change and window-change fallbacks (same setup): passed 84 vs 83 (McNemar p=1.0), cost/episode $0.712 → $0.598 (−16%), wall 56.4s → 54.1s, ~1 extra call/episode, 64% of post-action observations sent as change lists, 0 actions on a removed id; the Calendar flow that regressed now passes in 31 calls (whole trees 29)
  • Review fixes since then (partial native reads send a whole tree, rows folded into a layout wrapper are reported as no longer shown, the cache stores rows exactly as sent) only make change lists more conservative
  • First pair, before those fallbacks: Of the 5 tasks only the whole-tree arm passed, 4 trace to live-web lookup variance or a checker scoring a correct-looking answer 0; one (a long Calendar editing flow) took 49 calls instead of 27 and is the delta-related regression to watch

Checklist

  • Code follows project style guidelines
  • Self-reviewed my changes
  • Tests added/updated and passing (new tests pass the test-audit authoring gate)
  • No new warnings introduced
  • I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

@vercel

vercel Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated
docs Skipped Skipped Nov 17, 2026 1:53pm UTC

Request Review

@greptile-apps

greptile-apps Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 5/5

[Medium impact] The PR appears safe to merge; all three previous concerns are addressed.

Summary

The PR sends changed accessibility rows after actions on desktops with stable element IDs. It keeps full-tree reads as a fallback.

  • Computer-use actions can show only the accessibility rows that changed.

All three previous threads are addressed. They were unnumbered, so they have no entries in previousFindings. No new actionable issues were found.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Action returns an observation] --> B{Known base and stable IDs?}
  B -->|No| F[Send full projected tree]
  B -->|Yes| C{Change list is safe and fits?}
  C -->|No| F
  C -->|Yes| D[Send changed rows and removed IDs]
  D --> E[Cache what the model knows]
  F --> E
Loading

Reviews (3) · Last reviewed commit: "report folded rows, send whole trees for..." · Reviewed by Greptile

@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@greptile

@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@cubic-dev-ai review this PR

@cubic-dev-ai

cubic-dev-ai Bot commented Nov 17, 2026

Copy link
Copy Markdown
Contributor

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 5 files

Confidence score: 5/5

  • Automated review surfaced no issues in the provided summaries.
  • No files require special attention.

Turn on auto-fix | Re-trigger cubic

@waleedlatif1
waleedlatif1 merged commit d697c4a into staging Nov 17, 2026
153 of 157 checks passed
@waleedlatif1
waleedlatif1 deleted the improvement/computer-use-delta-observations branch November 17, 2026 18:10

This branch was previously deployed

1 inactive deployment
Preview — e40542bb Deployed Nov 17, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant