Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions apps/desktop/build/entitlements.mac.plist
Original file line number Diff line number Diff line change
Expand Up @@ -14,5 +14,12 @@
Hardened Runtime denies the camera no matter what the user allowed. -->
<key>com.apple.security.device.camera</key>
<true/>
<!-- Computer Use app commands add confirmed reminders, events and
contacts through EventKit and Contacts; the Hardened Runtime denies
that access without these keys. -->
<key>com.apple.security.personal-information.calendars</key>
<true/>
<key>com.apple.security.personal-information.addressbook</key>
<true/>
</dict>
</plist>
3 changes: 3 additions & 0 deletions apps/desktop/electron-builder.yml
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,9 @@ mac:
# unless the bundle declares why it wants the device.
extendInfo:
NSScreenCaptureUsageDescription: Mothership captures approved app windows to understand and verify computer-use actions.
NSRemindersFullAccessUsageDescription: Mothership adds reminders you confirm and lists your reminder lists so it can file them correctly.
NSCalendarsFullAccessUsageDescription: Mothership adds calendar events you confirm and lists your calendars so it can file them correctly.
NSContactsUsageDescription: Mothership adds contacts and contact groups you confirm.
NSMicrophoneUsageDescription: Sim uses your microphone for voice input in Chat and for meetings you join in the built-in browser.
NSCameraUsageDescription: Sim uses your camera for meetings you join in the built-in browser, such as Google Meet.
# macOS 15 blocks connections to private addresses (self-hosted servers on a
Expand Down
225 changes: 223 additions & 2 deletions apps/desktop/native/computer-use/ComputerUse.swift

Large diffs are not rendered by default.

3 changes: 3 additions & 0 deletions apps/desktop/native/computer-use/Info.plist
Original file line number Diff line number Diff line change
Expand Up @@ -11,5 +11,8 @@
<key>LSMinimumSystemVersion</key><string>14.0</string>
<key>LSUIElement</key><true/>
<key>NSHighResolutionCapable</key><true/>
<key>NSCalendarsFullAccessUsageDescription</key><string>Mothership adds calendar events you confirm and lists your calendars so it can file them correctly.</string>
<key>NSContactsUsageDescription</key><string>Mothership adds contacts and contact groups you confirm.</string>
<key>NSRemindersFullAccessUsageDescription</key><string>Mothership adds reminders you confirm and lists your reminder lists so it can file them correctly.</string>
<key>NSScreenCaptureUsageDescription</key><string>Mothership captures the app window you select to understand and verify computer-use actions.</string>
</dict></plist>
4 changes: 3 additions & 1 deletion apps/desktop/native/computer-use/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,9 @@ Compile each architecture with `swiftc -parse-as-library -O -target arm64-apple-

## Protocol

Persistent newline-delimited JSON over stdin/stdout: `{id,method,params}` returns `{id,result}` or `{id,error:{code,message,dispatchState?}}`. The producer contract is `worker/packages/contracts/src/computer-use.ts` in mothership, copied into the desktop bridge by contract sync. `request_permission` is a local onboarding method, never a model tool; params are `{permission:"accessibility"|"screenCapture"}`. Status/preflight never triggers prompts. Errors go through the protocol, and stdout contains no logs.
Persistent newline-delimited JSON over stdin/stdout: `{id,method,params}` returns `{id,result}` or `{id,error:{code,message,dispatchState?}}`. The producer contract is `worker/packages/contracts/src/computer-use.ts` in mothership, copied into the desktop bridge by contract sync. `request_permission` is a local onboarding method, never a model tool; params are `{permission:"accessibility"|"screenCapture"|"reminders"|"calendars"|"contacts"}`. Status/preflight never triggers prompts; it reports `reminders`, `calendars` and `contacts` as `granted`, `denied` or `not_determined`.

`app_command` takes `{command:{name,...}}` and runs one allowlisted command through EventKit or Contacts, never Apple Events or scripts. Reads: `list_reminder_lists`, `list_calendars` and `list_contact_groups` return `{kind:"command",items:[{id,title,allowsModifications}]}`. Creates return `{kind:"command",created:{id,title,list}}`: `create_reminder` (`title`, `notes`, `due` as `YYYY-MM-DD` or ISO 8601 with an offset, `listId`, `priority` 0/1/5/9), `create_event` (`title`, `start`/`end` as ISO 8601 with offsets or `YYYY-MM-DD` with `allDay`, at most 31 days, `location`, `notes`, `calendarId`, `alertMinutesBefore`), `create_contact_group` (`title`; an existing group with that name is returned instead of a duplicate) and `create_contact` (`givenName`, `familyName`, `organization`, `jobTitle`, up to 5 `emails` and `phones`, `groupId`). Without the matching access a command fails with `reminders_permission_required`, `calendars_permission_required` or `contacts_permission_required` before touching anything. There is no update, delete or search. The desktop service confirms each create with the user before sending it.

The helper observes up to 2000 accessibility nodes (depth 64, queue 4000, traversal deadline 8 seconds, each AX call timeout 250ms). Selected-window and focused-element ancestry are prioritized. The menu bar and menus are read in place, so an open menu's items and their submenus come first; items of closed menus are read last, capped at 80 menu nodes. Electron accessibility is enabled using its documented `AXManualAccessibility` attribute. The helper observes 100 windows and 1000 apps. Requests are bounded to 128 KiB during ingestion. Text/value input is capped at 32000 UTF-16 units. State references are per-process, per-app, expire after 60 seconds, and are consumed by every action that reaches the app; a request that fails before sending anything reports `dispatchState: "not_started"` and keeps them valid. An Accessibility failure counts as sent unless Accessibility rejected the request before the app saw it (unsupported attribute or action, illegal argument). Restarts invalidate all references. Screen captures exclude window shadows, preserve the selected window’s coordinate geometry, and are capped to 1600 pixels on the longest side and 8 MiB encoded. Complete JSON replies are capped at 16 MiB: oversized observations omit screenshots first, then truncate accessibility nodes, while preserving action outcomes. Screenshots are optional and failure is reported in `screenshotError` without discarding accessibility state. macOS 26 uses the screenshot-specific capture API; the macOS 14/15 API path includes only the selected window in a display filter and crops its exact frame. That older path requires the window to fit within one display and reports `capture_geometry_unavailable` otherwise.

Expand Down
18 changes: 18 additions & 0 deletions apps/desktop/native/computer-use/tests/NativeTests.swift
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,24 @@ import AppKit
precondition(isProtectedApp(bundle), bundle)
}
for bundle in ["com.apple.TextEdit", "com.apple.finder", "com.apple.Safari"] { precondition(!isProtectedApp(bundle), bundle) }
let dateOnly = try dueComponents("2027-05-04")
precondition(dateOnly.year == 2027 && dateOnly.month == 5 && dateOnly.day == 4 && dateOnly.hour == nil)
let timed = try dueComponents("2027-05-04T09:30:00Z")
precondition(timed.hour != nil && timed.minute != nil)
for due in ["2027-02-30", "2027-05-04T09:30:00", "May 4", "2027-5-4"] {
expectError("invalid_arguments") { _ = try dueComponents(due) }
}
expectError("unknown_command") { _ = try runAppCommand(AppCommand(name: "run_script")) }
expectError("invalid_arguments") { _ = try runAppCommand(AppCommand(name: "create_reminder", title: " ")) }
expectError("invalid_arguments") { _ = try runAppCommand(AppCommand(name: "create_reminder", title: "Pay rent", priority: 3)) }
expectError("invalid_arguments") { _ = try eventDate("2027-06-09T07:15:00", "Start", allDay: false) }
expectError("invalid_arguments") { _ = try eventDate("2027-06-09T07:15:00+01:00", "Start", allDay: true) }
precondition(try eventDate("2027-06-09T07:15:00+01:00", "Start", allDay: false) == Date(timeIntervalSince1970: 1_812_521_700))
expectError("invalid_arguments") { _ = try runAppCommand(AppCommand(name: "create_event", title: "Flight", start: "2027-06-09T08:10:00Z", end: "2027-06-09T07:15:00Z")) }
expectError("invalid_arguments") { _ = try runAppCommand(AppCommand(name: "create_event", title: "Trip", start: "2027-06-01T00:00:00Z", end: "2027-08-01T00:00:00Z")) }
expectError("invalid_arguments") { _ = try runAppCommand(AppCommand(name: "create_contact", jobTitle: "Engineer")) }
expectError("invalid_arguments") { _ = try runAppCommand(AppCommand(name: "create_contact", givenName: "Ana", emails: ["not-an-email"])) }
expectError("invalid_arguments") { _ = try runAppCommand(AppCommand(name: "create_contact_group", title: " ")) }
var queue = AXTraversalQueue<String>(limit: 1000)
queue.append(AXTraversalEntry(element: "menu", parent: nil, depth: 0, menu: true, priority: false))
queue.append(AXTraversalEntry(element: "content", parent: nil, depth: 0, menu: false, priority: false))
Expand Down
83 changes: 83 additions & 0 deletions apps/desktop/src/main/computer-use/service.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,12 @@ function setup(enabled = true, askForApps = true) {
kind: 'apps',
apps: [{ bundleId: 'com.example.Fixture', name: 'Fixture', pid: 42, isActive: false }],
}
if (method === 'app_command')
return {
kind: 'command',
command: 'create_reminder',
created: { id: 'r1', title: 'Pay rent', list: 'Reminders' },
}
if (method === 'get_app_state')
return {
kind: 'state',
Expand All @@ -48,6 +54,7 @@ function setup(enabled = true, askForApps = true) {
})
const native = { request, stop: vi.fn() }
const approveApp = vi.fn(async (): Promise<'always' | 'deny' | 'once'> => 'always')
const confirmAppCommand = vi.fn(async (_request: { appName: string; summary: string }) => true)
const onActivity = vi.fn()
const openPermissionSettings = vi.fn(async () => {})
const setStopShortcutActive = vi.fn((active: boolean) => active)
Expand All @@ -56,6 +63,7 @@ function setup(enabled = true, askForApps = true) {
native,
supported: true,
approveApp,
confirmAppCommand,
onActivity,
setStopShortcutActive,
openPermissionSettings,
Expand All @@ -77,6 +85,7 @@ function setup(enabled = true, askForApps = true) {
config,
native,
approveApp,
confirmAppCommand,
onActivity,
setStopShortcutActive,
openPermissionSettings,
Expand Down Expand Up @@ -116,6 +125,80 @@ describe('native computer use authority and lifecycle', () => {
expect(native.request.mock.calls.map(([method]) => method)).toEqual(['list_apps'])
})

it('confirms each app-command write before the helper runs it, and never observes after', async () => {
const { service, native, confirmAppCommand } = setup()
const create = {
action: 'app_command',
command: { name: 'create_reminder', title: 'Pay rent', due: '2027-05-04', priority: 1 },
}
confirmAppCommand.mockResolvedValueOnce(false)
await expect(service.execute('declined', 'chat', create)).rejects.toMatchObject({
details: { code: 'user_declined', dispatchState: 'not_started' },
})
expect(native.request.mock.calls.map(([method]) => method)).not.toContain('app_command')
expect(confirmAppCommand.mock.calls[0]?.[0].summary).toContain('Priority: High')

await expect(service.execute('confirmed', 'chat', create)).resolves.toMatchObject({
kind: 'command',
created: { id: 'r1' },
})
const methods = native.request.mock.calls.map(([method]) => method)
expect(methods.filter((method) => method === 'app_command')).toHaveLength(1)
expect(methods).not.toContain('get_app_state')
})

it.each([
{
name: 'create_event',
title: 'TP 1947: LIS to OPO',
start: '2027-06-09T07:15:00+01:00',
end: '2027-06-09T08:10:00+01:00',
},
{ name: 'create_contact_group', title: 'Hires Q1 2027' },
{ name: 'create_contact', givenName: 'Ana', emails: ['ana@example.com'] },
])('asks the user before every $name write', async (command) => {
const { service, native, confirmAppCommand } = setup()
confirmAppCommand.mockResolvedValueOnce(false)
await expect(
service.execute(`declined-${command.name}`, 'chat', { action: 'app_command', command })
).rejects.toMatchObject({ details: { code: 'user_declined' } })
expect(confirmAppCommand).toHaveBeenCalledOnce()
expect(native.request.mock.calls.map(([method]) => method)).not.toContain('app_command')
})

it('refuses an app command when the user denies the app, before asking to confirm', async () => {
const { service, native, approveApp, confirmAppCommand } = setup()
approveApp.mockResolvedValue('deny')
await expect(
service.execute('denied', 'chat', {
action: 'app_command',
command: { name: 'create_reminder', title: 'Pay rent' },
})
).rejects.toThrow('denied')
expect(confirmAppCommand).not.toHaveBeenCalled()
expect(native.request.mock.calls.map(([method]) => method)).not.toContain('app_command')
})

it('lists reminder lists without a write confirmation', async () => {
const { service, native, confirmAppCommand } = setup()
native.request.mockImplementation(async (method: string) =>
method === 'app_command'
? {
kind: 'command',
command: 'list_reminder_lists',
items: [{ id: 'l1', title: 'Sprint 14', allowsModifications: true }],
}
: { kind: 'apps', apps: [] }
)
await expect(
service.execute('lists', 'chat', {
action: 'app_command',
command: { name: 'list_reminder_lists' },
})
).resolves.toMatchObject({ items: [{ title: 'Sprint 14' }] })
expect(confirmAppCommand).not.toHaveBeenCalled()
})

it('binds observed elements to their originating chat', async () => {
const { state, click, native } = setup()
await state('owner')
Expand Down
Loading
Loading