Repository navigation
fix(mothership): skip the 30s workflow pickup wait when no chat view is reading #10148
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
61 changes: 61 additions & 0 deletions
61
apps/sim/lib/mothership/request/session/viewer.integration.ts
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,61 @@ | ||
| /** | ||
| * Chat stream reader presence against real Redis: a reader attached on one instance must be | ||
| * visible to the instance dispatching that turn's workflow tools, and must stop counting the | ||
| * moment it leaves, so a turn nobody reads no longer waits out the full pickup grace. | ||
| */ | ||
| import { afterAll, describe, expect, it, vi } from 'vitest' | ||
|
|
||
| const { redisUrl } = await vi.hoisted(async () => { | ||
| const { readTestRedisUrl } = await import('@sim/db/testing/test-infrastructure') | ||
| const url = readTestRedisUrl() | ||
| if (url) process.env.REDIS_URL = url | ||
| return { redisUrl: url } | ||
| }) | ||
|
|
||
| import { sleep } from '@sim/utils/helpers' | ||
| import { generateId } from '@sim/utils/id' | ||
| import { closeRedisConnection } from '@/lib/core/config/redis' | ||
| import { hasStreamViewer, holdStreamViewer } from '@/lib/mothership/request/session/viewer' | ||
|
|
||
| describe.runIf(Boolean(redisUrl))('chat stream reader presence', () => { | ||
| afterAll(async () => { | ||
| await closeRedisConnection() | ||
| }) | ||
|
|
||
| it('counts a reader as attached once its hold resolves', async () => { | ||
| const streamId = generateId() | ||
| const release = await holdStreamViewer(streamId) | ||
| try { | ||
| expect(await hasStreamViewer(streamId)).toBe(true) | ||
| } finally { | ||
| release() | ||
| } | ||
| }) | ||
|
|
||
| it('sees no reader on a stream nobody attached to', async () => { | ||
| expect(await hasStreamViewer(generateId())).toBe(false) | ||
| }) | ||
|
|
||
| it('keeps counting a stream while another reader stays attached', async () => { | ||
| const streamId = generateId() | ||
| const leaving = await holdStreamViewer(streamId) | ||
| const staying = await holdStreamViewer(streamId) | ||
| try { | ||
| leaving() | ||
| await expect.poll(() => hasStreamViewer(streamId)).toBe(true) | ||
| } finally { | ||
| staying() | ||
| } | ||
| }) | ||
|
|
||
| it('stops counting a reader as soon as it leaves, and never re-marks it', async () => { | ||
| const streamId = generateId() | ||
| const release = await holdStreamViewer(streamId) | ||
| release() | ||
| await expect.poll(() => hasStreamViewer(streamId)).toBe(false) | ||
| // Longer than one refresh: a reader still refreshing would have re-marked itself by now. | ||
| await sleep(3_500) | ||
|
|
||
| expect(await hasStreamViewer(streamId)).toBe(false) | ||
| }) | ||
| }) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,82 @@ | ||
| import { createLogger } from '@sim/logger' | ||
| import { toError } from '@sim/utils/errors' | ||
| import { generateId } from '@sim/utils/id' | ||
| import { getRedisClient } from '@/lib/core/config/redis' | ||
|
|
||
| /** | ||
| * Which chat streams have a reconnected reader attached right now, on any instance. | ||
| * | ||
| * A chat view starts client-routed workflow tools from the stream it reads. The original send | ||
| * knows in-process whether its own response is still attached; a view that re-attached through | ||
| * the reconnect route (a reload, a chat switch back, another tab) may sit on another instance, | ||
| * so each attachment holds its own entry here, scored by when it lapses, for as long as it | ||
| * stays attached. Leaving removes the entry at once; a reader that dies without leaving lapses. | ||
| */ | ||
|
|
||
| const logger = createLogger('SessionStreamViewer') | ||
|
|
||
| /** Long enough that one late refresh never makes an attached reader look gone. */ | ||
| const STREAM_VIEWER_TTL_MS = 10_000 | ||
| const STREAM_VIEWER_REFRESH_MS = 3_000 | ||
|
|
||
| function getViewersKey(streamId: string) { | ||
| return `mothership_stream:${streamId}:viewers` | ||
| } | ||
|
|
||
| async function markStreamViewer(streamId: string, attachmentId: string): Promise<void> { | ||
| const redis = getRedisClient() | ||
| if (!redis) return | ||
| const key = getViewersKey(streamId) | ||
| await redis | ||
| .multi() | ||
| .zadd(key, Date.now() + STREAM_VIEWER_TTL_MS, attachmentId) | ||
| .pexpire(key, STREAM_VIEWER_TTL_MS) | ||
| .exec() | ||
| } | ||
|
|
||
| async function unmarkStreamViewer(streamId: string, attachmentId: string): Promise<void> { | ||
| const redis = getRedisClient() | ||
| if (!redis) return | ||
| await redis.zrem(getViewersKey(streamId), attachmentId) | ||
| } | ||
|
|
||
| function warnMarkFailed(streamId: string, error: unknown) { | ||
| logger.warn('Could not record a chat stream reader', { | ||
| streamId, | ||
| error: toError(error).message, | ||
| }) | ||
| } | ||
|
|
||
| /** | ||
| * Records a reader attached to the stream until the returned release runs. Resolves once the | ||
| * first mark is written, so the reader counts before it reads its first event. | ||
| */ | ||
| export async function holdStreamViewer(streamId: string): Promise<() => void> { | ||
| const attachmentId = generateId() | ||
| const refresh = () => | ||
| markStreamViewer(streamId, attachmentId).catch((error) => warnMarkFailed(streamId, error)) | ||
| await refresh() | ||
| const interval = setInterval(refresh, STREAM_VIEWER_REFRESH_MS) | ||
| return () => { | ||
| clearInterval(interval) | ||
| unmarkStreamViewer(streamId, attachmentId).catch((error) => warnMarkFailed(streamId, error)) | ||
| } | ||
| } | ||
|
|
||
| /** | ||
| * Whether a reconnected reader is attached to the stream. Answers true whenever it cannot | ||
| * tell (no Redis, a failed read), so an unknown never cuts a present view's pickup window. | ||
| */ | ||
| export async function hasStreamViewer(streamId: string): Promise<boolean> { | ||
| const redis = getRedisClient() | ||
| if (!redis) return true | ||
| try { | ||
| return (await redis.zcount(getViewersKey(streamId), Date.now(), '+inf')) > 0 | ||
| } catch (error) { | ||
| logger.warn('Could not read whether a chat stream reader is attached', { | ||
| streamId, | ||
| error: toError(error).message, | ||
| }) | ||
| return true | ||
| } | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.