Refactor screenshot masking to reduce code duplication

[shrey150]

Summary

Eliminates code duplication in screenshot masking functionality by extracting shared utilities and consolidating helper methods:

• Extracted selectorsToLocators() and DEFAULT_MASK_COLOR to shared screenshotUtils.ts
• Unified ToolMaskConfig as a type alias for AgentMaskConfig
• Consolidated screenshot capture logic in V3CuaAgentHandler into reusable helpers

Test Plan

• ✅ TypeScript compilation passes
• ✅ ESLint and Prettier checks pass
• ✅ Public API type tests pass
• ✅ No functional changes to screenshot masking behavior

🤖 Generated with Claude Code

---

Summary by cubic

Unifies screenshot masking behind shared helpers and a single mask config, removing duplicated logic across tools and handlers. Defaults stay the same; masking is optional and off unless provided.

• Refactors
  • Added AgentMaskConfig and mask option on execute/stream.
  • Threaded mask config from V3AgentHandler into createAgentTools and all relevant tools (screenshot and vision).
  • waitAndCaptureScreenshot now accepts a number or { delayMs, mask }, where mask = { selectors, color }.
  • Introduced captureWithMask and moved selectorsToLocators and DEFAULT_MASK_COLOR to screenshotUtils; centralized apply/cleanup flow.
  • Both CUA and V3 handlers now use captureWithMask; CUA updates screenshot dimensions via a helper.

^{Written for commit c1d9ad6. Summary will update on new commits.}

[changeset-bot]

⚠️ No Changeset found

Latest commit: c1d9ad6

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[greptile-apps]

Greptile Summary

This PR successfully consolidates screenshot masking functionality by extracting shared utilities (selectorsToLocators, DEFAULT_MASK_COLOR) into screenshotUtils.ts and unifying the ToolMaskConfig type as an alias for AgentMaskConfig.

Key improvements:

• Eliminated code duplication across 8+ tool files by centralizing masking logic
• Maintained backward compatibility in waitAndCaptureScreenshot() by supporting both legacy number parameter and new options object
• Added proper cleanup handling with try/finally blocks to ensure mask overlays are removed
• Refactored V3CuaAgentHandler screenshot capture into reusable helper methods (captureScreenshotWithMask, updateScreenshotDimensions)

Minor observation:

• clickAndHold tool accepts maskConfig parameter but doesn't capture screenshots, making the parameter unused (though this maintains API consistency)

Confidence Score: 5/5

• This PR is safe to merge with minimal risk
• The refactoring is well-executed with proper abstraction, maintains backward compatibility, includes appropriate error handling with cleanup blocks, and passes all type checks. The changes are purely structural with no functional behavior changes to screenshot masking.
• No files require special attention

Important Files Changed

Filename	Overview
packages/core/lib/v3/understudy/screenshotUtils.ts	Added selectorsToLocators() and DEFAULT_MASK_COLOR utilities for screenshot masking
packages/core/lib/v3/agent/utils/screenshotHandler.ts	Enhanced waitAndCaptureScreenshot() with masking support and backward compatibility
packages/core/lib/v3/handlers/v3CuaAgentHandler.ts	Extracted screenshot capture logic into reusable helper methods with masking support
packages/core/lib/v3/agent/tools/clickAndHold.ts	Added unused maskConfig parameter without screenshot capture functionality

Sequence Diagram

sequenceDiagram
    participant Agent as Agent Handler
    participant Tool as Vision Tool
    participant Handler as Screenshot Handler
    participant Utils as Screenshot Utils
    participant Page as Browser Page

    Agent->>Agent: Set maskConfig from execute options
    Agent->>Tool: Pass maskConfig to tool
    Tool->>Handler: Call waitAndCaptureScreenshot(page, options)
    Handler->>Utils: selectorsToLocators(page, maskConfig.selectors)
    Utils-->>Handler: Return Locator[]
    Handler->>Utils: applyMaskOverlays(locators, color)
    Utils->>Page: Inject mask overlay elements
    Utils-->>Handler: Return cleanup function
    Handler->>Page: page.screenshot()
    Page-->>Handler: Buffer
    Handler->>Utils: runScreenshotCleanups(cleanupTasks)
    Utils->>Page: Remove mask overlay elements
    Handler-->>Tool: base64 screenshot
    Tool-->>Agent: Tool result with screenshot

Loading

[greptile-apps]

_{15 files reviewed, 1 comment}

_{Edit Code Review Agent Settings | Greptile}

[cubic-dev-ai]

No issues found across 15 files