"Computer Use Agents: From SFT to Classic RL", Maksym Shamrai

"Computer Use Agents: From SFT to Classic RL", Maksym Shamrai

• 2.
  ● Research Scientist@ MacPaw AIR ● Applied Mathematics PhD Candidate @ Institute of Mathematics of NAS of Ukraine ● Lecturer @ Kyiv Academic University Хто я?
• 3.
  Computer Use problem 01 CommercialCUA 02 Open CUA 03 04 RL Evaluation and Benchmarks 05
• 4.
  Computer Use
• 5.
  https://arxiv.org/pdf/2501.12326
• 6.
  1. Feeding screenshotof a current state to CUA https://platform.openai.com/docs/guides/tools-computer-use
• 7.
  2. CUA returnsan action it considers appropriate https://platform.openai.com/docs/guides/tools-computer-use
• 8.
  3. Action isexecuted in an environment https://platform.openai.com/docs/guides/tools-computer-use
• 9.
  4. New screenshotis taken https://platform.openai.com/docs/guides/tools-computer-use
• 10.
  Markov decision process https://www.ibm.com/think/topics/reinforcement-learning
• 11.
  ● Multimodal LargeLanguage Model ● Multi-Agent System (bunch of MLLMs) What is the agent? ● Screenshot ● Metadata (e.g. accessibility tree, list of active applications etc.) What is the state?
• 12.
  ● Virtual Machine(safety considerations) ● Actual OS (in Docker, VMWare, UTM etc.) ● or dedicated (fake) one What is the environment? https://github.com/e2b-dev/desktop/
• 13.
  ● Anything youwant to allow and can develop ● Typically you want your agent to be able to click and type at least ● It is called action space What is the action?
• 14.
  ● In ComputerUse there is no direct reward ● It could be handcrafted evaluation script or automated LLM judge ● This makes it difficult but not impossible What is the reward?
• 15.
  Browser Use? PhoneUse? etc. ● Same but different ● Have to use corresponding environment and action space ○ Browser – Playwright, Selenium, BrowserGym. ○ Android – AndroidEnv. ● Typically easier than Computer Use (smaller state space, more data)
• 16.
  Commercial Computer UseAgents
• 17.
  Claude Computer use ●Out since October 2024 ● Still beta ● Regular Claude Sonnet 3.5/4 with the tool ● https://docs.anthropic.com/en/docs/agents-and-tools/tool -use/computer-use-tool
• 18.
  OpenAI Computer use ●Out since January 2025 ● Also beta ● GPT-4o or o3 ● https://platform.openai.com/docs/guides/tools-computer- use
• 19.
  OpenAI CUA demohttps://github.com/mshamrai/openai-cua-gradio
• 20.
  Training
• 21.
  Recap: Training pipelineof LLMs https://cameronrwolfe.substack.com/p/demystifying-reasoning-models
• 22.
  Recap: Training pipelineof LLMs https://cameronrwolfe.substack.com/p/demystifying-reasoning-models
• 23.
  Trajectories dataset Task: Reply toJohn Doe that I like the agenda Initial state
• 24.
  Trajectories dataset click on“John Doe”
• 25.
  Trajectories dataset click on“Reply”
• 26.
  Trajectories dataset type …
• 27.
  Trajectories dataset ● Trajectoryis a list of screenshots (states) and actions that correspond to the successful completion of a task ● trajectory = (s0, a0, s1, a1, …) ● So, dataset consists of pairs (task, trajectory) ● Better to add “thoughts” before action
• 28.
  Supervised Fine-Tuning (SFT) ●From each record compose a “conversation” ● Train a LLM using SFT
• 29.
  Open Computer UseMLLMs
• 30.
  CogAgent ● Out sinceDecember 2023 ● New version released in December 2024 ● Still not as good as others https://arxiv.org/abs/2312.08914
• 31.
  Agent S2 ● Outsince April 2025 ● Multi-agent framework ● Best to use commercial LLMs as a planner https://arxiv.org/pdf/2504.00906
• 32.
  ● Out sinceJanuary 2025 ● New version released in April 2025 (current SOTA) https://arxiv.org/pdf/2501.12326 https://seed-tars.com/1.5
• 33.
  UI-TARS https://arxiv.org/pdf/2409.12191 ● Based onQwen2-VL ● New version on Qwen2.5-VL ● Trained on 50B tokens of manual annotated data (not open sourced)
• 34.
  RL in ComputerUse
• 35.
  Recap: Reinforcement Learningfrom Human Feedback https://huggingface.co/blog/rlhf
• 36.
  PPO ● Introduced byOpenAI in 2017 ● Actor-Critic algorithm ● Still a good baseline for any RL task https://openai.com/index/openai-baselines-ppo/
• 37.
  DPO https://arxiv.org/pdf/2305.18290
• 38.
  DPO ● Introduced in2023 ● Designed specifically for LLMs ● Does not require a reward or value model ● Trains on pairs of outputs where one is a winner and the other is a loser ● Derived in closed-form ● Slightly less effective than PPO, but cheaper and offline
• 39.
  SFT vs DPO https://arxiv.org/pdf/2501.12326
• 40.
  GRPO https://arxiv.org/pdf/2402.03300
• 41.
  GRPO ● Introduced byDeepSeek in 2024 ● Used to train a reasoning model ● Modification of PPO ● Without value model (critic) ● Rule-based reward (only works with automatically evaluated problems) ● Needs less resources than PPO but still online
• 42.
  ARPO https://arxiv.org/pdf/2505.16282
• 43.
  ARPO https://arxiv.org/pdf/2505.16282
• 44.
  ARPO https://arxiv.org/pdf/2505.16282
• 45.
  “Classic” RL
• 46.
  Android in theWild (AitW) https://github.com/google-research/google-research/tree/master/android_in_the_wild
• 47.
  https://arxiv.org/pdf/2406.11896 ● NeurIPS 2024 ●Android Use?
• 48.
  DigiRL ● Off-Policy RLvia Advantage-Weighted Regression ● Offline + Online RL ● Automatic evaluation with Gemini
• 49.
  ● Out sinceFebruary 2025 ● Android Use! https://arxiv.org/pdf/2502.15760
• 50.
  Digi-Q https://arxiv.org/pdf/2502.15760 ● Actor-Critic ● Offline!
• 51.
  Digi-Q https://arxiv.org/pdf/2502.15760
• 52.
  DigiRL & Digi-Q https://digirl-agent.github.io/DigiQ-agent.github.io/
• 53.
  Evaluation and Benchmarks
• 54.
  Evaluation ● Use pre-recordedtrajectories ● Compare actions (perplexity, rule- based) Offline ● Use the environment and run the agent ● Evaluate the execution using rules (or automatically) Online Pros: ● Cheap and fast (no need to run the environment) Cons: ● Unreliable due to the dynamic nature of the environment Pros: ● Reliable metric Cons: ● Expensive and slow ● Have to run several times
• 55.
  OSWorld https://os-world.github.io/ ● Online benchmark ●369 manually labeled computer tasks with automatic evaluation code ● Ubuntu, Windows, and macOS
• 56.
  OSWorld Tasks: ● Can youenable the 'Do Not Track' feature in Chrome to enhance my online privacy? ● Could you turn my image into CYMK mode? ● Fill all the blank cells in B1:E30 with the value in the cell above it. ● Color the first three textboxes on slide 1 yellow, red, and green, respectively, in top-to-bottom order. Use exactly these colors—no variations (e.g., no dark red, light green, etc.). ● Make the line spacing of first two paragraph into double line spacing ● I need to include the experiment results from "~/Documents/awesome-desktop/expe-results.xlsx" into the currently writing report. Specifically, extract the results of GPT-4 and insert a table into the "Main Results" section of my report. ● I click in terminal: terminal->132x43 to change terminal size but after each reboot terminal size is set to default setting and I have to change it again. Help me set it permanently ● Please help me install an extension in VS Code from a local VSIX file "/home/user/test.vsix".
• 57.
  OSWorld https://os-world.github.io/
• 58.
  ScreenSpot Pro https://huggingface.co/datasets/likaixin/ScreenSpot-Pro ● High-resolution ●Grounding (1 action)
• 59.
  ScreenSpot Pro https://likaixin2000.github.io/papers/ScreenSpot_Pro.pdf
• 60.
  ScreenSpot Pro https://gui-agent.github.io/grounding-leaderboard/
• 61.
  Q&A