{"id":1033,"date":"2026-07-01T13:26:43","date_gmt":"2026-07-01T13:26:43","guid":{"rendered":"https:\/\/feedsta.ai\/blog\/gemini-2-5-computer-use-model-browser-ui-automation\/"},"modified":"2026-07-19T06:43:48","modified_gmt":"2026-07-19T06:43:48","slug":"gemini-2-5-computer-use-model-browser-ui-automation","status":"publish","type":"post","link":"https:\/\/feedsta.ai\/blog\/gemini-2-5-computer-use-model-browser-ui-automation\/","title":{"rendered":"Gemini 2.5 Computer Use Model for Browser and UI Automation"},"content":{"rendered":"

Google has published documentation for a Gemini 2.5 Computer Use model, a variant of Gemini 2.5 that interacts with graphical user interfaces by looking at a screen and performing actions on it. Available through the Gemini API as gemini-2.5-computer-use-preview-10-2025<\/code>, the model accepts screenshots and a natural-language instruction and returns suggested UI actions for developers to execute in a browser loop.<\/p>\n

Google introduces a computer-use model in the Gemini API<\/h2>\n

According to Google’s developer documentation, the model accepts screenshots of an interface along with a natural-language instruction, then returns suggested actions such as clicks, typing, scrolling, and navigation. Developers can execute those actions through a Playwright-based browser loop that returns the updated screenshot for the next turn.<\/p>\n

How does the model work?<\/h3>\n

Each request to the model includes the conversation history, the user’s request, and one or more screenshots. The response is a list of function calls representing UI operations:<\/p>\n

    \n
  • click_at<\/code> with x and y coordinates<\/li>\n
  • type_text<\/code> with content to enter into a focused field<\/li>\n
  • key_combination<\/code> for keyboard shortcuts<\/li>\n
  • scroll<\/code>, scroll_to<\/code>, and hover_at<\/code> for navigation<\/li>\n
  • go_back<\/code>, go_forward<\/code>, refresh<\/code>, and search<\/code> for browser controls<\/li>\n
  • wait<\/code> and screenshot<\/code> for pacing and additional visual context<\/li>\n<\/ul>\n

    After an action runs against the browser, a fresh screenshot is added to the next request so the model can observe the result and decide what to do next.<\/p>\n

    What tooling and integration does Google provide?<\/h3>\n

    Google provides reference Python code built on Playwright for orchestrating the browser loop. The documentation also describes an excluded_predefined_functions<\/code> parameter that lets developers restrict which actions the model can emit, for example blocking drag operations in environments where they are not supported.<\/p>\n

    Under the hood, the model uses Gemini’s standard structured output feature, constrained to a predefined set of UI function calls. A safety system runs in parallel: it evaluates the user’s prompt, the model’s proposed actions, and the screen state, and can short-circuit the loop before an unsafe action executes.<\/p>\n

    What is the model available for and where?<\/h3>\n

    The model is positioned for tasks such as filling forms, navigating multi-step web workflows, and testing user interfaces from a screenshot-driven prompt. Google notes that the preview is intended for browser-based automation first, with broader computer control use cases listed as a longer-term direction in the documentation.<\/p>\n

    Developers can experiment with the model through the Gemini API in Google AI Studio or Vertex AI, and the documentation includes sample code, configuration guidance, and links to the supported SDKs.<\/p>\n

    FAQ<\/h2>\n

    What is the Gemini 2.5 Computer Use model?<\/h3>\n

    It is a variant of Gemini 2.5 documented by Google for interacting with graphical user interfaces the way a person would, by looking at a screen and performing actions on it. It is available in the Gemini API as gemini-2.5-computer-use-preview-10-2025<\/code>.<\/p>\n

    How does the model drive a browser?<\/h3>\n

    Each request includes the conversation history, the user’s request, and one or more screenshots. The model returns function calls such as click_at<\/code>, type_text<\/code>, key_combination<\/code>, scroll<\/code>, go_back<\/code>, and screenshot<\/code>, which developers execute through a Playwright-based browser loop. A fresh screenshot is then sent back so the model can decide the next step.<\/p>\n

    Where can developers try the model and what use cases does Google target?<\/h3>\n

    Developers can experiment through the Gemini API in Google AI Studio or Vertex AI. Google positions the preview for browser-based tasks such as filling forms, navigating multi-step web workflows, and screenshot-driven UI testing, with broader computer control described as a longer-term direction.<\/p>\n