Our Open-Source RPA Blog

Run Anthropic Computer Use in Your Web Browser

Fri, 05 Dec 2025 00:00:00 +0100

We’ve integrated Anthropic Computer Use into the Ui.Vision browser extension. This gives everyone the option to use/demo Anthropic Computer Use directly in their browser - no Docker required. You only need to install the open-source Ui.Vision browser extension for Chrome, Edge, or Firefox.

Step-by-Step Instructions

Install the Ui.Vision browser extension for Chrome, Edge, or Firefox
Install the XModule native app (available for Windows, Mac, and Linux). This app allows the browser extension to simulate mouse clicks and keyboard typing
Enter your Anthropic API key into the Ui.Vision extension. That’s all - you’re ready to go!

Computer Use Tutorial

To get started, we recommend trying one of the pre-made Computer Use macros. You’ll find them at the top of the macro list in Ui.Vision. Macros that use Computer Use start with “CU_”.

Computer Use (inside Ui.Vision) playing TicTacToe in the browser:

(Soon:) If you want to have a chat-like conversation with Claude Computer Use, open the sidebar and select the AI tab. Then start prompting.

Desktop Automation

Since Ui.Vision can handle both browser and desktop automation, you can also demo Anthropic Claude Computer Use directly on your desktop. To do so, go to the Ui.Vision settings and switch from browser automation to desktop automation.

Anthropic Computer Use Pricing

The Ui.Vision browser extension is open-source and free for everyone to use. The only costs involved are the Anthropic API credits. For example, running the PlayTicTacToe demo macro will cost approximately $0.30 in API usage credits.

Computer Use Insights from Our Experience

Here are some insights we’ve gained from using Computer Use:

Computer Use appears to be trained to move the mouse before clicking. However, this is often unnecessary. Tell it “No mouse movements, only clicks.” This simple tweak can halve the automation time and API cost!
Keep the browser viewport (for browser automation) or the desktop screen size (for desktop automation) as small as possible. The smaller the screenshot area, the faster and cheaper the API response. Smaller screen sizes also improve the accuracy of the returned x,y values.
Shorter prompts often work better than longer ones. Ironically, if you ask Claude for prompting help, it will write long and often overly detailed prompts.

Prompts can be as simple as “Fill out this form with random data” or “Search for a flight.” To get started, check out the new Computer Use demo macros for example prompts that work well.

In general, we’ve found that - at least for now - Computer Use works better for browser automation than desktop automation. Google’s Project Jarvis team seems to have reached the same conclusion.

Project Jarvis - Google's new AI Chrome Extension?

Mon, 02 Dec 2024 00:00:00 +0100

Update Dec 5, 2024: We integrated Anthropic Computer Use into Ui.Vision. So if you want to control your browser with AI, you can do it with Ui.Vision already now.

–

Rumours have it, that Google may release “Project Jarvis” (its reported codename) Chrome extension this month. It would perform tasks like research gathering, product purchasing, and flight booking.

The website The Information leaked this information in October, just a week after Anthropic announced its Computer Use API. One could imagine that this “leak” was on purpose. Just as another “leak” a week later: A Chrome Web Store page described Jarvis as “a helpful companion that surfs the web for you” – and then disappeared again.

Powered by a future version of Google’s Gemini, Jarvis reportedly works exclusively with Chrome. The tool aims to automate everyday web tasks by taking screenshots, interpreting them, and performing actions like clicking buttons or entering text. Current iterations reportedly take “a few seconds” between actions - very similar in performance to Anthropic’s Computer Use.

Based on our experience with AI browser automation (“Project Ui.Vision”), a proof of concept seems more likely than a full release this month. While Anthropic’s Computer Use and Google’s Project Jarvis show promise, they’re not yet ready for typical users. No way that such a Jarvis tool can reliably execute a task like “book me the lowest carbon emissions flight from New York to Boston, and I want an aisle seat” from start to finish. However, the already available technology fits well in power user tools, which is why we’re incorporating Computer Use into the open-source Ui.Vision RPA software.

The Google team appears to have learned from Anthropic’s Computer Use launch. Our tests show that Anthropic’s Computer (October 2024 release) performs better with browser automation than desktop automation, likely because websites have more consistent usage patterns than desktop apps. Most website automation is basically some kind of form filling. In addition. web applications show no variation between operating systems (Mac, Linux, Windows).

Project Jarvis vs. Anthropic Computer Use

Current score: 1-0 for Anthropic, as Google Gemini 2.0/Project Jarvis hasn’t entered the competition yet.

Anthropic Computer Use – First Impressions, Test and Pricing

Mon, 04 Nov 2024 00:00:00 +0100

Anthropic just announced an exciting new capability for their AI assistant: Claude computer use, which allows it to interact with your computer by looking at your screen and controlling your mouse and keyboard.

How to use Claude computer use

Using Claude computer use is surprisingly straightforward: the system works by letting Claude see your screen through screenshots and then giving you instructions about what to do next.

Getting started is made easy with the Claude computer use demo, which Anthropic has provided on GitHub. While there isn’t yet an official Claude computer use tutorial, the Docker demo environment is helpful. The Claude computer use Docker setup is the quickest way to begin.

We tested the Anthropic Claude computer use features with tasks like opening a web browser, installing software, and editing computer code. So how good is it?

Computer use security

What makes this particularly impressive is Claude’s ability to “precisely” (kind of, see our test video below) identify where to click on the screen. Previous AI models struggled even with giving a ballpark estimate of x/y coordinates of items in a screenshot.

However, Anthropic is being appropriately cautious about security risks, particularly warning about potential misuse through malicious websites that might try to override Claude’s normal behavior. Mouse clicks themselves are not the key risk here. The main risk is - according to our test - the fact that Claude can also run shell commands. Shell commands can do more harm more quickly than most mouse clicks. Especially because you can monitor Claude’s mouse clicks visually, but the shell commands are executed in the background.

Computer use test

So what did we test and how did Claude Computer Use work:

Open Browser (ok)
Install Browser extension (ok)
Add extension icon to browser toolbar (failed, wrong approach)
Select macro and run it (ok)
Code edit (failed: Knew what to do, but fat fingered the edit)
Notepad edit test (ok, fat fingered edit but found workaround)

The video below shows the test and their results:

Just as a ballpark number for the Computer use pricing: Doing the above test sequence cost us around two US$ in Anthropic API usage credits.

Summary

Anthropic Computer use is the first meaningful progress in the RPA software space in years.

We would not recommend Anthropic Computer use for any kind of production use (yet). Computer use is still slow and error prone. But the use of LLM (Large Language Models) finally moves the needle to true RPA (Robotic Process Automation).

For years, we at the UI.Vision open-source RPA software team argued that visual computer automation is the future. It is great that we found a strong (informal) ally with Claude and Anthropic.

Computer vision industry gathers in the Länd

Thu, 19 Sep 2024 00:00:00 +0200

Vision, the trade fair for machine vision, returns to in the Länd from 8-10 October, 2024.

Held every two years in Stuttgart, Germany, the event plays host to the complete range of computer vision technology, with the conference programme designed to help visitors find out about all innovations in machine-vision and RPA technologies, both current and in-development for the future. The exhibition floor, meanwhile, will be home to exhibitors from all over the world, showcasing the latest systems and components for integrators.

2024 > 2022

Despite an admittedly challenging year for the industry, the event looks set to be bigger and better than ever. Earlier this year, Florian Niethammer, Manager of Exhibitions and Events at Messe Stuttgart revealed at a Vision/VDMA round table event in Stuttgart that the Vision show had already surpassed the exhibitor numbers experienced in 2022, with 385 exhibiting companies, which he described as “a pleasing development path with the Vision show.” The 2022 event itself saw a rise in visitor number over the previous iteration, with some 6,505 visitors walking through the doors.

Mr. Niethammer said that the organisers have been receiving more enquiries and expressions of interest every day, and he expected to break the 400 exhibitor mark and achieve a growth of at least 10% in exhibitor numbers by October. At the time of going to press, that number stands at 468. He believes that this reflects the positive expectations of the industry, despite some difficult market conditions. He said: “We see that, despite consolidation and merger and acquisition activities in the market, the imaging and machine vision field is still a very attractive field for many new companies to enter. Thus, we have accomplished around 25% of new companies that will exhibit for the very first time.”

Is iMacros no longer available?

Thu, 25 Jan 2024 00:00:00 +0100

In 2022, we told our readers on our automation blog that iMacros, once the most used web automation tool in the world, would soon stop being available. Now, in January 2024, we see that the company owning iMacros has not only stopped updating and supporting it (which we expected) but also completely removed it from the Chrome and Firefox web stores.

This means you can no longer download iMacros from the Chrome and Firefox stores.

Those who read our 2022 post about iMacros should be okay because we suggested downloading the latest versions of iMacros for Firefox XPI and the matching version of Firefox browser.

Trying to download it now is late, but not too late. So, how can you still get iMacros?

How to download iMacros?

The best version of iMacros was iMacros for Firefox V8.9.7. If you continue to use iMacros, get and keep this version. You can still find the “iMacros for Firefox V8.9.7 XPI” installer on some not-so-trustworthy sites and GitHub repositories. As always, be careful with downloads from unknown sources. When we checked for this blog post, we did not find any malicious iMacros XPI files. But better be safe than sorry. So for your safety, here is the information to check for:

The original “imacros_for_firefox-8.9.7-fx.xpi” file size is 356,330 bytes.
The MD5 checksum (a kind of digital fingerprint) of this iMacros installer is 10722343b9e0b98fc753cd2aeb0e23f5. We highly suggest checking this checksum (<- not our tool, just a link) before installing the file. If the checksum matches, it means the file is okay:

Best iMacros alternative?

The best thing to do is to switch to a good, open-source alternative to iMacros. Look at our list of alternatives to iMacros from 2022, which is still up-to-date.

RPA saves Hospital over 50,000 man-hours

Tue, 14 Nov 2023 00:00:00 +0100

Singapore General Hospital, Singapore’s flagship hospital, has been exploring robotic process automation.

Meanwhile many RPA automation use cases have been successfully implemented, saving 52,500 man-hours and gaining almost $2 million in productivity, the hospital said in a statement.

A most recent use case is the RPA for patient-reported outcome measures collection at the SGH Physiotherapy Department – the first such use case in Singapore. Following a trial between March and December last year, the technology was able to save about 1,350 hours of manual calculations and documentation.

SGH launched its first RPA use case in contact centre operations. Since then, it has explored the use of RPA in extracting and processing information from an online form for Medication Delivery Service, as well as automatic billing of patients for laboratory tests performed at Specialist Outpatient Clinics.

“While RPA enhances our processes back-end, it may also benefit our patients depending on the use case. SGH]will continue to look into wider adoption of RPA to take on repetitive manual tasks, and augment clinical workflows. By doing so, staff can focus on more complex work that requires thought process,” said Mr Chan, assistant manager of the Artificial Intelligence and Automation Unit and the RPA lead at SGH.

Aleph Alpha vs ChatGPT

Mon, 17 Apr 2023 00:00:00 +0200

The company, founded in 2019 by a former Apple manager, is considered one of the biggest hopefuls for an independent European “artificial intelligence”. According to the company, the “Luminous” program can compete with ChatGPT from market leader Open AI. The results of a standardized performance comparison published in February 2023 had shown this, the company claims. So how good is it?

One of the few public-facing Aleph Alpha projects is Lumi the ~~chat client~~ “digital citizen assistant” at the City of Heidelberg’s website. While certainly no ChatGPT, you can ask it about local high schools, holidays or on which day the trash cans are emptied.

In our non-scientific quick test Lumi’s answers were somewhat underwhelming. Lumi is certainly better than the average help chat robot - and in a time before ChatGPT that would have been impressive.

Rumor on the German-language internet (e. g. this podcast) has it that software manufacturer SAP is considering an investment in Heidelberg-based AI start-up Aleph Alpha. The decision is likely to be made soon. SAP wants to participate in the second round of financing now underway, the talk is of an amount below 100 million euros. When asked, the Walldorf-based company would not comment on this, nor would Aleph Alpha founder Jonas Andrulis.

Aleph Alpha had successfully completed a funding round of around €23 million in 2021. This had been used to increase the number of employees to around 50, build a fast AI data center in Bayreuth and launch reference projects.

Best iMacros Alternatives

Mon, 12 Dec 2022 00:00:00 +0100

(Last updated: January 2024)

iMacros, a popular web automation tool, is being discontinued by its new owner. This is sad news, as iMacros was the grandfather of all web automation tools. It literally invented this category.

Now, looking to the future, what are the best alternatives to iMacros? Many current iMacros users have contacted us for advice, and so we have decided to summarize our thoughts on the situation in this blog post. While there are many web automation tools available, there are only a few that we can recommend as true iMacros alternatives.

Here is our list of the best iMacros alternatives. Please note that this is not a ranking; the best replacement for iMacros will depend on your specific needs and use case.

iMacros Alternative: Google Puppeteer

Google Puppeteer is a library developed by Google that provides a high-level API to control Chrome or Chromium over the DevTools Protocol. It is essentially a headless browser, meaning that it allows you to control a web browser programmatically without the need for a graphical user interface.

Puppeteer can be used to automate tasks that would normally be performed manually in a web browser, such as clicking links, filling out forms, and scrolling pages. It can also be used to capture screenshots, create PDFs, and perform other types of web scraping and testing.

The drawback of Google Puppeteer is that it is a “framework” and, as such, it requires programming skills to automate tasks. This means that every iMacros macro would become a standalone program with Puppeteer. If that works for you, Puppeteer is a powerful alternative. However, it is a completely different approach to web automation. Additionally, Puppeteer does not support real-user simulation commands like IMAGESEARCH and DirectScreen. It can only automate at the XPath locator level.

PRO:

If the use of a framework is acceptable, this is the best framework to replace iMacros.
iMacros Scripting Interface commands are replaced by Puppeteer API calls.
Open-Source

CONTRA:

Since it is a framework, it requires significant programming skills
No support for real-user simulation (iMacros commands like IMAGESEARCH or DS).
No Record & Replay feature

iMacros Alternative: Bunker down with Firefox

iMacros as iMacros alternative? Yes, hear us out: While iMacros for Google Chrome will likely stop working at some point in 2023, the Mozilla Firefox team is way behind in its transition to Manifest V3. So there is a good chance that you can get another 2-3 years of usage out of the current iMacros for Firefox version. But keep in mind that there is no guarantee for this, and one Firefox update gone wrong could kill the iMacros extension by mistake. If that happens, there will be nobody around to fix and update iMacros.

If you choose to continue using iMacros, we strongly recommend that you download the latest version of both iMacros and Firefox. Do this as soon as possible. January 2024 update: iMacros is no longer available in the Chrome and Firefox stores, and you need to do a Google search to find iMacros. Disabling automatic Firefox browser updates can help extend the lifespan of iMacros, but it is important to note that doing so may pose a security risk, depending on your specific needs. Additionally, newer websites may eventually stop functioning correctly on older versions of Firefox.

PRO:

Staying with iMacros means no need to spend time on conversion
Free (but not open source)

CONTRA:

Eventually, even iMacros for Firefox will stop working
Using an older Firefox browser can be a security risk
The iMacros support forum will likely close at some point in 2023.

iMacros Alternative: UI.Vision RPA

Our UI.Vision RPA software is a free, open-source browser extension that allows you to automate tasks in the web browser. It is available for use with Chrome, Firefox, and Edge on Windows, macOS, and Linux platforms.

UI.Vision RPA provides a visual interface for creating and editing automation scripts, known as “macros,” which can be used to automate repetitive tasks in the web browser. These macros can be triggered manually or automatically using a schedule, and they can be run in multiple browser tabs or windows at the same time.

UI.Vision RPA supports a wide range of actions, including clicking elements, filling out forms, scrolling pages, and extracting data from web pages. It also includes features for debugging and testing automation scripts, as well as a command line interface (similar to the iMacros Scripting interface) for integration with other tools and services.

If this sounds similar to what iMacros does, it’s because it is. There is even an iMacros Alternative page with iMacros migration hints. In many ways, you can think of UI.Vision as the next generation of iMacros. As such, it is probably the best all-around iMacros alternative.

PRO:

Overall concept very similar to iMacros
Free and Open-Source
Free conversion help in the UI.Vision RPA forum
iMacros “DirectScreen” and IMAGESEARCH commands have equivalent real-user simulation commands.

CONTRA:

User interface (UX) different from iMacros, so a learning curve is required despite overall similarity
Macro source code is stored in JSON format instead of iMacros plain text format

What about other tools?

There are plenty of automation frameworks, most of which are designed for web testing: Cucumber, Selenium, Selenium IDE, Watir, Silk Test, Telerik Test Studio, Power Automation, HP UFT (aka QTP), Katalon. These are good tools for their core use case, but they are not good iMacros alternatives:

If they are a framework, they require programming skills. If using a framework is ok for you, the Puppeteer framework (see our list above) is the better iMacros alternative.
If they are a software, they are designed for web testing use, and not for web automation or web scraping.
All tools except iMacros and UI.Vision rely on the use of XPath locators to automate the web. In our opinion, Computer vision and text recognition is the future of web automation. The locator approach is outdated and over time, less and less websites can be automated with locators alone.

By the way: The three iMacros alternatives that we recommend are all free. This also saves you the time of searching for iMacros cracks, iMacros free license keys, or iMacros serial keys. You get the power of the iMacros full version for free.

Did we miss a good iMacros alternative? If so, please let us know.

We recommend this step-by-step approach:

Don’t panic! :-)
Leave iMacros for Chrome as soon as possible. It can break anytime.
Keep your old macros running in iMacros for Firefox.
Create all new web automation tasks directly in the open-source UI.Vision RPA software.
Whenever you need to modify an existing iMacros macro, migrate it to the UI.Vision RPA software at the same time.

RPA as sustainability tool

Fri, 07 Oct 2022 00:00:00 +0200

Sustainability has become a strategic priority in modern organizations. The sustainability keyword is present in regulatory filings, in annual reports and even on organizational charts. In fact, the latest Gartner Insights Sustainable Business Survey found that executives involved with sustainability strategy and initiatives plan to increase their sustainability investments over the next years. Often it serves only as “green washing” but there are some real benefits as well:

From Traditional IT to RPA

Traditional IT sustainability use cases have included low-emissions hardware, cost-efficient power and cooling, smaller data centers, cloud migration and decommissioned equipment recycling, to name a few. Now, robotic process automation (RPA), which uses software robots to carry out simple, structured and repetitive business processes such as data entry, has emerged as a vital and strategic catalyst for sustainability.

RPA can automate or partially automate processes that traditionally rely on paper consumption and incur printing costs. For instance, by integrating optical character recognition, sometimes known as text recognition, with RPA, users can extract and repurpose data from scanned documents, images and image-only PDFs and convert documents to searchable PDF. This allows them to easil access and edit the original document, while eliminating the need for manual data entry and extra printouts.

Save time

RPA automates routine IT processes such as manual data entry. And deploying a software-as-a-service solution eliminates the need for expensive infrastructure and maintenance investments such as servers.

Redistribute resources.

Saved resources—human time and materials—can be revamped and applied to a business’ higher level, strategic priorities.

Uncover new insights.

Leveraging data from process mining and RPA infused with AI enables new patterns to surface, highlighting opportunities for further automation. Companies can then act on those insights, thus sparking a virtuous cycle of efficiency and innovation.

Decrease monotonous tasks.

RPA frees employees from tasks that add little value to engage in more meaningful, higher value activities, fostering a culture of creativity and innovation.

Increases competitiveness.

RPA helps best-in-class organizations use data to make better decisions, improve their customer experience and become providers that others want to do business with.

RPA Software V8 released

Mon, 22 Aug 2022 00:00:00 +0200

The RPA software release V8 includes two major parts:

Updated Open-Source RPA Core

We have updated and rewritten large parts of the UI Vision RPA core to add “Chrome Manifest V3” support. Technical details aside, what this means is that the new RPA version supports the latest Chrome and Edge extension standards, and is future proof. Google will “turn off” all older “Manifest V2” extensions in early 2023, but you can now rest assurred that Ui Vision Selenium IDE++ will keep on automating, clicking and typing.

New XModule with local OCR support

The latest UI.Vision XModule brings integrated OCR support. This RPA OCR runs 100% local. It is fast and works on offline machines without Internet connection. XModules local OCR is currently available for Windows and Mac. Linux local OCR support is planned.

Try it out! This feature will change the way you automate.

Example: Often you want to click a certain word a complex website. Sounds simple, or? But with most other tools you have learn how to deal with changing element IDs and changing XPath. Some other tools allow you also click based on x/y coordinates, and a few support visual automation with image search like UI Vision does. But no other tool has direct support for clicking on text. So instead of

Click | xpath=//*[@id="step1_form"]/div/div/div/div/select

You can simply write

Xclick | ocr=Login

=> Web automation & web testing was never easier.

Our Open-Source RPA Blog

Run Anthropic Computer Use in Your Web Browser

Step-by-Step Instructions

Computer Use Tutorial

Desktop Automation

Anthropic Computer Use Pricing

Computer Use Insights from Our Experience

Project Jarvis - Google's new AI Chrome Extension?

Project Jarvis vs. Anthropic Computer Use

Anthropic Computer Use – First Impressions, Test and Pricing

How to use Claude computer use

Computer use security

Computer use test

Summary

Computer vision industry gathers in the Länd

2024 > 2022

Is iMacros no longer available?

How to download iMacros?

Best iMacros alternative?

RPA saves Hospital over 50,000 man-hours

Aleph Alpha vs ChatGPT

Best iMacros Alternatives

iMacros Alternative: Google Puppeteer

iMacros Alternative: Bunker down with Firefox

iMacros Alternative: UI.Vision RPA

What about other tools?

What iMacros migration path do we recommend?

RPA as sustainability tool

From Traditional IT to RPA

Save time

Redistribute resources.

Uncover new insights.

Decrease monotonous tasks.

Increases competitiveness.

RPA Software V8 released

Updated Open-Source RPA Core

New XModule with local OCR support