Citrix Automation, Image and Text Automation 4.1

Summary

The UiPath Essential training video introduces image-based automation for virtual machines, focusing on Citrix environments. It covers the limitations of traditional recording methods and presents alternative tools like ClickImage and ClickText for interacting with interface elements. The training also demonstrates using keyboard actions for navigation and introduces Select&Copy and ScrapeRelative actions for extracting information from applications. The script is a practical guide to automating tasks in virtual environments with UiPath.

Takeaways

• 📚 UiPath Essential Training introduces the fourth type of recording, Citrix, which is designed for automating virtual machines but can also be useful for other types of automation.
• 🖥️ Virtual Machines (VMs) run on a server and stream only the image of the interface to the user, requiring special tools and techniques for automation that UiPath provides.
• 🚫 A limitation of Citrix recording is that actions cannot be automatically recorded and played back; the 'robot' must be guided through the process manually.
• 🔍 The process of teaching the UiPath robot involves finding and interacting with screen elements such as buttons and text fields, similar to how a human would operate them.
• 🖱️ 'ClickImage' is a UiPath action that allows clicking on unique screen elements, which is useful when traditional selectors cannot be used.
• ✍️ 'ClickText' uses OCR to scan and identify text on the screen, making it versatile even when graphical elements like themes or text sizes vary.
• 🔍 'ClickImage' is fast and reliable but can be sensitive to graphical variations, while 'ClickText' is more adaptable but may be prone to errors with misidentified text.
• 🔑 An alternative to mouse interactions is using keyboard actions like the Tab key for navigation and Space to activate functions, offering a faster and safer method for both desktop and web applications.
• 📋 'Select&Copy' is a straightforward method for extracting text from selectable fields within a virtual machine, utilizing clipboard operations.
• 🔍 'ScrapeRelative' is an advanced action for extracting text from non-selectable fields by locating fixed elements on the screen and using OCR to extract relevant information.
• 🛠️ The 'ScrapeRelative' action consists of a series of steps including finding an anchor image, calculating the region based on the anchor, and extracting text using OCR, followed by resetting the clipping region.

Q & A

What is the main focus of the UiPath Essential training session described in the transcript?

The main focus of the training session is on image-based automation, particularly for Citrix environments and virtual machines, and how to deal with the limitations of automating interfaces that are streamed as images to the user.

Why can't UiPath automatically record a set of actions for playback in a Citrix environment?

UiPath cannot automatically record a set of actions for playback in a Citrix environment because the interface is streamed as an image, and UiPath cannot address the interface through the operating system as it does with Basic recording and selectors.

What are the two primary methods introduced in the training for interacting with elements on a virtual machine interface?

The two primary methods introduced are using 'ClickImage' to click on elements based on a unique image, and 'clickTxt' which uses OCR to scan the screen for text and interact with elements relative to the identified text.

How does the 'ClickImage' action differ from the 'clickTxt' action in terms of its sensitivity to graphical variations?

The 'ClickImage' action is faster and more reliable but is more sensitive to graphical variations such as colors or background details. It can fail if these elements change, making it prone to errors in fluctuating environments. In contrast, 'clickTxt' is less sensitive to such variations because it relies on OCR.

What alternative method can be used to avoid the limitations of mouse interactions in virtual machine automation?

An alternative method is to use keyboard actions such as the Tab key for navigation and keyboard shortcuts to activate functions within the application, avoiding the need for mouse interactions altogether.

What are the two methods discussed for extracting information from virtual machines?

The two methods for extracting information from virtual machines are 'Select&Copy', which works for selectable text, and 'ScrapeRelative', which uses screen scraping techniques relative to a fixed element to extract non-selectable text.

How does the 'ScrapeRelative' action work in the context of screen scraping?

The 'ScrapeRelative' action works by first finding an anchor image, then calculating the region relative to that anchor image, and finally using OCR to extract the content of that region.

What are the limitations of using 'ClickTxt' for automation in virtual environments?

The limitations of 'ClickTxt' include its reliance on OCR, which can be thrown off track by even a single misidentified letter, making it less accurate in environments where text recognition is critical.

What is the importance of the 'Accuracy' parameter in the 'ClickImage' action?

The 'Accuracy' parameter in the 'ClickImage' action determines how closely the images must match to be considered found. A value of 1 requires a 100% match, while a value of 0.8 provides a balance between accuracy and reliability, reducing the likelihood of errors due to minor graphical variations.

How can keyboard actions be combined to navigate and interact with an application more efficiently?

Keyboard actions can be combined by chaining multiple key presses, such as typing a value followed by a Tab key to move to the next field, and then repeating the process. This method allows for rapid navigation and interaction without the need for mouse clicks.

What is the role of the 'TypeInto' action in the context of the training?

The 'TypeInto' action is used to fill in text fields within the application. It is one of the three primary actions taught in the training for interacting with virtual machine interfaces, alongside 'clickText' and 'ClickImage'.