ROBOTIC PROCESS

AUTOMATION
Dr.Jeeshma Mary Paul
Assistant Professor
Dept. of Robotics and Automation
ASIET
 RECORDING
In UiPath there are 4 types of recording
• Basic recording
• Desktop recording
• Web recording
• Citrix recording
Basic Recording

• This is used to record the actions of applications that have a single window.

• Basic Recording uses a full Selector. It works better for applications performing a
single action.

• It is not suitable for applications with multiple windows.

 RECORDING
• There are two types of selectors, partial selectors
and full selectors.

• A Full selector has all the attribute to recognize a

control or application. The Basic recording uses full
selectors.

• Basic recording generates different activities and

places them directly in the sequence with full
selector.
 RECORDING
Desktop Recording

• This is similar to Basic recording with the added

advantage of working with multiple actions.

• It is most suitable for automating Desktop

applications.

• Desktop recorder generates Partial selectors.

• The Partial selectors, have a hierarchical

structure.
 RECORDING

• They are split into parent child views for

recognizing the UI element properly.

Please note in the image there is a Attach Window

activities and other activities are nested under it.
This flow is generated Desktop recorder
 RECORDING
Web Recording
• Web Recording can be done by using the Web recorder.

• For recording web actions, the UiPath extension for that browser should
be installed.

• Otherwise, you will not able to automate tasks or actions using Web
recording.

• You just have to click on the Setup icon and then click on Setup
Extensions.
 RECORDING
Web Recording
• Now, choose your browser and click on it.

• The UiPath extension will be added to your specified browser.

• Web Recording is similar to Desktop Recording. You just have to record

the actions and save it.
 RECORDING
Steps for web recording:

1. Create a Blank project.

2. Drag and drop a Flowchart activity.
3. Now, click on the Recording icon and choose Web recording.
4.You can record your actions on the web on your own and then save it.
5.In our case, we have opened a web page using Google Chrome and logged in to
https://www.google.com.
6.Then, we started the recording by clicking on the Record button of the web recorder.
7.Next, we typed some text in the search bar of Google and performed the Click
activity.
8.Then, we pressed the Esc key to exit the recording and clicked on the Save and Exit
button.
 RECORDING
Steps for web recording:

9.Now, a recording sequence is generated in our Designer panel.

10.Connect this sequence to the Start node.
11.Hit the Run button to see the result. In the following screenshot, you can see the
sequence generated by the Web record
 RECORDING
In the following screenshot, you can see the sequence generated by the Web record

Note: Before running the UiPath workflow, make sure you are on the Google homepage
 RECORDING
Citrix Recording

• When dealing with the Remote Desktop connection, methods such as Basic
Recording and Desktop Recording cannot be used.

• In an RDP environment, images will be sent from one desktop to another, and will
be mapped by analyzing the position of the pointer of the mouse button.

• Hence, basic and desktop recording cannot be used, as these recording techniques
fail to interact with the images.

• In a Citrix environment, we have the Click Text and Click Image activities, using
which we can work with images with ease.
 RECORDING
• The activities that are listed in a Citrix Recording are :
1. Click Image
2. Click Text
3. Type
4. Select & Copy
5. Screen Scraping
6. Element
7. Text
8. Image

All these activities are used extensively in a Citrix environment.

 SCREEN SCRAPING
• Screen Scraping is a method of extracting data from documents, websites,
and PDFs.

• It is a very powerful method for extracting text.

• We can extract text using the Screen Scraper wizard.

• The Screen Scraper wizard has three scraping methods:

a. Full Text
b. Native
c. OCR
 SCREEN SCRAPING
Full Text

• The Full text activity is used to extract information from various types of
documents and websites.
• It has a 100% accuracy rate. It is the fastest method among all three
methods.
• It even works in the background.
• It is also capable of extracting hidden text. However, it is not suitable for
Citrix environments.
 SCREEN SCRAPING
Native

• This is similar to the Full text method but has some differences.
• It has a slower speed than the Full text method. It has a 100% accuracy
rate, like the Full text method.
• It does not work in the background.
• It has an advantage over the Full text method in that it is also capable of
extracting the text’s position.
• It cannot extract hidden text.
• It also does not work with a Citrix environment.
 SCREEN SCRAPING
OCR (Optical Character Recognition)

• This method is used when the previous two methods fail to extract
information.
• It uses the two OCR engines: Microsoft OCR and Google OCR.
• It has also a scale property: you can choose the scale level as per your
need.
• Changing the scale property will give the best results:
 SCREEN SCRAPING
Let us consider an example of extracting text from the UiPath website's main page:

1. Create a Blank project and give it a meaningful name.

2. Log on to the UiPath website by logging in to httpps://www.uipath.com in your
browser.
3. Drag and drop a Flowchart activity on the Designer panel.
4.Click on the Screen Scraping icon and locate the area from which you want to
extract the information.
5.Just choose an area on the UiPath website.
 SCREEN SCRAPING
Let us consider an example of extracting text from the UiPath website's main page:

6.A window will pop up stating that the AUTOMATIC method failed to scrape this
UI Element.
 SCREEN SCRAPING
7.. Try choosing another method. We shall choose the Full text method. This
too will fail. Next,
choose the Native method. This will also fail, as you can see in the following
screenshot:
 SCREEN SCRAPING
8. This time, choose the OCR Scraping method. You can clearly see the
extracted text:
 SCREEN SCRAPING
When to use OCR
• There are some scenarios where normal activities such as Get Text and Click
Text activities fail to extract the text or perform an action.
• This is when OCR comes in, giving us the flexibility to perform actions when
existing activities fail to do their job.
• It is a text recognition technique that transforms printed documents that are
scanned into electronic formats.
• OCR is used mainly for images, scanned documents, PDFs, and so on, to extract
information or perform actions.
• Extracting information or data from images, scanned documents, or PDFs is a
very tedious job.
• Normal activities are not recommended for extracting these types of inputs.
• OCR uses a different method and approach to extract the information.
 SCREEN SCRAPING
Types of OCR

• There are two OCRs available in UiPath Studio:

1. Microsoft OCR (MODI)

2. Google OCR (Tesseract)

Advantages of Google OCR

• Multiple language support can be added in Google OCR

• Suitable for extracting the text from a small area
• It has full support for color inversion
• It can filter only allowed characters
 SCREEN SCRAPING
Advantages of Microsoft OCR

• Multiple languages are supported by default

• It is suitable for extracting text from a large area

• OCR is not 100% accurate.

• It is useful for extracting text that other methods cannot successfully do.

• It works with all applications, including Citrix.

• Microsoft and Google's OCRs are not the optimum for every situation.
Sometimes, we have to look for more advanced OCRs to recognize more
sophisticated text, such as handwritten documents and so on.
 SCREEN SCRAPING
• There is another OCR available in UiPath Studio, known as the Abbyy
OCR Engine.
• You can find this OCR engine in the Activities panel by searching for
OCR.
• If you cannot find this OCR listed in the Activities panel, you need to
install the Uipath.Core.Activities package
 SCREEN SCRAPING
• Eg: Screen scrapping of image without OCR
SCREEN SCRAPING
 SCREEN SCRAPING
• Eg: Screen scrapping of text without OCR
 SCREEN SCRAPING
How to use OCR
• Suppose we have an image and we have to extract the text in it.
• In such a scenario, OCR becomes very handy.
• In the following example, we are going to use a random Google image
with some text in it, as follows:
 SCREEN SCRAPING
To extract text from the previous image, perform the following steps:

1. Open UiPath Studio and click on a Blank project. Give it a meaningful name.
2.On the Designer panel, drag and drop a Flowchart activity.
3.Next, drag and drop a Get OCR Text activity from the Activities panel and set it as
the start node.
4.Double-click on it and click on the Indicate on screen option.
5.Choose the specific area from which you want to extract the text from the image.
6.In our case, we are using an image that we have searched for on Google.
7.Now, click on the Text property of the Get OCR Text activity.
8.A window will pop up, right-click inside the window and choose Create Variable.
9.Give it a meaningful name, press Enter, and click on the OK button.
10.A variable will be created with that name:
 SCREEN SCRAPING
11.Drag and drop the Message box activity.
12.Connect it to the Get OCR with textactivity.
13.Double-click on the Message box activity and specify the variable name
that you have created earlier in the expression box.(In our case, it is the
result variable).
14. Click on debug file to see the result.
 SCREEN SCRAPING
• Eg: Screen scrapping of image in stored in system using OCR
1. Open UiPath Studio and click on a Blank project. Give it a meaningful
name.
2.On the Designer panel, create a new sequence
3. Drag and drop a Load Image activity
4.Select the location of your image
5.Click on the Load Image activity and in its properties create a variable
“inputImage” to store the image
 SCREEN SCRAPING
• Eg: Screen scrapping of image in stored in system using OCR
6. Drag and drop a Tesseract OCR activity
7.Click on the Tesseract OCR activity and in its
properties use variable “inputImage” as Input
and create a variable “outputText” to store the
result

8. Drag and drop a Message box activity to

display the result.
9. Click on Debug file to run it.
 SCREEN SCRAPING
• Eg: Screen scrapping of image with OCR

Input image

Output text
 SCREEN SCRAPING
Avoiding typical failure points

• There are many scenarios where the normal implementation would fail.

• We will work with the following entities to tackle failure points:

a. Selectors
b. Scope of the variable
c. Delay
d. Element Exists
e. Try/ Catch
f. toString method
 SCREEN SCRAPING
Selectors

• Sometimes, it is tedious to deal with selectors while working with them.

• This is because a selector has attributes, title, and class properties.
• When we select a UI element using the selector, it stores all these
properties.
• Different instances of an application may have different properties of a
UI element.
• The problem with selectors is when you select a UI element, it captures
its properties.
• These properties will differ when we select the UI element of a different
instance of an application with the selector.
• Hence, the property will differ and the selector will fail to recognize the
same UI element of another instance of the application.
 SCREEN SCRAPING
• We can easily fix this problem by using wildcard characters or by
attaching it to a live element.
• Two wildcard characters are available with UiPath:

1.The question mark symbol,?, which replaces one character

2. The asterisk symbol, that is,* , which replaces a number of

characters

• We have to simply replace the variables (the name that changes

continuously) with wildcard characters.
• We can also use the Attach to live element option from the selector
property window and indicate the element again.
• It automatically detects the variable properties and fixes them for us.
 SCREEN SCRAPING
Scope of the variable

• Sometimes we create a variable inside a Sequence or Do activity.

• In doing so, the scope of the variable is limited to only that activity.

• When we try to access a variable from outside its scope, it cannot be accessed.

• We have to change the scope of the variable.

Delay Activity
SCREEN SCRAPING
• In some situations, we have to wait for a particular action.
• For example, when opening the Outlook application, it needs to connect to the
server (for synchronization).
• When it is opened, it takes some time (the UI element is not stable at this
stage).
• In the meantime, the robot's activity is waiting for the UI element to be stable
so that it can perform the action.
• After waiting for some time, if the UI element is not stable, it results in an
error because the activity cannot find the UI element.
• Thus, we have to add a Delay activity to ensure that the UI element is stable
for action.
• Specify the time for the delay in the expression text box of the Delay activity.
• This activity will delay the process for the specified period of time.
 SCREEN SCRAPING
Element Exists

• This activity is used to ensure that the required Element Exists.

• It is used to ensure that the element we are looking for exists in this
context.
• This is a good way of checking whether the activity exists or not.

Try/Catch

• This is an exception handling mechanism used to tackle exceptions.

• Put all suspicious activities inside the try block. If an error occurs, it can
be detected by the catch block.
 SCREEN SCRAPING
toString

• Sometimes, we forget to use the toString method with variables and we

end up with an error.
• For example, when outputting an integer variable inside the Message
box, we have to apply the toString method.