Scribd
Upload
147 views28 pages

GUI Automation with Sikuli

Sikuli is an open-source automation tool that uses image recognition to identify and control graphical user interfaces. It allows testing and automation of applications without access to source code or APIs. Sikuli works by taking screenshots of the screen and using computer vision algorithms to find patterns and click or type in the appropriate places. It supports Windows, Mac, and Linux and can automate tasks like proxy configuration, software testing, and game automation by waiting for images to appear and simulating user interactions programmatically. While powerful, it is limited by the instability of interfaces changing and needing the elements to be visible on the screen.

Uploaded by

rifanif
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd
147 views28 pages

GUI Automation with Sikuli

Sikuli is an open-source automation tool that uses image recognition to identify and control graphical user interfaces. It allows testing and automation of applications without access to source code or APIs. Sikuli works by taking screenshots of the screen and using computer vision algorithms to find patterns and click or type in the appropriate places. It supports Windows, Mac, and Linux and can automate tasks like proxy configuration, software testing, and game automation by waiting for images to appear and simulating user interactions programmatically. While powerful, it is limited by the instability of interfaces changing and needing the elements to be visible on the screen.

Uploaded by

rifanif
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd
You are on page 1/ 28

Sikuli Automation

What is Sikuli ?
Visual approach to search and automation of graphical
GUI Automation
user interfaces using screenshots
Sikuli was an open-source research project at the
User Interface Design Group at MIT.
Why Sikuli ?
Motivation

Things usually needed to automate GUI:


▪ support from developers
▪ API access
▪ language/OS dependency
▪ position/naming dependencies
Sikuli in Brief

▪ Sikuli automates anything you see on the


screen
▪ Sikuli uses image recognition to identify
and control GUI components
▪ Sikuli is useful when there is no easy
access to a GUI’s internal or source code
DEMO
Proxy setting
automation
How it works ?
System Design
Multiplatform Support

Mac Windows Linux


How to use it ?
Find visual objects

find( )
Finder

f = Finder(path-to-imagefile)

f.find(path-to-imagefile, [similarity])

// iterates through Match objects


while(f.hasNext):
print “found match: “ + f.next().getScore()
Region

Region(x, y, w, h)
Region(region)
Region(Rectangle)

 Search in given region


 Observe a region in background for changes
Act on visual objects

click( )
Mouse actions

click(image)
doubleClick(image)
rightClick(image)
hover(image)
dragDrop(image_target, image_destination)
Typing & ‘Reading’ actions

type(text)
type(image, text)
paste(text)
paste(image, text)
text(image)

 text() uses The Tesseract OCR Engine


Button & Keys actions

mouseDown(button)
mouseUp(button)
keyDown(keys)
keyUp(keys)

 They can be used along with KeyModifiers


Wait actions

wait(image)
waitVanish(image)
exist(image)

 Time out can be defined


Event handler

onAppear(image, handler)
onVanish(image, handler)
onChange(image, handler)
Spatial operator

find( ).left().find( )
Performance

200 milisecond
Find() for 100x100 target on a 1600x1200 screen
Limitations

▪ Screenshots – unstable interfaces


▪ Visibility constraints
What is it used for ?
Applications

▪ Task automation/Macros
▪ Software automation testing
▪ Demo/Presentation
▪ Gaming
Thanks!
SlidesCarnival icons are editable shapes.

This means that you can:


● Resize them without losing quality.
● Change line color, width and style.

Isn’t that nice? :)

Examples:

You might also like