CS 352: Computer Graphics
Input
and
Interaction
Chapter 3 - 2
Interactive Computer Graphics
What good is Computer Graphics?
JS1k canvas examples: 1 2 3 4 5 Games, visual demosof what value? Is there a Christian perspective?
Communication of information to the user
Data visualization, simulation, GUI Even a word processor is an "interactive graphics program"
Communication between users
"Collaborative environments" are hot (multi-player games?)
Social networking is transforming the world
Interaction is an essential component
Chapter 3 - 3
Interactive Computer Graphics
Interaction
Much of the fun and utility of graphics is in interaction: accepting user input that affects the display Paint programs, modeling, games, word processors, spreadsheets, powerpoint
User initiates input events such as clicking on a menu or drawing a circle Computer response by changing graphical display
Chapter 3 - 4
Interactive Computer Graphics
Chapter 3 - 5
Interactive Computer Graphics
Projects 3
Projects 3 will be a paint program
we'll learn 2-D graphics, interaction, event-loop programming, double-buffering simple animation, basic image processing Pen, line, and rectangle tools Color, pen size selectors Image processing (sharpen, blur, detect edges) Open, save images Toolbar and menu for controlling application
Features:
Chapter 3 - 6
Interactive Computer Graphics
How to set up an application UI?
How How How How
to to to to
make make make make
a menu? a color picker? a toolbar? toolbar buttons pop in and out?
Chapter 3 - 7
Interactive Computer Graphics
How to paint?
How do I make a colored line follow the mouse or fingertip?
Interactive prog:
Input devices Event handlers Event loop
Chapter 3 - 8
Interactive Computer Graphics
Input devices
Interaction requires handling input devices
Physical: mouse, fingertip, keyboard, joystick,
digitizer, accelerometer, head tracker
Logical:
Text Locator Valuator (slider) Stroke Color picker
How to read?
Chapter 3 - 9
Interactive Computer Graphics
Input devices
Sample mode
Program reads the current state of input device (frequently) Each click or motion generates an event that goes on a queue Program can process events from queue
Event mode
HTML?
Chapter 3 - 10
Interactive Computer Graphics
Event-loop programming
Events: button press, slider value change,
menu selection, mouse move, keypress
Event loop:
Poll (or wait) for events Process events, update state Update display (ideally: wait for rest of frame time to elapse)
Chapter 3 - 11
Interactive Computer Graphics
State-driven
Typically, the event loop is driven by a big state table
e.g. depressing the paintbrush tool releases other tools, puts me in "paint mode"
Good libraries will handle some of the bookkeeping for you You still typically have to handle some state
Chapter 3 - 12
Interactive Computer Graphics
State diagram
Chapter 3 - 13
Interactive Computer Graphics
Event loop in HTML/JS
HTML/JS provides event queue, support for many basic events (mousedown, mouseup
mouseover, mousemove, keypress, key release, value change, button click, etc.)
You are on your own for higher-level events, e.g. clicking on a toolbar tool It's also possible to set a function to run every 15 ms, sample input devices
Chapter 3 - 14
Interactive Computer Graphics
Events for painting?
mousedown:
go into paint mode store mouse position draw initial dot exit paint mode if in paint mode
mouseup:
mousemove:
draw a line from old mouse position to current set old mouse position to current position
Chapter 3 - 15
Interactive Computer Graphics
Event handling with jQuery
Binding events to functions
$(cpaint.canvas).bind('mousedown', cpaint.drawStart)
Processing events
cpaint.drawStart = function(ev) { var x, y; x = ev.pageX - $(cpaint.canvas).offset().left; y = ev.pageY - $(cpaint.canvas).offset().top; ev.preventDefault(); cpaint.paintmode = true; cpaint.oldX = x; cpaint.oldY = y;
Chapter 3 - 16
Interactive Computer Graphics
Looking neat and spiffy
How to avoid crinkles in your paint strokes?
Chapter 3 - 17
Interactive Computer Graphics
Looking neat and spiffy
How to avoid crinkles in your paint strokes?
Draw connected paths
Or just use line caps
Chapter 3 - 18
Interactive Computer Graphics
Menus
There are many jQuery menu plug-ins
<ul id="mainmenu"> <li>File <ul> <li id="menuNew">New</li> <li id="menuOpen">Open</li> <li id="menuSave">Save</li> ------------------------------------$('#menuNew').bind('click', cpaint.clear); $('#menuOpen').bind('click',cpaint.open); $('#menuSave').bind('click',cpaint.save);
Chapter 3 - 19
Interactive Computer Graphics
Toolbar
How to make a toolbar?
How should buttons behave?
State diagram?
Chapter 3 - 20
Interactive Computer Graphics
Buttons
Widgets may have several states State should be evident in visual feedback E.g. how exactly does a button work? States?
State transition diagram
Most buttons: six states, with six different appearances
neutral neutral-highlighted neutral-depressed selected selected-highlighted selected-depressed
Events: mousedown, mouseup, enter, exit Transitions: what happens in each state under each event?
Chapter 3 - 21
Interactive Computer Graphics
Button state diagram
(buttons selectable, not unselectable)
N S
NH
SH N: neutral H: highlighted NHD SHD
(usually mouse over)
Press Move
S: selected D: mouse down
Chapter 3 - 22
Interactive Computer Graphics
Other button considerations
Could also consider
Tooltips Whether button merely clicks and releases or can be selected Whether button can be unselected (e.g. B/I/U vs. Left, Center, Right)
Want consistent appearance, behavior over whole program or whole computer Really need a library implementation and a strict set of UI guidelines
Chapter 3 - 23
Interactive Computer Graphics
4-State toolbar buttons: CSS
.toolbarCell { background-color:#ddd; width:20pt; height:20pt; border: solid #eee 2px; -webkit-box-shadow: 2px 2px 2px #666; }
#markerButton { background: url(img/paintbrush.png) no-repeat center center; }
.selected { -webkit-box-shadow: inset 2px 2px 2px #666; }
.toolbarCell:hover { border:solid #555 2px; }
Chapter 3 - 24
Interactive Computer Graphics
4-State toolbar buttons: JS
$('#markerButton').bind('click', {tool:"marker"}, cpaint.selectTool); -------------------
cpaint.selectTool = function(ev) { cpaint.tool = ev.data.tool;
// get tool name
$('.toolbarCell').each(function(index) { // unselect $(this).removeClass('selected'); // others }); var tool = '#' + cpaint.tool + 'Button'; // get ID $(tool).addClass('selected'); // select
Chapter 3 - 25
Interactive Computer Graphics
Paintbrush size selector
How?
Chapter 3 - 26
Interactive Computer Graphics
Color picker
Google "jQuery color picker"
Chapter 3 - 27
Interactive Computer Graphics
How draw a line?
What kind of feedback is normal?
Rubber Banding
Events and actions?
Chapter 3 - 28
Interactive Computer Graphics
Save & restore
Create your own off-screen canvas Copy it back on each mouse movement Events and actions?
Chapter 3 - 29
Interactive Computer Graphics
Save & Restore event handling
Mousedown
enter line mode remember mouse position as startx, starty save screen draw initial dot paste saved screen draw line from startx, starty to current mouse pos exit line mode
Mousemove
Mouseup
Chapter 3 - 30
Interactive Computer Graphics
Analysis
Drawbacks?
slow eats memory bandwidth for breakfast
copy smallest possible rectangle? Only points from line?
possible flickering
use double buffering?
Chapter 3 - 31
Interactive Computer Graphics
Double buffering
Have two frame buffers, "front" and "back" Draw into back buffer At vertical retrace time, swap buffers
This is a fundamental graphics technique not built into canvas though, some drawing aggregation seems to happen automatically, behind the scenes; not usually necessary in canvas, but browser dependent
Chapter 3 - 32
Interactive Computer Graphics
Fake Double Buffering in Canvas
Create off-screen canvas
canvasBuffer = document.createElement('canvas'); canvasBuffer.width = canvas.width; canvasBuffer.height= canvas.height; canvasBufferContext= canvasBuffer.getContext('2d');
Draw into off-screen canvas
canvasBufferContext.[drawSomeStuff]
Copy onto display canvas
context.drawImage(canvasBuffer, 0, 0);
Chapter 3 - 33
Interactive Computer Graphics
How to erase?
Draw background color
Chapter 3 - 34
Interactive Computer Graphics
Save, Load?
Save
Copy pixels to an image so users can right-click, save-as
Load
What are the security risks? Can only load files from the same server Can use a file chooser if it's a local app Security policies are not entirely in place
Chapter 3 - 35
Interactive Computer Graphics
Resizing
How would you allow the user to shrink, expand image? What would happen to image resolution?
Chapter 3 - 36
Interactive Computer Graphics
Store drawing commands
Realistically, to redraw effectively, you have to save all drawing commands
pen color red fill color blue line width 5 circle 5 10 12 textfont Times 12 text 10 10 "hello world"
Replay commands to redraw scene Could store commands in a file (e.g. Mac PICT file) For some kinds of drawings, files are smaller and more accurate than bitmaps
Chapter 3 - 37
Interactive Computer Graphics
Other toolkits
HTML offers basic UI capabilities in a crossplatform context Writing your own UI extensions is tedious jQuery plugins extend capabilities Ideally, all UI elements ought to be built in In the real world, they come in platformspecific 'toolkits' or 'windowing libraries' E.g. MFC, QT, OpenLook, Cocoa
Chapter 3 - 38
Interactive Computer Graphics
Image processing
Image processing examples:
blur, sharpen detect edges enhance contrast noise reduction posterize fisheye redeye reduction find faces
Chapter 3 - 39
Interactive Computer Graphics
How to fade an image?
Could average pixel colors with white Or just decrease alpha
Chapter 3 - 40
Interactive Computer Graphics
How to blur an image?
Blurring is an averaging process Convolution: apply a convolution kernel, e.g.
1/9 1/9 1/9 1/9 1/9 1/9 1/9 1/9 1/9
Center on pixel of interest Multiply each value by color val underneath Add results Gaussian smoothing
Chapter 3 - 41
Interactive Computer Graphics
Gaussian smoothing
Chapter 3 - 42
Interactive Computer Graphics
Chapter 3 - 43
Interactive Computer Graphics
How to sharpen an image?
It's a differentiation process What's Unsharp Masking?
Chapter 3 - 44
Interactive Computer Graphics
How to sharpen an image?
It's a differentiation process What's Unsharp Masking?
Subtract blurred version from original Add back to original
Chapter 3 - 45
Interactive Computer Graphics
Sharpening
Subtract neighbors Like subtracting a blurred version of the image Unsharp Masking E.g. convolve with a kernel like one of these:
Chapter 3 - 46
Interactive Computer Graphics
Edge detection?
Wikipedia
Chapter 3 - 47
Interactive Computer Graphics
Sobel Edge Operators
Look for horizontal and vertical variation
1 0 -1 2 0 -2 1 0 -1 -1
-2 -1
0
0 0
1
2 1
Could do this at different sales or resolutions Note: results maybe positive or negative numbers; must normalize
Chapter 3 - 48
Interactive Computer Graphics
Image Compression
What if you created say six different lowerresolution images and only stored the difference at each resolution?
Note: most of the data is in the highestfrequency component An early image compression technique
Chapter 3 - 49
Interactive Computer Graphics
The alg that changed imagery
Discrete cosine transform (DCT)
(Wikipedia)
Chapter 3 - 50
Interactive Computer Graphics
JPEG image compression
DCT to convert to frequency domain Perceptual modeling Coefficient quantization Entropy coding
Chapter 3 - 51
Interactive Computer Graphics
Research in image processing and computer vision
How would you clear up a blurry photograph?
How would you recognize a face in a photograph?
Chapter 3 - 52
Interactive Computer Graphics
Summary
Event loop programming Interaction and event handling in HTML State diagrams Painting Rubber banding Double buffering Basic image processing and convolutions "Photoshop Nano"
