Air Canvas: Hands-On
Drawing with MediaPipe
Explore how to transform hand movements into digital strokes using
MediaPipe and OpenCV, creating an intuitive air drawing experience.
Agenda
Introduction to Hand Tracking
Understanding MediaPipe's role in real-time hand landmark detection.
Setting Up the Environment
Installing necessary libraries and initialising video capture.
Drawing Logic: From Gesture to Line
How finger movements translate into drawing actions on the canvas.
Erasing and Interacting
Implementing the clear canvas functionality and application controls.
Live Demonstration & Q&A
Showcasing the Air Canvas in action and addressing queries.
The Power of MediaPipe Hands
Real-time Hand Landmark Detection
MediaPipe Hands is a machine learning solution that
detects D landmarks of a hand from a single frame.
It9s highly accurate and performs in real-time, making it
ideal for interactive applications.
Robust hand and finger tracking.
Supports single or multiple hands.
Outputs D coordinates for each landmark.
Core Components: Setup and Initialisation
The foundation of our Air Canvas lies in correctly setting up MediaPipe and OpenCV for video processing and rendering.
import cv2
import numpy as np
import mediapipe as mp
mp_hands = mp.solutions.hands
hands = mp_hands.Hands(max_num_hands=1)
mp_draw = mp.solutions.drawing_utils
cap = cv2.VideoCapture(0)
canvas = None
prev_x, prev_y = 0, 0
OpenCV (cv ): For camera input, image manipulation, and display.
NumPy (np): Essential for array operations, particularly for our drawing canvas.
MediaPipe (mp): The core library for hand detection and landmark extraction.
Finger Detection Logic
Accurately identifying which fingers are extended is crucial for translating gestures into drawing commands.
def fingers_up(landmarks):
tips = [4, 8, 12, 16, 20]
fingers = [landmarks[tips[0]].x < landmarks[tips[0] - 1].x] # Thumb check
for i in range(1, 5):
fingers.append(landmarks[tips[i]].y < landmarks[tips[i] - 2].y) # Other fingers
return fingers
Landmark Coordinates: We use the Y-coordinates of finger tips relative to their knuckle points.
Thumb Exception: The thumb's orientation is checked differently (X-coordinate) due to its unique movement.
Boolean Array: The function returns a list of booleans, indicating which fingers are up.
Real-time Processing Loop
The main loop continuously captures video, processes hand landmarks, and updates the canvas.
while cap.isOpened():
ret, frame = cap.read()
if not ret: break
frame = cv2.flip(frame, 1)
if canvas is None:
canvas = np.zeros_like(frame) * 255 # Initialise white canvas
rgb = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
results = hands.process(rgb)
if results.multi_hand_landmarks:
for lm in results.multi_hand_landmarks:
mp_draw.draw_landmarks(frame, lm, mp_hands.HAND_CONNECTIONS)
# ... drawing logic continues ...
Frame Capture: Reads frames from the webcam and flips them horizontally for natural mirroring.
Canvas Initialisation: Creates a blank canvas overlay the same size as the video frame.
Hand Processing: Converts the frame to RGB and processes it with MediaPipe to find hand landmarks.
Drawing and Erasing Interactions
Gesture recognition dictates when to draw, where to draw, and when to clear the canvas.
Drawing Mode Line Smoothing Erasing Action
When only the index finger is The `prev_x`, `prev_y` variables Raising all five fingers clears the
extended, the system enters drawing ensure continuous lines by storing entire canvas, providing a quick way
mode. The index finger's tip acts as the previous point, creating a to restart or undo.
the pen, drawing a blue line on the smooth stroke between frames.
canvas.
Visual Output: Merging Layers
The final step involves overlaying the drawing canvas onto the live camera feed for a seamless visual experience.
merged = cv2.addWeighted(frame, 0.5, canvas, 0.5, 0)
cv2.imshow("Air Canvas", merged)
if cv2.waitKey(1) & 0xFF == ord('q'):
break
Blending Frames: `cv .addWeighted` merges the live video frame and the drawing canvas with a % opacity blend for
each, allowing both to be visible.
Display Window: The combined image is displayed in a window titled "Air Canvas."
Exit Condition: Pressing 'q' terminates the application, releasing camera resources.
Potential Enhancements & Future Scope
Colour Palette: Implement gestures to change drawing
colours dynamically.
Brush Size Control: Vary brush thickness based on
hand distance from the camera or finger spread.
Shape Recognition: Recognise specific hand gestures
for drawing predefined shapes (circles, squares).
Key Takeaways & Next Steps
Real-time Hand Tracking Simple Yet Effective Code Endless Possibilities
MediaPipe provides a powerful The core logic for drawing and This project serves as a foundation
and accessible tool for gesture- erasing is concise and easy to for more complex touchless
based interaction. extend. interfaces.
For next steps, experiment with adding new features or integrating with other Python libraries for advanced visual effects.
The full source code is available on GitHub.
