Harnessing the Power of AI with Windows Ink
Download as PPTX, PDF2 likes503 views
The document discusses the features and advantages of the Windows Ink platform, highlighting its strength in supporting analog and digital pen usage among U.S. adults. It emphasizes increased productivity, user engagement, and the wide adoption of pen-enabled devices, as well as technical details about ink analysis and recognition capabilities. Lastly, it provides resources for learning more about the platform and opportunities to engage with the development team.
Harnessing the Power of AI with Windows Ink
- 4. type this
- 5. 3D in Windows 10 Windows Ink Microsoft Edge Faster to done Microsoft Office Cortana More creative, more productive 3rd in purchase intent 212m U.S. adults use analog or digital pen Processor/Ram/Storage Battery Life Pen Switch Storage from HDD to SSD Increase screen resolution Windows Hello 2-in-1 purchase intent when shown new scenarios
- 6. 3D in Windows 10 Windows Ink Microsoft Edge Faster to done Microsoft Office Cortana More creative, more productive 3rd in purchase intent 212m U.S. adults use analog or digital pen 10x more pen enabled devices than iOS or Android 2x More pen devices year over year 2x More user engagement year over year 8pts higher average NPS scores for pen attached systems 60% Top 10 revenue producing apps use Windows Ink 50% More apps using Windows Ink in Windows Store
- 9. UWP and Win32 The Windows Ink Platform Low Level APIs / Engine Controls HID Hardware Pointer (PointerPoint, WM_Pointer) Radial Controller DirectInk Ink Presenter Rendering InkStrokeContainer Low Latency Beautiful Ink Core WetStroke Update Source Custom Dry Built-in Brushes and Drawing Attributes Acceleration Off-thread Input Stroke Creation Prediction Smoothing XAML Controls InkCanvas InkToolbar TextBox SmartInk InkAnalysis Local Ink Classification Layout Analysis Handwriting Recognition Shape Recognition InkAnalysis Cloud Ink Classification Layout Analysis Handwriting Recognition Shape Recognition Cloud Service APIs
- 10. UWP and Win32 The Windows Ink Platform Low Level APIs / Engine HID Hardware Pointer (PointerPoint, WM_Pointer) Radial Controller DirectInk Ink Presenter Rendering InkStrokeContainer Low Latency Beautiful Ink Core WetStroke Update Source Custom Dry Built-in Brushes and Drawing Attributes Acceleration Off-thread Input Stroke Creation Prediction Smoothing SmartInk InkAnalysis Local Ink Classification Layout Analysis Handwriting Recognition Shape Recognition InkAnalysis Cloud Ink Classification Layout Analysis Handwriting Recognition Shape Recognition Cloud Service APIs Controls XAML Controls InkCanvas InkToolbar TextBox
- 11. <InkCanvas x:Name=“m_inkCanvas”/> <InkToolbar TargetInkCanvas=“{x:Bind m_inkCanvas}”/> XAML Ink Canvas XAML Ink Toolbar
- 13. Delete Words Delete Lines Insert words into a sentence Join two words Add a new line Overwrite Words
- 14. UWP and Win32 The Windows Ink Platform Low Level APIs / Engine HID Hardware Pointer (PointerPoint, WM_Pointer) Radial Controller SmartInk InkAnalysis Local Ink Classification Layout Analysis Handwriting Recognition Shape Recognition InkAnalysis Cloud Ink Classification Layout Analysis Handwriting Recognition Shape Recognition Cloud Service APIs Controls XAML Controls InkCanvas InkToolbar TextBox DirectInk Ink Presenter Rendering InkStrokeContainer Low Latency Beautiful Ink Core WetStroke Update Source Custom Dry Built-in Brushes and Drawing Attributes Acceleration Off-thread Input Stroke Creation Prediction Smoothing
- 15. • Ballpoint Pen • Highlighter • Tilt Pencil Model how analog pencil works Natural pencil inking – dynamic range, layering • Erasing Stroke erase – erases full stroke Erase all • Stencils Built-in Brushes and Erasing
- 16. Pen Hardware & Ecosystem Update Ink Acceleration Using GPU preemption, Windows no longer has to wait for the next scanout to get ink on the screen. Microsoft Pen Program Microsoft Pen
- 17. UWP and Win32 The Windows Ink Platform Low Level APIs / Engine HID Hardware Pointer (PointerPoint, WM_Pointer) Radial Controller Controls XAML Controls InkCanvas InkToolbar TextBox DirectInk Ink Presenter Rendering InkStrokeContainer Low Latency Beautiful Ink Core WetStroke Update Source Custom Dry Built-in Brushes and Drawing Attributes Acceleration Off-thread Input Stroke Creation Prediction Smoothing SmartInk InkAnalysis Local Ink Classification Layout Analysis Handwriting Recognition Shape Recognition InkAnalysis Cloud Ink Classification Layout Analysis Handwriting Recognition Shape Recognition Cloud Service APIs
- 19. Ink Analysis Root WritingRegion Drawing Drawing DrawingWritingRegion … … ListItem x2Line Line InkWord InkWord InkWord InkWord … … InkBullet Paragraph
- 20. For Each Text Node • RecognizedText – The string representing the recognized text in the node Note: Line, ListItem, and Paragraph nodes also have a RecognizedText property • TextAlternates – The list of alternates for the recognized text • Understand slanted text via RotatedBoundingRect Text Recognition Avani hello, halo, hollo
- 21. Shape Recognition Triangles Equilateral Triangles Isosceles Triangles Right Triangles Squares Rectangles Diamonds Quadrilaterals Parallelograms TrapezoidsPentagons Hexagons Circles Ellipses Ink Drawing
- 22. Microsoft Confidential Demo - InkEditor
- 23. Important InkAnalysis Concepts and Tips
- 24. Microsoft Confidential Stateless Ink Analysis Find the nodes you’re interested in Extract strokes Send to InkAnalyzer Analyze the strokes Do cool stuff var inkAnalyzer = new InkAnalyzer(); private async void AnalyzeButton_Clicked(object sender, RoutedEventArgs e) { if (!inkAnalyzer.IsAnalyzing) { inkAnalyzer.ClearDataForStrokes(strokes); var strokes = inkCanvas.InkPresenter.StrokeContainer.GetStrokes(); inkAnalyzer.AddDataForStrokes(strokes); var results = await inkAnalyzer.AnalyzeAsync(); } }
- 25. Microsoft Confidential Stateful (Incremental) Ink Analysis Find the nodes you’re interested in Strokes changed Update InkAnalyzer Analyze the strokes Do cool stuff private void InkPresenter_StrokesCollected(InkPresenter sender, InkStrokesCollectedEventArgs args) { inkAnalyzer.AddDataForStrokes(args.Strokes); } private void InkPresenter_StrokesEraseded(InkPresenter sender, InkStrokesErasedEventArgs args) { foreach (var stroke in args.Strokes) { inkAnalyzer.RemoveDataForStroke(stroke.Id); } }
- 26. Microsoft Confidential Stateful (Incremental) Ink Analysis Find the nodes you’re interested in Strokes changed Update InkAnalyzer Analyze the strokes Do cool stuff private void StrokesUpdated(IReadOnlyList<uint> updatedStrokeIds) { var strokeContainer = inkCanvas.InkPresenter.StrokeContainer; foreach(var id in updatedStrokeIds) { inkAnalyzer.ReplaceDataForStroke(strokeContainer.GetStrokeById(id)); } } private async void AnalyzeButton_Clicked(object sender, RoutedEventArgs e) { if (!inkAnalyzer.IsAnalyzing) { var results = await inkAnalyzer.AnalyzeAsync(); } }
- 27. Microsoft Confidential var results = await _inkAnalyzer.AnalyzeAsync(); var root = inkAnalyzer.AnalysisRoot; // Query for particular node kind var drawings = root.FindNodes(InkAnalysisNodeKind.InkDrawing); var paragraphs = root.FindNodes(InkAnalysisNodeKind.Paragraph); var words = root.FindNodes(InkAnalysisNodeKind.InkWord); // Document tree traversal foreach (var paragraph in writingRegion.Children) {…} foreach (var word in line.Children) {…} var line = word.Parent; Query Document Structure Find the nodes you’re interested in Strokes changed Update InkAnalyzer Analyze the strokes Do cool stuff
- 28. Microsoft Confidential var results = await _inkAnalyzer.AnalyzeAsync(); var root = inkAnalyzer.AnalysisRoot; var drawings = root.FindNodes(InkAnalysisNodeKind.InkDrawing); foreach (var drawing in drawings) { var shape = drawing as InkAnalysisInkDrawing; if(shape.DrawingKind == InkAnalysisDrawingKind.Circle) { convertInkToXAMLShape(shape); var strokeIds = shape.GetStrokeIds(); inkAnalyzer.removeDataForStrokes(strokeIds); foreach(var id in strokeIds) { strokeContainer.GetStrokeById(id).Selected = true; } strokeContainer.DeleteSelected(); } } Mapping between strokes and InkAnalysisNode
- 29. Microsoft Confidential Use Prior Knowledge Find the nodes you’re interested in Strokes changed Update InkAnalyzer Analyze the strokes Do cool stuff private void InkPresenter_StrokesCollected(InkPresenter sender, InkStrokesCollectedEventArgs args) { inkAnalyzer.AddDataForStrokes(args.Strokes); foreach (var stroke in args.Strokes) { inkAnalyzer.SetStrokeDataKind(stroke.Id, InkAnalysisStrokeKind.Writing); //inkAnalyzer.SetStrokeDataKind(stroke.Id, InkAnalysisStrokeKind.Drawing); } }
- 32. Microsoft Confidential Cloud Service For Smart Ink Consistent Ink Recognition Experience Across Devices and Platforms Faster and Frequent Improvements to the core recognition capabilities Personalized Recognition with the Microsoft Cloud and Microsoft Graph Compute Power for larger ML models for better accuracy
- 33. Microsoft Confidential Reachouttous! • Learn more: • aka.ms/WindowsInkSample • aka.ms/DigitalInkBlog • aka.ms/WinHEC2017 • aka.ms/InputScope • aka.ms/InkPresenterClass • aka.ms/HandwritingView • Code Samples • aka.ms/SimpleInkSample • aka.ms/ComplexInkSample • aka.ms/InkAnalysisSample Interested in being a co-engineering partner for our cloud solution, or have an ink related question for the team? Contact us here: [email protected] Follow us on Twitter for the latest ink and pen news from our team! Twitter: WindowsInk