Following complex step-by-step procedures is a routine part of everyday life, yet the length and structural complexity of such tasks make them highly error-prone. Designing effective human-centered AI assistants therefore requires systems that can perceive user actions, reason over procedural structure, and deliver timely, interpretable feedback. In this talk, I introduce *CaptainCook4D*, a large-scale egocentric 4D dataset comprising 384 recordings (94.5 hours) of individuals preparing recipes in real kitchen environments, designed to support the study of procedural understanding and error-aware assistance. Building on this dataset, I present a unified framework for Perceptually-Enabled Task Guidance based on Neuro-Symbolic Dynamic Probabilistic Models (NSDPMs). The framework integrates multimodal perception with explicit task representations in the form of recipe task graphs and first-order probabilistic rules, enabling robust inference of user progress, step dependencies, and error states under noisy, real-world conditions. I further describe an interactive augmented reality system deployed on HoloLens 2 that operationalizes this framework for real-time user guidance. The system combines object detection, spatial mapping, and step prediction in a low-latency pipeline, continuously estimating procedural state using NSDPMs. This enables the AR interface to adapt instructions, highlight relevant objects in context, and provide timely alerts for preparation, measurement, timing, and technique errors. The talk concludes with design insights and open challenges for building scalable, error-aware AR assistants that support complex procedural tasks.

Deep learning has revolutionized fields such as image recognition, robotics, and speech processing. However, it often struggles in critical scenarios, struggling with challenges like misinterpreting occlusions, susceptibility to adversarial attacks, and poor generalization across distributional shifts. By integrating symbolic and probabilistic reasoning into deep learning, these limitations can be addressed, providing systems that are more interpretable, consistent, and capable of handling uncertainty. Neuro-symbolic-probabilistic models (NSPMs) merge deep learning’s perceptual strengths with structured reasoning, making AI both more reliable and explainable. However, inference in these models remains a challenge—exact inference is NP-hard and generally intractable in practical applications. In this talk, I will present my ongoing research on “neural solvers” that enable scalable and accurate inference within NSPMs, making them viable tools for real-time systems. I will also showcase how these solvers improve efficiency, reducing inference times from seconds to microseconds. Through experimental results and practical use cases, I will demonstrate how these advancements enable the deployment of NSPMs in time-critical applications, where rapid and accurate decision-making is essential.