Advanced Robotics Applications In Engineering

Not every foundation model needs to be gigantic. We trained a 1.5M-parameter neural network to control the body of a humanoid robot. It takes a lot of subconscious processing for us humans to walk, maintain balance, and maneuver our arms and legs into desired positions. We capture this “subconsciousness” in HOVER, a single model that learns how to coordinate the motors of a humanoid robot to support locomotion and manipulation. We trained HOVER in NVIDIA Isaac, a GPU-powered simulation suite that accelerates physics by 10,000x faster than real time. To put the number in perspective, the robots undergo 1 year of intense training in a virtual “dojo”, but take only ~50 minutes of wall clock time on one GPU card. The neural net then transfers zero-shot to the real world without finetuning. HOVER can be *prompted* for various types of high-level motion instructions that we call “control modes”. To name a few: - Head and hand poses - can be captured by XR devices like Apple Vision Pro. - Whole-body poses - via MoCap or RGB camera. - Whole-body joint angles - Exoskeleton. - Root velocity command - Joysticks. What HOVER enables: - A unified interface for us to control the robot using whichever input devices are convenient at hand. - An easier way to collect whole-body teleoperation data for training.  - An upstream Vision-Language-Action model to provide motion instructions, which HOVER translates to low-level motor signals at high frequency. HOVER supports any humanoid that can be simulated in Isaac. Bring your own robot, and watch it come to life! It's a big teamwork from NVIDIA GEAR Lab and collaborators: Tairan He, Wenli Xiao, Toru Lin, Zhengyi Luo, Zhenjia Xu, Zhenyu Jiang, Jan Kautz, Changliu Liu, Guanya Shi, Xiaolong Wang Team leads: Jim Fan, Yuke Zhu Website: https://lnkd.in/g6WrJyRC Paper: https://lnkd.in/g99yWTPa

Car inspection drive by AI! 🔦 BMW Group has become the first automaker to use AI-driven robots at scale for paint inspection and processing, rolling out the technology at its German plant. These robots from KUKA inspect, sand, polish, and mark vehicle surfaces, ensuring higher quality and shorter lead times. Using pattern projection and advanced cameras, the system detects even the smallest flaws in the paintwork, creating a 3D image to guide the robots. Unlike traditional automation, these AI-powered robots adapt their process for each vehicle, performing 1,000 unique inspections daily. While robots handle most of the work, human workers still refine edges and tight spaces. AI assists by projecting laser guidance, ensuring precise manual finishing. 👨🏻🔧 BMW is now exploring further enhancements, such as real-time fault prevention and automated documentation. What a time to be a robotics guy! 😮💨 🔔 Hit the bell on my profile to never miss a robot story.

Researchers at the City University of Hong Kong have developed miniature, caterpillar-like robots that might change the way we deliver medications and perform surgeries inside the human body. What Are These Millirobots? - Biodegradable and Soft: Made from a gelatin-like material combined with iron oxide microparticles, these tiny robots are about the size of a fingernail. Their soft composition allows them to move through the body without harming delicate tissues. - Magnetic Control: The iron oxide particles make the robots responsive to external magnetic fields. This means doctors can further control the direction their movement precisely, guiding them to specific locations within the body. - Inspired by Insects: Mimicking the walking and gripping abilities of caterpillars, these robots can roll, fold, and even grasp small objects with their claw-like appendages. This flexibility enables them to navigate complex internal environments like the gastrointestinal tract. How Do They Work? The robots can be coated with medications. Once guided to the target area, they unfold their bodies to release the drug directly where it's needed, potentially increasing the treatment's effectiveness and reducing side effects. Their ability to grasp and transport objects opens up possibilities for performing surgical tasks without the need for large incisions or invasive instruments. After completing their mission, the robots naturally break down over a few days into harmless substances, eliminating the need for surgical retrieval. While still in the experimental phase, these tiny robots have shown promise in laboratory tests. The researchers successfully guided them through a model of the gastrointestinal system, demonstrating their potential for real-world medical use. Would you be comfortable with such technology being used in medical treatments? #innovation #technology #future #management #startups

Teaching robots to learn only from RGB human videos is hard! In Feel The Force (FTF), we teach robots to mimic the tactile feedback humans experience when handling objects. This allows for delicate, touch-sensitive tasks—like picking up a raw egg without breaking it. There are three super simple ideas that makes FTF work: 1. record human touch responses using a latex glove retrofitted with AnySkin. 2. output tactile responses from Point Policy. 3. use a PD controller to pick with desired tactile response. For the delicate tasks we look it, we find that learning without touch feedback is quite poor. This is intuitive as vision based policies find it difficult to reason about critical forces before breakage. This work was led by Ademi Adeniji & Zhuoran Chen, and a wonderful collaboration with Vincent Liu, Venkatesh Pattabiraman, Siddhant Haldar, Raunaq Bhirangi & Pieter Abbeel. For the paper, videos and more details: https://lnkd.in/eVHp5tfF

If you told farmers 10 years ago that robots powered by sunlight would replace chemicals, it would sound ridiculous. These solar-powered rovers use vision AI to identify and remove weeds at the plant level. No herbicides, no operators… not even lasers or anything crazy. Just going back to the original method of dealing with weeds, pulling them out, just done by machines now. The hard part is not building a robot that works in one field. It is building one that works in every field. Different crops, different soil, different weeds, different growth stages, different geographies. Farming has no standard environment. So Aigen trained their system using NVIDIA Cosmos foundation models and Isaac Sim pipelines to simulate millions of agricultural scenarios before deploying anything in the real world. On the ground, each rover runs inference using NVIDIA Jetson Orin to distinguish crops from weeds while moving. These systems need to become cheaper and more accessible than traditional methods. Once that happens, adoption becomes obvious, especially as demand for food keeps scaling globally. Farmers spend billions on herbicides. If robots can replace even part of that, you change both the cost structure and the environmental footprint at the same time. Follow Endrit Restelica for more.

China is leaving everyone behind in robotics. The world's first humanoid with autonomous battery swapping. UBTECH's Walker S2 can replace its own power source without human help: → Walks to the charging station → Removes depleted battery → Inserts fresh battery → Returns to work This changes everything. Traditional robots: Work 5 hours → Stop for charging → Wait Walker S2: Work 5 hours → Swap battery in seconds → Keep working The implications are staggering: • 24/7 factory operations with zero downtime • Robots that maintain themselves • No human intervention needed • Continuous industrial productivity Meanwhile Figure AI is taking the opposite approach. Their F.03 battery delivers 5 hours of runtime but can't be swapped. CEO Brett Adcock believes integration beats modularity. My takeaway: I think fast charging matters tremendously when every minute charging is lost productivity. We're now witnessing robots that can literally take care of themselves. I’m interested to see these two philosophies battle it out over the years to come. The future just got a lot more interesting. Follow me Alex Banks for daily AI highlights and insights. P.S. I did a full breakdown of Figure's approach in my recent newsletter. I cover the most important AI developments each week. Subscribe here: https://lnkd.in/eYrGWHej

The real challenge in AI today isn’t just building an agent—it’s scaling it reliably in production. An AI agent that works in a demo often breaks when handling large, real-world workloads. Why? Because scaling requires a layered architecture with multiple interdependent components. Here’s a breakdown of the 8 essential building blocks for scalable AI agents: 𝟭. 𝗔𝗴𝗲𝗻𝘁𝗶𝗰 𝗙𝗿𝗮𝗺𝗲𝘄𝗼𝗿𝗸𝘀 Frameworks like LangGraph (scalable task graphs), CrewAI (role-based agents), and Autogen (multi-agent workflows) provide the backbone for orchestrating complex tasks. ADK and LlamaIndex help stitch together knowledge and actions. 𝟮. 𝗧𝗼𝗼𝗹 𝗜𝗻𝘁𝗲𝗴𝗿𝗮𝘁𝗶𝗼𝗻 Agents don’t operate in isolation. They must plug into the real world:  • Third-party APIs for search, code, databases.  • OpenAI Functions & Tool Calling for structured execution.  • MCP (Model Context Protocol) for chaining tools consistently. 𝟯. 𝗠𝗲𝗺𝗼𝗿𝘆 𝗦𝘆𝘀𝘁𝗲𝗺𝘀 Memory is what turns a chatbot into an evolving agent.  • Short-term memory: Zep, MemGPT.  • Long-term memory: Vector DBs (Pinecone, Weaviate), Letta.  • Hybrid memory: Combined recall + contextual reasoning.  • This ensures agents “remember” past interactions while scaling across sessions. 𝟰. 𝗥𝗲𝗮𝘀𝗼𝗻𝗶𝗻𝗴 𝗙𝗿𝗮𝗺𝗲𝘄𝗼𝗿𝗸𝘀 Raw LLM outputs aren’t enough. Reasoning structures enable planning and self-correction:  • ReAct (reason + act)  • Reflexion (self-feedback)  • Plan-and-Solve / Tree of Thought These frameworks help agents adapt to dynamic tasks instead of producing static responses. 𝟱. 𝗞𝗻𝗼𝘄𝗹𝗲𝗱𝗴𝗲 𝗕𝗮𝘀𝗲 Scalable agents need a grounding knowledge system:  • Vector DBs: Pinecone, Weaviate.  • Knowledge Graphs: Neo4j.  • Hybrid search models that blend semantic retrieval with structured reasoning. 𝟲. 𝗘𝘅𝗲𝗰𝘂𝘁𝗶𝗼𝗻 𝗘𝗻𝗴𝗶𝗻𝗲 This is the “operations layer” of an agent:  • Task control, retries, async ops.  • Latency optimization and parallel execution.  • Scaling and monitoring with platforms like Helicone. 𝟳. 𝗠𝗼𝗻𝗶𝘁𝗼𝗿𝗶𝗻𝗴 & 𝗚𝗼𝘃𝗲𝗿𝗻𝗮𝗻𝗰𝗲 No enterprise system is complete without observability:  • Langfuse, Helicone for token tracking, error monitoring, and usage analytics.  • Permissions, filters, and compliance to meet enterprise-grade requirements. 𝟴. 𝗗𝗲𝗽𝗹𝗼𝘆𝗺𝗲𝗻𝘁 & 𝗜𝗻𝘁𝗲𝗿𝗳𝗮𝗰𝗲𝘀 Agents must meet users where they work:  • Interfaces: Chat UI, Slack, dashboards.  • Cloud-native deployment: Docker + Kubernetes for resilience and scalability. Takeaway: Scaling AI agents is not about picking the “best LLM.” It’s about assembling the right stack of frameworks, memory, governance, and deployment pipelines—each acting as a building block in a larger system. As enterprises adopt agentic AI, the winners will be those who build with scalability in mind from day one. Question for you: When you think about scaling AI agents in your org, which area feels like the hardest gap—Memory Systems, Governance, or Execution Engines?

Researchers at Hong Kong University MaRS Lab have just published another jaw dropping paper featuring their safety-assured high-speed aerial robot path planning system dubbed "SUPER". With a single MID360 lidar sensor they repeatedly achieved autonomous one-shot navigation at speeds exceeding 20m/s in obstacle rich environments. Since it only requires a single lidar these vehicles can be built with a small footprint and navigate completely independent of light, GPS and radio link. This is not just #SLAM on a #drone, in fact the SUPER system continuously computes two trajectories in each re-planning cycle—a high-speed exploratory trajectory and a conservative backup trajectory. The exploratory trajectory is designed to maximize speed by considering both known free spaces and unknown areas, allowing the drone to fly aggressively and efficiently toward its goal. In contrast, the backup trajectory is entirely confined within the known free spaces identified by the point-cloud map, ensuring that if unforeseen obstacles are encountered or if the system’s perception becomes uncertain, the system can safely switch to a precomputed, collision-free path. The direct use of LIDAR point clouds for mapping eliminates the need for time-consuming occupancy grid updates and complex data fusion algorithms. Combined with an efficient dual-trajectory planning framework, this leads to significant reductions in computation time—often an order of magnitude faster than comparable SLAM-based systems—allowing the MAV to operate at higher speeds without sacrificing safety. This two-pronged planning strategy is particularly innovative because it directly addresses the classic speed-safety trade-off in autonomous navigation. By planning an exploratory trajectory that pushes the speed envelope and a backup trajectory that guarantees safety, SUPER can achieve high-speed flight (demonstrated speeds exceeding 20 meters per second) without compromising on collision avoidance. If you've been tracking the progress of autonomy in aerial robotics and matching it to the winning strategies emerging in Ukraine, it's clear we're likely to experience another ChatGPT moment in this domain, very soon. #LiDAR scanners will continue to get smaller and cheaper, solid state VSCEL based sensors are rapidly improving and it is conceivable that vehicles with this capability can be built and deployed with a bill of materials below $1000. Link to the paper in the comments below.

Humanoid robots are making robotics visible again. But the real challenge is not simply building a robot that can walk, lift, or manipulate objects. The real challenge is building the full stack around it. Any robot operating in the real world depends on far more than one impressive subsystem: • Sensors and compute • Embedded software • Perception and AI models • Planning and control • Safety systems • Cloud connectivity • Fleet operations • Cybersecurity • Data pipelines • Integration with customer infrastructure This is where robotics becomes difficult. A humanoid demo may show capability. But a production robot must show reliability, safety, maintainability, and economic value, day after day, in messy real-world environments. That requires deep integration across hardware, software, AI, cloud, safety, and operations. In my view, the real moat in robotics will not be one component. It will be integration complexity. The companies that scale robotics successfully will be those that can turn many complex subsystems into one reliable product experience. This also changes how robotics teams need to be built. The strongest robotics organizations will not look like pure hardware teams or pure AI teams. They will look like full-stack systems organizations, combining AI/ML, embedded software, controls, cloud platforms, safety engineering, cybersecurity, product integration, and field operations. Humanoids may be the visible symbol of the next robotics wave. But the real winner will be the team that can integrate the full stack well enough to make robots reliable, safe, and useful in the real world. Curious how others see this: Is the next robotics moat hardware, AI, or full-stack integration? #Robotics #AI #Humanoids #AutonomousSystems #IndustrialAI #SystemsEngineering

computer vision + robotics 🔥 🔥 🔥 Over the last few days, I trained my Reachy Mini robot to track and follow a human face. I fine tuned RF-DETR Nano on a custom face detection dataset. The system maps pixel coordinates of a detected face to yaw and pitch commands for head control. I plan to release the code soon. The main issue is inertia. The robot head has significant mass. At higher angular velocities, inertia causes overshoot. During the next control step, the controller overcompensates. This behavior leads to oscillations. ⮑  RF-DETR: https://lnkd.in/dVQRpvWU #computervision #opensource #objectdetection #robotics