Skills useful to learn for robotics engineering
👋 Hi, this is Gergely with a subscriber-only issue of the Pragmatic Engineer Newsletter. In every issue, I cover challenges at Big Tech and startups through the lens of engineering managers and senior engineers. If you’ve been forwarded this email, you can subscribe here. Skills useful to learn for robotics engineeringHelpful software engineering, AI engineering, and robotics fundamentals to know for getting into robotics. Also: advice about studying this exciting discipline at university
Robotics is a very hot industry, and today, the hottest place within it is humanoid robotics. We previously published two deepdives on this topic with Sandor Felber, who’s been a Robot Learning Researcher at MIT, and a Robotics R&D Engineer at Tesla in California, among other roles. The articles cover relevant topics at the intersection of AI, robotics, and software engineering. Since the last deepdive, Sandor has cofounded Nyro Humanoids, an early-stage startup headquartered in San Francisco that aims to build and deploy humanoid robots in the real world. In the third and final deepdive in this series, we take a close look at skills useful for joining this field, covering:
Previous issues cover: Robotics basics for software engineers (part 1):
Robotics for software engineers: humanoid robots (part 2):
With this, it’s over to Sandor: We're standing on the threshold of a robotics revolution. Just as OpenAI's ChatGPT “moment” transformed how we think about artificial intelligence (AI), the robotics industry is approaching its own breakthrough. This looks less like a step-like change, and more of a gradual transformation – one that will fundamentally change how we approach physical AI. At Nyro Humanoids, we're creating the intelligence that powers humanoid systems capable of operating where humans cannot – or should not – go. From disaster response missions to save lives, to potentially dangerous construction sites, and toxic industrial environments that require hazardous activities which can put health at risk, our autonomous humanoid robots represent the cutting edge of what we call ‘physical AI’. Our mission is to deploy intelligent humanoid robots in high-risk environments to protect human life and expand the boundaries of what's possible. Whether it's navigating collapsed buildings during search and rescue operations, handling hazardous materials, or operating in challenging conditions, we are developing the cognitive capabilities that enable robots to think, adapt, and act autonomously when every second counts. The same breakthroughs that have revolutionized language models are now being applied to physically embodied intelligence. There are computers with arms and legs – robots! – which can understand their environment, make complex decisions, and execute precise physical actions in real time. What follows is a comprehensive guide to the skills, technologies, and mindset that I’ve developed on my journey. Whether you're a software engineer looking to make the leap into robotics, a student considering the next move, or you’re simply curious about this rapidly-evolving field, this deepdive is a roadmap for becoming part of the robotics revolution. The future isn't just about smarter software, it's about intelligence that can move, manipulate, and operate in the physical world. At Nyro Humanoids, we are building it one training run at a time – and we’re also hiring. Building robots is a multidisciplinary endeavour that blends pragmatic software engineering, AI expertise, and a deep understanding of robotics fundamentals. What follows is a breakdown of the key skills that have proven invaluable to me every day in engineering robotics software and hardware. 1. Software engineering skillsSoftware, electrical, and mechanical engineering are the backbone of robotics. Let’s consider software engineering, where skills that prioritize performance, scalability, and reliability, are critical catalysts required to build robots that succeed in real-world applications. Depending on the kind of robotics you get into, some areas of interest might be: Communication protocols, such as:
Multithreading and multiprocessing: managing parallel processes in C/C++, Python, or Rust, for robotics systems, is crucial. Often, you may want to stream in two, or with high latency sensitivity. Vectorization: leveraging parallelization in modern CPU/GPU architectures such as NVIDIA’s RTX 5090s graphics card, with GPUs to speed up computationally-heavy tasks. Some pragmatic examples:
CUDA and cuDNN: CUDA is NVIDIA’s parallel computing platform and API. cuDNN stands for CUDA Deep Neural Network. These frameworks allow for:
Here’s a plain-english cheat sheet for speeding up robot ML and onboarding:
Rules of thumb:
Complexity analysis for resource-constrained devices. It’s necessary to ensure the coded algorithms can scale efficiently, as a system’s complexity expands to multiple tasks, or sets of tasks. For example, if the model-based or learned controller (one that controls a robot using some sort of a neural network) requires 50ms to execute a small subset of potential tasks, it will probably be hard to scale it to process many other tasks, while maintaining a sufficiently high control frequency for agile ones. Control frequency is how often a robot's control system updates or executes its control loop. Being able to maintain control frequency while processing additional tasks is often related to robustness, agility, or speed-related metrics. 2. AI skillsAs mentioned above, robotics increasingly intersects with AI, and this is especially true of tasks that require autonomy, perception, and decision making. I personally found online resources from Andrej Karpathy, Pieter Abbeel, and some other greats of robotics to be more useful than many books which quickly become obsolete in this rapidly transforming field – no pun intended. Areas it’s good to be proficient in: Machine Learning (ML) basics: Core principles for training models and extracting insights from data. For more, check out our deepdive, The machine learning toolset. Data science and probability theory: both are used to understand uncertainty, and how to calculate and make probabilistic decisions. Much of robotics runs on uncertainty that must be tamed. Decision-making systems and cognitive science: modelling behaviour, navigation, and task planning. Cognitive science is the study of the mind and its processes, which can be highly relevant, especially when constructing humanoid robots. Deep learning and representational learning: useful for developing perception systems for vision or audio. Deep learning is a subset of machine learning utilizing neural networks for tasks like classification and regression analysis. Representational learning is the process of extracting meaningful patterns from raw machines. This allows robots to develop useful abstractions for their environments and tasks. A book I enjoyed reading on multi-agent reinforcement learning is “Multi-Agent Reinforcement Learning: Foundations and Modern Approaches”. Reinforcement learning (RL) and imitation learning: used to teach robots to learn optimal actions through trial and error and via human demonstrations. A good resource on this is Spinning Up by OpenAI. Diffusion models and multi-agent systems: Leveraging cutting-edge approaches for multi-robot collaboration and planning for more efficient routing and trajectories. Quantization and pruning: Reducing model size and inference latency by lowering precision (e.g., INT8 quantization) and removing redundant weights for efficient deployment on edge devices. Quantization and pruning complement each other: prune redundant weights, then store the survivors in INT8 to slash model size and latency. Train with quantization-aware training, where every forward and backward pass mimics 8-bit math, so the network learns weight values and activation ranges that hold up after real quantization, giving a compact, edge-friendly model, with almost zero accuracy loss. Note from Gergely: There are plenty of books and online resources on these topics, while search engines like Perplexity or Kagi can provide recommendations. For example, for the query: “What are books on Diffusion models and multi-agent systems?” The search engine returns several suggestions that can be good starting points, if books are your thing. Search by the format you want. Full subscribers to Pragmatic Engineer get a year of Perplexity Pro, and 3 months of Kagi Ultimate for free. 3. Robotics fundamentalsA solid grounding in mathematics, physics, and hands-on engineering, is non-negotiable for designing, building and deploying robots:
These varied skills combine to overcome the inherent complexity of robotics. Each contributes to the ultimate goal of creating functional, scalable, and reliable robots that perform effectively in the real world. 4. Advice for studying a Master’s in RoboticsPursuing a postgraduate degree in robotics is a strategic move for mastering interdisciplinary skills, preparing for this rapidly-evolving field, and unlocking opportunities in academia, industry, and entrepreneurial ventures. Opting for university could be a worthwhile investment if you’re serious about getting involved, regardless of age. If that sounds appealing, I have some tips for making the most of it... Subscribe to The Pragmatic Engineer to unlock the rest.Become a paying subscriber of The Pragmatic Engineer to get access to this post and other subscriber-only content. A subscription gets you:
|
Comments
Post a Comment