Embodied Artificial Intelligence Safety

Spring 2025. 16-886. Monday / Wednesday 11:00-12:20.

Announcements

Week 8 Lecture Notes

Mar 24 · 0 min read

System-level anomalies [notes] and [slides] posted online.

Course Overview

Safety is a nuanced concept. For embodied systems, like robots, we commonly equate safety with collision-avoidance. But out in the “open world” it can be much more: for example, a safe mobile manipulator should understand when it is not confident about a requested task and understand that areas roped off by caution tape should never be breached. However, designing systems with such a nuanced understanding is an outstanding challenge, especially in the era of large robot behavior models.

In this graduate seminar class, we study the question of if (and how) the rise of modern artificial intelligence (AI) models (e.g., deep neural trajectory predictors, large vision-language models, and latent world models) can be harnessed to unlock new avenues for generalizing safety to the open world. From a foundations perspective, we study safety methods from two complementary communities: control theory (which enables the computation of safe decisions) and machine learning (which enables uncertainty quantification and anomaly detection). Throughout the class, there will also be several guest lectures from experts in the field. Students will practice essential research skills including reviewing papers, writing project proposals, and technical communication.

Prerequisites

The course is open to graduate students and advanced undergraduates. While there are no strict prerequisites, familiarity with sequential decision-making, machine learning, optimization, and probability are highly encouraged. Experience with high-level programming languages like Python or MATLAB are also strongly encouraged.

Schedule (Tentative)

Control-Theoretic Safety Foundations

Jan. 13

Course Overview [Slides]: Syllabus

Jan. 15

Sequential Decision-Making [Notes]

Jan. 20

No Class MLK Day

Jan. 22

Safety Filtering [Notes]: Data-Driven Safety Filters, Model Predictive Sheilding, Safety & Liveness of Filters

Jan. 27

Safety Filter Synthesis via Optimal Control [Notes]: HJ Reachability Overview, HJ Viscosity Solution

Jan. 29

Synthesis & Robust Safety I [Notes]: Differential Games, HJ Reach-Avoid Games I, HJ Reach-Avoid Games II

Feb. 3

Robust Safety II [Notes]

Feb. 5

Computational Frameworks for Safety [Notes]: HW #1 Due Discounted Reachability, ISAACS, DeepReach

Frontiers I

Feb. 10

Updating Safety Online [Notes] [Slides]: Reachability Adapted with Gaussian Processes, Local Updates, Parameterized Reachability

Feb. 12

Semantic Safety: Paper Reading Safety Representations from Language, SALT

Feb. 17

~~Belief-Space Safety~~ Project Brainstorming [Slides]: Deception Game, Analyzing Models that Adapt Online

Feb. 19

From Belief to Latent-Space Safety [Notes] [Slides]: Deception Game, Latent Safety Filters

Feb. 24

Latent-Space Safety: Paper Reading Latent Safety Filters, LS3

Feb. 26

Failure Monitoring & Recovery via VLMs: HW #2 Due: Feb 28 Paper Reading LLM Fallbacks, FOREWARN

Mar. 3

No Class Spring Break 🏝️

Mar. 5

No Class Spring Break 🏝️

Machine Learning & Statistical Safety Foundations

Mar. 10

Uncertainty Quantification I [Notes]: On the Calibration of Modern NNs, Prof. Eric Nalisnick’s research and talks

Mar. 12

Uncertainty Quantification II [Notes]: Mid-term Report Due: March 14Deep Ensembles, Deep Laplace Approx, Gaussian Processes Book, What Are Bayesian NN Posteriors Really Like?

Mar. 17

Uncertainty in Large Behavior Models: Paper Reading EnsembleDAgger, Robots that Ask for Help

Mar. 19

Guest Lecture Prof. Anushri Dixit (UCLA), Conformal Prediction [Video]: Gentle Intro to Conformal, Perceive With Confidence

Mar. 24

“System-level” Anomalies [Notes] [Slides]: Not All Errors, System-Level OOD, BYOVLA

Mar. 26

Risk-Aware Decision-Making: Paper Reading What is Risk in Robotics?, Risk-Calibrated Interaction

Mar. 31

Guest Lecture Ran (Thomas) Tian (UC Berkeley) Alignment [Slides]: CPL, Max Alignment Min Feedback

Frontiers II

Apr. 2

Guest Lecture Prof. Max Simchowitz (CMU), The Pitfalls of Imitation Learning when Actions are Continuous: HW #3 Due Pitfalls of IL

Apr. 7

Guest Lecture Dr. Masha Itkina (TRI), Out-of-Distribution and Failure Detection

Apr. 9

Controlling In-Distribution: Paper Reading In-D CBF, Lyapunov Density Models

Apr. 14

Guest Lecture Dr. Haruki Nishimura (TRI), Statistical Assurances for Learned Policies: Robot Learning as an Empirical Science, Misinterpretations of Statistical Tests

Apr. 16

Statistical Assurances: Paper Reading Statistical Safe Set Verification, How Generalizable is My BC Policy?

Project Presentations

Apr. 21

Project Presentations: Slides Due 11:59 pm ET, April 20 Presenters: (Arun, Steven), Tiffany, (Yishu, Yining H.), Emanuel, Chenyang, Yiqi, (Shuyang, Yikai, Alex)

Apr. 23

Project Presentations: Project Report Due May 1 Presenters: Adrian, Andrew, (Kush, Saksham, Saudamini), Jessie, Haochen, (Philip, Yuemin, Yining S.)