Embodied Artificial Intelligence Safety

Spring 2025. 16-886. Monday / Wednesday 11:00-12:20.

Announcements

Week 6 Lecture Notes

Feb 21 · 0 min read

Ken’s guest lecture on belief-space and latent-space reachability [notes] and posted online safety updates [slides].

Course Overview

Safety is a nuanced concept. For embodied systems, like robots, we commonly equate safety with collision-avoidance. But out in the “open world” it can be much more: for example, a safe mobile manipulator should understand when it is not confident about a requested task and understand that areas roped off by caution tape should never be breached. However, designing systems with such a nuanced understanding is an outstanding challenge, especially in the era of large robot behavior models.

In this graduate seminar class, we study the question of if (and how) the rise of modern artificial intelligence (AI) models (e.g., deep neural trajectory predictors, large vision-language models, and latent world models) can be harnessed to unlock new avenues for generalizing safety to the open world. From a foundations perspective, we study safety methods from two complementary communities: control theory (which enables the computation of safe decisions) and machine learning (which enables uncertainty quantification and anomaly detection). Throughout the class, there will also be several guest lectures from experts in the field. Students will practice essential research skills including reviewing papers, writing project proposals, and technical communication.

Prerequisites

The course is open to graduate students and advanced undergraduates. While there are no strict prerequisites, familiarity with sequential decision-making, machine learning, optimization, and probability are highly encouraged. Experience with high-level programming languages like Python or MATLAB are also strongly encouraged.

Schedule (Tentative)

Control-Theoretic Safety Foundations

Jan. 13

Course Overview [Slides]: Syllabus

Jan. 15

Sequential Decision-Making [Notes]

Jan. 20

No Class MLK Day

Jan. 22

Safety Filtering [Notes]: Data-Driven Safety Filters, Model Predictive Sheilding, Safety & Liveness of Filters

Jan. 27

Safety Filter Synthesis via Optimal Control [Notes]: HJ Reachability Overview, HJ Viscosity Solution

Jan. 29

Synthesis & Robust Safety I [Notes]: Differential Games, HJ Reach-Avoid Games I, HJ Reach-Avoid Games II

Feb. 3

Robust Safety II [Notes]

Feb. 5

Computational Frameworks for Safety [Notes]: HW #1 Due Discounted Reachability, ISAACS, DeepReach

Frontiers I

Feb. 10

Updating Safety Online [Notes] [Slides]: Reachability Adapted with Gaussian Processes, Local Updates, Parameterized Reachability

Feb. 12

Semantic Safety: Paper Reading Safety Representations from Language, SALT

Feb. 17

~~Belief-Space Safety~~ Project Brainstorming [Slides]: Deception Game, Analyzing Models that Adapt Online

Feb. 19

From Belief to Latent-Space Safety [Notes] [Slides]: Deception Game, Latent Safety Filters

Feb. 24

Latent-Space Safety: Paper Reading Latent Safety Filters, LS3

Feb. 26

Failure Monitoring & Recovery via VLMs: HW #2 Due Paper Reading LLM Fallbacks, FOREWARN

Mar. 3

No Class Spring Break 🏝️

Mar. 5

No Class Spring Break 🏝️

Machine Learning & Statistical Safety Foundations

Mar. 10

Bayesian Uncertainty Quantification: Gaussian Process Lecture Notes from Drew Bagnell, GPs Book

Mar. 12

Ensembles: Mid-term Report Due: March 14Ensembles, EnsembleDAgger

Mar. 17

Uncertainty in Large Behavior Models: Paper Reading Diffusion Policy, TBD

Mar. 19

Guest Lecture Prof. Anushri Dixit (UCLA), Conformal Prediction: Gentle Intro to Conformal, KnowNo, Perceive With Confidence

Mar. 24

Alignment: CPL, Max Alignment Min Feedback

Mar. 26

Risk-Aware Decision-Making: Paper Reading What is Risk in Robotics?, Risk-Calibrated Interaction

Mar. 31

“System-level” Anomalies: Not All Errors, System-Level OOD, BYOVLA

Apr. 2

Guest Lecture Prof. Max Simchowitz (CMU), Mathematical Foundations of Robotic Behavior Cloning: HW #3 Due

Frontiers II

Apr. 7

Guest Lecture Dr. Masha Itkina (TRI), Out-of-Distribution and Failure Detection

Apr. 9

Controlling In-Distribution: Paper Reading In-D CBF, Lyapunov Density Models

Apr. 14

Guest Lecture Dr. Haruki Nishimura (TRI), Statistical Assurances for Learned Policies

Apr. 16

Statistical Assurances: Paper Reading Statistical Safe Set Verification, How Generalizable is My BC Policy?

Project Presentations

Apr. 21

Project Presentations: Slides Due 11:59 pm ET, April 20 Presenters TBD

Apr. 23

Project Presentations: Project Report Due May 1 Presenters TBD