Rigorous numerics for optimizers
I use computer-assisted proofs to study quasiperiodic and chaotic structures, and routes to chaos, in adaptive optimizers such as RMSProp and Adam. My goal is to understand which mechanisms in an adaptive optimizer drive chaotic behavior.
Our recent RMSProp paper proves that, for positive-definite quadratic losses, trajectories eventually enter a compact trapping region even when the learning rate makes the minimizer unstable. We derive explicit bounds on this region and use numerical continuation to study invariant circles, tori, and chaotic motion within it.
