Image based Model Predictive Controller for Autonomous Driving

With cameras being one of the most vital sensors for perception and planning it is more intuitive to design controllers that are able to operate directly on the camera data. In this work we present two approaches for designing a model predictive controller (MPC) that is able to directly operate in the perspective coordinates and show their equivalence to the standard MPC formulation. Consequently, it eliminates the need for dedicated modules for converting the output of the planner and estimating the state of system in the 3D coordinates, thereby enabling a lean design of the system architecture. We apply this method for the task of automated lane following and lane changing, a common use case arising in autonomous driving, and demonstrate its effectiveness.