End-to-End Imitation Learning for Autonomous Vehicle Steering on a Single-Camera Stream

Vehicles can follow roads based on a forward-looking camera, but this has to be done reliably in all circumstances. In daily traffic, they can encounter many unforeseen situations. Training for those situations in simulations should prepare them for such encounters, but this requires simulated worlds with enough complexity. In this paper, we compare different convolutional neural networks trained to follow the roads in one of the most complex environments available in the simulation environment CARLA: the map Town 3. Still, during training the vehicle encounters a disproportionate number of simple straight roads, so care has to be taken on the balance in the training set. End-to-end learning for autonomous vehicles have been shown before, but not for the complex worlds used in this paper. After the training, the vehicle can follow the road reliably in the training map, a behavior that can be transferred to a non-complex map with circumstances it has not seen before. Complex situations remain difficult to learn without high-level commands. The learned behavior has been validated on a map which is just released with the latest version of the CARLA simulator, Town 10HD. The Xception network architecture performs best in our benchmark with success rates of 34% and 90% for complex validation town Town 10HD and non-complex validation town Town 6 respectively.