Teaching Cars to Drive and UAVs to Race

Teaching Cars to Drive and UAVs to Race

Internship Description

At the Image and Video Understanding Lab (IVUL) at KAUST, we have developed a photo-realistic simulator (denoted UE4Sim) based on the open-source computer game engine Unreal Engine. The UE4Sim simulator has been designed to facilitate the integration of computer vision and machine learning techniques into a realistic looking 3D environment with the following advantages. (1) By facilitating the generation of 3D worlds, UE4Sim enables the quick and automatic acquisition of large amounts of labelled data to be used for training data-hungry machine learning models (specifically deep neural networks) targeting a multitude of computer vision and machine learning applications ranging from self-navigating cars/drones and aerial tracking to indoor 2D/3D scene understanding and 3D reconstruction. (2) UE4Sim provides simple-to-use connections with third party software to allow for real-time evaluation of AI techniques. The photo-realism of UE4Sim facilitates the transfer of the learned models to the real-world.

 

In this internship project, we plan to develop UE4Sim further, motivated by the goal of teaching a car to drive in previously unseen scenarios and unmanned aerial vehicles (UAVs) to race through obstacle courses. All this functionality will be done within UE4Sim with an ultimate aim at transfer this learned knowledge to real world vehicles.

​​

Deliverables/Expectations

Improvements on the UE4Sim simulator to make it more streamlined, efficient, and developer friendly for future development and integration with various types of learning (e.g. deep learning and reinforcement learning)
Development of deep learning methods to estimate future positions of the vehicles (called waypoints) directly from images (perception network)
Development of reinforcement learning methods to generate the appropriate vehicle controls, e.g. steering wheel angle or pitch/yaw/roll (control network)
System integration of the perception and control network within UE4Sim​

Faculty Name

Bernard Ghanem

Field of Study

​Electrical Engineering or Computer Science