Map customisation for running Autoware in CARLA/LGSVL

Key concepts

Autoware: An open-source software built on top of ROS which allows advanced control of virtual and physical Autonomous Vehicles. Autoware is supported by both CARLA and LGSVL, allowing perception and control of virtual autonomous agents. It consists of 3 layers: Sensing, Computing and Actuation.

  • Sensing uses a multitude of sensors including LIDAR, cameras, GNSS, and IMU systems to build up a picture of the environment. LIDAR generates a point cloud representation of its environment, but cameras allow for extraction of additional information, e.g., traffic lights and obstacle detection.
  • Computing acts to process the data received from sensing modules, including further perception processing to allow the vehicle to determine the state of the world around it. Decision and planning processes use this data to determine the best course of action for the vehicle given its current state and the predicted actions of entities around it. Crucially, the decision making process can also be overruled by driver interaction.
  • Actuation allows Autoware to control an Autonomous vehicle, within its known physical capabilities. Autoware’s actuation is compatible with both physical and simulated autonomous vehicles.

Simulators: The benefits of simulators in Autonomous Vehicle research are plentiful. They allow detailed examination of scenarios with a high danger or cost factor and facilitate repeated examination of any scenario which may not be feasible for practical trials. They also act as a safeguard against mistakes or bugs during trials and allow the implementation of easy scenario generation which can reduce testing time and increase the range of examples examined, including crucial ‘edge-cases’: scenarios which happen so rarely that predicting them manually can be virtually impossible. However, these benefits are only valid as long as a simulator accurately represents the world they are designed to replace. High-fidelity is essential. To this end, two state-of-the-art open-source simulators are identified:

  • CARLA: CARLA is a dedicated open-source simulator designed for researchers in Autonomous Vehicles. Built on Unreal Engine 4, it focuses on high fidelity simulation and providing a wide range of options for customisable scenarios. It was designed to implement a large number of aspects of urban driving, including pedestrians, crossings, and traffic rules. Effort is made to make these interactions realistic, for instance, pedestrians are heavily weighted to walk on pavements and cross at dedicated crossing points but will not always do so. Vehicles are ‘equipped’ with a variety of sensors that can mimic those required by AV control stacks, in most cases these are noisy representations of ground-truth information given by the simulator.
  • LGSVL: Similar in conception and purpose to CARLA, LGSVL is an open-source Unity-based simulator, designed by LG Electronics. This simulator has a modular framework separating environment, sensor, and dynamics and control systems into discrete aspects, and includes both a visual editor for scenario design and a Python API for agent control. The simulator allows for easy compatibility with autonomous driving controllers such as Autoware and Apollo. An additional emphasis is also placed on multi-entity systems, allowing multiple AVs to run in the same simulation. In terms of simulation construction, LGSVL’s scenarios are easier to construct manually, but lack some autonomy in pedestrian and other vehicle behaviours compared to CARLA’s. The simulator also emphasises high-fidelity sensor systems, in most cases separate from any ground-truth readings, as well as an extensibility for users to add more, including V2X systems.

Requirements

  • Ubuntu 18.04
  • ROS Melodic
  • Unreal Engine 4.24
  • Unity 2019.4.18f1
  • Autoware.AI 1.14.0
  • LGSVL 2021.1
  • CARLA source built
  • CARLA 0.9.10.1
  • carla-ros-bridge
  • carla-autoware

Usage and significance

While both CARLA and LGSVL allow users to create customised maps, these maps must be manually created in the editor-mode, which requires a lot of human intervention and is time-consuming. Moreover, although both simulators offer the option to directly import the 3D map models into the scene, the driving routes in these maps need to be manually annotated, if the corresponding road specification files (OpenDRIVE/Lanelet2/Apollo HD map) are not presented. While some software, such as RoadRunner, allows direct exporting of map models and road annotations to CARLA, the model still has to be manually created in the editor-mode.

More importantly, there does not exist a straightforward option to directly use Autoware within these customised maps. It should be noted that two types of files are required to enable Autoware running in any simulators, i.e., the Point Cloud Data (PCD) and vector maps. The PCD data is used for vehicle localisation and vector maps are needed for path planning. Both files are required and should be aligned properly to enable Autoware working on these customised maps.

Thus, it would be beneficial to develop an automatic pipeline that can automatically generate the customised map and associated Autoware files from the same source of data. Such a process would help to increase the amount of testing scenarios exponentially and dramatically speed up the process for replicating edge-cases happened in on-road driving. It would also be vital for virtual verification of AVs and can be used to test and exploit the capabilities of AV control stacks.

Gap in existing methods

While a variety of map-generation tools for Unity exist, several strong limitations exist which prevent them from being suitable for generating maps for use with Autoware. The limitations come from 3 main sources: cost (the process should ideally be open source end-to-end), Point Cloud Data generation, and road vector generation. All existing state-of-the-art methods feature at least one of these critical limitations which make them unsuitable.

Mapbox: An open source software designed to generate high-definition maps from OpenStreeMap data, charged with volume pricing. A dedicated SDK integrated with Unity allows the straightforward creation of many types of maps including urban cityscapes. With additional plugins, Mapbox can also create vectors representing roads from OSM data. However, the maps created do not necessarily generate corresponding point cloud data, and the road vectors lack the resolution to determine lanes or other details essential for AV control.

Citygen3D: At a cost of over £75 for the extension, Citygen3D is not an open source utility, making it unsuitable for an end-to-end process which relies on otherwise freely available tools. Otherwise, its functionality is comparable to Mapbox. The road generation is complex and detailed, but is almost entirely aesthetic, generating no vectors or data usable in any advanced manner.

WRLD Maps: With scaling functionality depending on pricing models, WRLD maps are generated using the WRLD app, and can then imported into Unity through a separate SDK. However, WRLD apps use a variety of features not useful for AV simulations, and lack many of those that are. Most notably, the maps are comparatively low-fidelity, and there is a complete absence of road vectors.

AccuCities: AccuCities is not a map generation tool but offers purchasing of high-definition maps of several UK cities in a variety of formats. However, these are naturally limited to regions which have already been created and offers no extensibility.

Overall, most of the state-of-the-art map generation methods have severe limitations which make them unsuitable for simulation with Autoware. Often this is because they are not open source, but even the ones that are do not generate the required files for Autoware to run fully. Thus, there is a considerable vacuum when it comes to an end-to-end process for procedurally generating maps for use with Autoware.

Procedure

  1. Download map from OpenStreetMap

    As the first step of creating customised maps, a unified data source describing the contents and specifications of the real world map should be sought. Owing to its open source nature and detailed available information, OpenStreetMap is hence selected as the data source for the proposed study . A script was developed based on the osmWebWizard from SUMO/tools to facilitate the filtering and downloading of osm data from the website and to convert the file to required format (direct export from the website has a node number restriction of 50000). The osm data consist of three primary components, node, way and relation, and contains additional information about elements within the area, such as coordinates, type of the element, speed limit, building height, etc. An example of area near London Bridge is shown in the figure below.

  2. Create road/building model and road annotation

    With the OpenStreetMap data downloaded, a model of the map should be created accordingly. We have created a unity-based map generation tool, but other commercialized map generation tools (listed in previous section) can also be used. A free sample of the same area provided by AccuCities (shown below) was used for this demonstration. While other 3D model format may also be supported by CARLA and LGSVL, FBX was preferred in this study.

    Along with the 3D map model, a conversion script was also created to generate OpenDRIVE file from the downloaded osm file. The OpenDRIVE format is a common language describing road networks, containing the geometry of roads, lanes and other objects , and is supported in both CARLA and LGSVL.

  3. Import map into CARLA/LGSVL

    With the 3D map model and OpenDRIVE file obtained for the same map, the next step is to import these files into the simulators. Both simulators have their own procedure for adding new maps.

    For CARLA, both files need to be placed inside the Import folder with a specific structure. The map can then be created by executing make import. Afterwards, the customised map can be inspected in the editor by running make launch, and a distributed map package can be created with make package ARGS="--packages=map_name”. The customised map loaded in CARLA is illustrated below, the pink denotes the routes generated from OpenDRIVE.

    This map package can be placed in the “Import” folder of any precompiled version of CARLA, and the map can be imported via “./ImportAssets.sh”. Loading the map can be achieved using the “PythonAPI/util/config.py” with argument “-m” to specify the name of this map. To demonstrate the effect of this map in CARLA, a manual control vehicle is created to drive within the scene, as shown in the recording below.



    For LGSVL, the FBX model file can be saved as a prefab and directly dragged into the scene. The road annotations can be added by selecting the OpenDRIVE file in the “Simulator/Import HD Map” function. Mesh colliders need to be added to the GameObjects, and Tag/Layer needs to be assigned accordingly. The scene will then need to be saved under a folder with the same name of the map, created under the External/Environments folder. Afterwards, the “Simulator/Build” function can be used to build the distributed map asset and uploaded to the WebUI. A visualization of the map inside LGSVL is shown below and the blue lines are the routes generated from OpenDRIVE.

    To demonstrate the effect of this map in LGSVL, a manual control vehicle is created to drive within the scene, as shown in the recording below.



  4. Generate files for Autoware As outlined in the previous section, two types of files (PCD and vector map) are needed to run Autoware on customised maps. To facilitate the recording of PCD, an automatic recording script was developed upon the CALRA’s Autopilot function and pcl_recorder from carla-ros-bridge. An AV with LIDAR sensor will be spawn at different locations in the map, and automatically navigate through the map in the Autopilot mode while recording the PCD data of its surrounding area. This process will run iteratively and will stop automatically once the entire map is covered. These PCD can then be concatenated to form the PCD of the customised map, as shown in the figure below.

    For the vector map generation, it should first be noted that there are two different types of vector map required by Autoware, i.e. the Autoware LaneLet2 and Aisan vector map. The Autoware Lanelet2 format is a derivative of the standard Lanelet2 vector map, which needs additional tags, such as “ele”, “local_x” and “local_y”. A conversion script was hence developed to directly generate Autoware Lanelet2 from the OpenDRIVE format.

    Meanwhile, the Aisan vector map consists of multiple csv files, each corresponding to a specific element of the road. However, it should be noted that the Aisan vector map is proprietary, and not much documentation can be found on its specification. Luckily, Autoware has released a function to generate the Aisan vector map from Autoware Lanelet2. It can be easily obtained via

    rosrun lanelet_aisan_converter lanelet2aisan _map_file:= _origin_lat:= _origin_lon:= _save_dir:=<optional: path to save dir>

  5. Set up Autoware

    With all the files obtained, they need to be put into the corresponding folder in the container of Autoware. This can be done either by using the docker cp command or by directly putting the files into the “carla-autoware/autoware-contents/maps” folder for CARLA and “shared_dir/autoware-data” for LGSVL on the host machine. For CARLA, all that is needed is to split the files into corresponding folder. Meanwhile, for LGSVL, a new folder with the map name needs to be created in the “shared_dir/autoware-data”. It is suggested to copy/paste the “data” and “my_launch” folder from another map and replace the PCD and vector maps. Additionally, paths to the TF, PCD and vector maps need to be specified in the my_map.launch as well.

  6. Simulation

    To start the simulation with Autoware, a CARLA server or LGSVL simulator needs to be running. Owing to the hardware restriction, a latency issue was encountered when using customised maps within CARLA, but the validity of the process/maps was confirmed by its developers. For LGSVL, a simulation needs to be manually created, specifying the map to be the uploaded customised map, vehicle to be Lexus2016RXHybrid or any vehicle model that support Autoware AI 1.14.0, set in the “Sensor Configuration”. The connection should be set as “localhost:9090”, if not otherwise specified.

    After the simulator is set, Autoware can be launch with runtime manager. Vehicle start and target position can be set via rviz. The simulation is then ready to go.


Project Details

Team

Panagiotis Angeloudis   Supervisor
Felix Feng   Researcher
Dan Mead   Researcher

Funding