How to Enable Camera-Lidar Fusion for Simulators?

[Somdit]

Hi everyone,

I am trying to use LiDAR-Camera fusion for object detection when using the simulators (both CARLA and AWSIM). Right now I am focusing on the Autoware-carla interface. What I did is to launch Autoware simulator interface through the following command:

ros2 launch autoware_launch e2e_simulator.launch.xml map_path:=$HOME/autoware_map/Town01 vehicle_model:=sample_vehicle sensor_model:=awsim_sensor_kit simulator_type:=carla carla_map:=Town01 perception_mode:=camera_lidar_fusion

Since I believe both AWSIM and carla_ros bridge only publish raw camera data to /sensing/camera/traffic_light/image_raw for traffic light detection instead of object detection, I modify part of the code here:

https://github.com/autowarefoundation/autoware.universe/blob/main/simulator/autoware_carla_interface/src/autoware_carla_interface/carla_ros.py#L137

to publish camera data to /sensing/camera/camera0/image_rect_color so that YOLOX can receive the sensor data.

I also noticed that YOLO is not launched by default, thus I launched YOLO using the following command: ros2 launch autoware_tensorrt_yolox yolox.launch.xml and I checked that objects could be properly labeled (one traffic cone I spawned) when I visualized the topic /tensorrt_yolox/out/image

It looks like the topic /perception/object_recognition/detection/rois0, which I think is used by the perception module, is showing that the traffic cone is detected (as shown in the figure below).

However, I am unsure what the next step is to properly use LiDAR-Camera fusion as the ego vehicle won't stop when the traffic cone is only detected by the YOLO but not the LiDAR detection model. Are there any additional steps I need to do to properly enable sensor fusions?

Any guide would be appreciated!

[badai-nguyen]

@Somdit Thank you for your question.
Since there are some steps for object detection from rois and pointcloud fusion, It is very helpful if you can visualize and clarity what step we lost the traffic cone by monitoring some topics such as:

• /perception/object_recognition/detection/clustering/camera_lidar_fusion/objects
• /perception/object_recognition/detection/objects
• /perception/object_recognition/objects

[Somdit]

Hi @badai-nguyen, thank you so much for your reply

As you can see from the Rviz screenshot, the traffic cone was detected by Yolo but was not labeled in the scenario
I have monitored the three topics you have mentioned in rqt, both /perception/object_recognition/detection/objects and /perception/object_recognition/objects are not showing any objects (as shown by the second image).

While the /perception/object_recognition/detection/clustering/camera_lidar_fusion/objects included one object, when I visualize this topic in Rviz (as shown by the first image), the detected object is actually not valid since it belongs to the vector map.

So it looks like the traffic cone was lost on all three topics. This is why I assume I might have missed some steps when launching Autoware with camera-lidar fusion, and the object detection from the camera sensor is not correctly fused with lidar detection.

Please let me know if you need any additional information, thanks for your help!

[badai-nguyen]

@Somdit I'm sorry for late response and thank you for your sharing and detailed explanation.
Based on your observation, it seem the Traffic Cone roi was fail to fuse with pointcloud by roi_pointcloud_fusion node.
Could you recheck:

• pointcloud before fusion node (/perception/object_recognition/detection/pointcloud_map_filtered/pointcloud)
• If there are some pointcloud of cone remained, I think you need to check the timestamp/delay between pointcloud and rois topic.

[Somdit]

@badai-nguyen No worries, I truly appreciate your response.

• I have visualized the topic /perception/object_recognition/detection/pointcloud_map_filtered/pointcloud in Rviz and the pointcloud of traffic cone indeed remained (as shown in the figure, a few clustered points in front of the car).

• I also checked the delay between topics pointcloud before fusion and rois0 using the below script, and the output of delay seems to be fine (0.05 ~ 0.10 secs).

Are there anything else I should check? Is my way of using camera-lidar fusion correct, in which I have to launch yolox after launching autoware using ros2 launch autoware_tensorrt_yolox yolox.launch.xml?

Thank you!

class TimestampChecker(Node):
    def __init__(self):
        super().__init__('timestamp_checker')

        qos_profile = QoSProfile(reliability=ReliabilityPolicy.BEST_EFFORT, depth=10)

        self.pointcloud_sub = self.create_subscription(
            PointCloud2,
            '/perception/object_recognition/detection/pointcloud_map_filtered/pointcloud',
            self.pointcloud_callback,
            qos_profile
        )
        self.objects_sub = self.create_subscription(
            DetectedObjectsWithFeature,
            '/perception/object_recognition/detection/rois0',
            self.rois_callback,
            10
        )

        self.last_pointcloud_time = None
        self.last_rois_time = None

    def pointcloud_callback(self, msg):
        self.last_pointcloud_time = msg.header.stamp
        self.check_delay()

    def rois_callback(self, msg):
        self.last_rois_time = msg.header.stamp
        self.check_delay()

    def check_delay(self):
        if self.last_pointcloud_time and self.last_rois_time:
            delay = abs((self.last_pointcloud_time.sec + self.last_pointcloud_time.nanosec * 1e-9) -
                        (self.last_rois_time.sec + self.last_rois_time.nanosec * 1e-9))
            self.get_logger().info(f"Delay between PointCloud and ROIs: {delay:.4f} seconds")

def main(args=None):
    rclpy.init(args=args)
    node = TimestampChecker()
    rclpy.spin(node)
    node.destroy_node()
    rclpy.shutdown()

[badai-nguyen]

@Somdit Thank you for your clarification.

I also checked the delay between topics pointcloud before fusion and rois0 using the below script, and the output of delay seems to be fine (0.05 ~ 0.10 secs).
As my experience, since the lidar's and camera's rates are the same (10hz) so the difference between their header time is usually stable. Could check why its range is relatively large?

• Additionally, I recommend you checking the parameters for fusion which define in the launcher repository:
  https://github.com/autowarefoundation/autoware_launch/blob/322c686f4fcda4b7b6f4c6c17740e4968f1405d1/autoware_launch/config/perception/object_recognition/detection/image_projection_based_fusion/fusion_common.param.yaml#L3-L6
• You also can find the documentation for the parameters here https://autowarefoundation.github.io/autoware.universe/main/perception/autoware_image_projection_based_fusion/

Please let me know if any I can assist.

[Somdit]

Hi, @badai-nguyen

As my experience, since the lidar's and camera's rates are the same (10hz) so the difference between their header time is usually stable. Could check why its range is relatively large?

Are you saying that even if the output of delay varied between either 0.05 or 0.10 secs, this is actually relatively large? What values should I anticipate when measuring the delay between these two intervals?

Do you think it is possible that Carla Bridge is actually publishing the sensor data at different rates? If that's the case, I might investigate further into the bridge to see if I can find a way to align them.

Thanks again.

[badai-nguyen]

Hi @Somdit

Are you saying that even if the output of delay varied between either 0.05 or 0.10 secs, this is actually relatively large? What values should I anticipate when measuring the delay between these two intervals?

It is depended on match_threshold parameter you set. Please refer here

Do you think it is possible that Carla Bridge is actually publishing the sensor data at different rates? If that's the case, I might investigate further into the bridge to see if I can find a way to align them.

Honestly, I do not have experience with Carla, so I cannot provide specific comments on that. However, since the fusion result depends significantly on how precise the synchronization between the lidar and camera is, you need to ensure that the timestamps and scan phases of each sensor are correctly synchronized.

BTW, I created a PR (autowarefoundation/autoware_universe#9481) to add a debug option for roi_pointcloud_fusion node that might be useful for you to debug this issue.

[Somdit]

Hi @badai-nguyen
Thanks for adding the debug option, this will definitely be helpful for me to debug the issue. After updating to the latest Autoware with your implementation, I tried to launch roi_pointcloud_fusion by setting the debug model to true,

ros2 launch autoware_image_projection_based_fusion roi_pointcloud_fusion.launch.xml debug_mode:=true

but I was not able to find debug topics like the ones you mentioned /perception/object_recognition/detection/clustering/roi_cluster/roi_pointcloud_fusion/debug/image_raw0..6

Am I launching it in a correct way? Do you mind giving me details of how to enable the debug mode to visualize these images you mentioned?