Privately Owned Vehicle Work Group Meeting - 2026/03/02 - Slot 1&2

[TranHuuNhatHuy]

Agenda

• Milestones
• VisionPilot 0.9 development
• Testing and performance
• Code Quality & Configuration
• EgoLanes
• AutoSpeed
• ObjectFinder
• AutoSteer
• ZENOH integration
• CARLA integration
• E2E Research and Literature Review

VisionPilot now has its now branding image!

Milestones

• VisionPilot 0.9 release by end of February and March.
• VisionPilot 1.0 should be ready by end of June for production-ready.

VisionPilot 0.9 development

• Vision Pilot 0.9 requires adding auto speed with longitudinal control for autonomous driving. Currently being undertaken by @atanasko and @pranavreddy23 .
• Speed planning and control:
  • Speed planning will incorporate RSS logic to maintain safe distance from lead vehicle.
  • Speed control uses simple rules: match lead car speed if safe distance maintained.
  • Speed limit will be set as static config parameter, defaulting to 70 mph or 120 kph.
  • Speed planning will limit speed on high curvature roads to minimize lateral force.
• Lane departure warning:
  • Expected to be added to VisionPilot 0.9.
  • Warning triggered when drift ratio exceeds 0.5 of lane width.
• Forward collision warning and AEB features planned for future implementation.
• Automotive cameras should meet minimum HD resolution with 8 megapixels or so to be considered sufficient.
• There might be several software redesigns and major codebase refactoring soon.

VisionPilot longitudinal + lateral integration

@pranavreddy23 has recently merged PR #248, featuring the longitudinal perception pipeline (AutoSpeed, ObjectFinder/CIPO) into the existing lateral control system for 0.9 release, including:

• Separating the workload into dedicated, asynchronous threads:
  • Capture
  • Lateral Inference
  • Longitudinal Inference
  • Unified Display.
• Implementing a 10 FPS double-buffered capture to prevent frame backlog and enforced cv::Mat.clone() to resolve shallow-copy memory bugs across threads.
• Building a synchronized display that overlays both lateral (lane masks, steering) and longitudinal (bounding boxes, CIPO) outputs onto a single standard-resolution frame of 1280 x 720.

Steering angle fix

@atanasko recently fixed a steering angle bug in AutoSteer. PR merged in PR #247.

CARLA simulation pipeline using scenario runner for VisionPilot 0.5

@atanasko successfully finished CARLA simulation pipeline using scenario runner for VisionPilot 0.5 at PR #246, including:

• Modified version of Visionpilot 0.5 to expose steering angle via ROS2 node.
• CARLA ROS2 bridge nodes to publish CARLA video stream and publish VisionPilot steering angle.

Code Quality & Configuration

As VisionPilot starts gaining more and more attentions from fellow developers, it is imperative to make VisionPilot, and thus PoV repo, better accessible to the public. This includes cleaning the codes and repos, with proper documentations and supports from the PoV maintainer.

• @TranHuuNhatHuy is currently working on codebase refactoring and improving documentations within the next few weeks.
• Also a user encountered fp32 compatibility issues when running models on Jetson hardware:
  • @TranHuuNhatHuy is to take charge of ensuring x86/Jetson compatibility for fp16/fp32 handling.
  • Meanwhile, @atanasko is reviewing codes from that user for Jetson compatibility issues.

Regarding some recent concerns and agreements:

• Parameters should be stored in config files rather than hardcoded to avoid recompilation.
• As mentioned above, minor code changes (fp16 conversion, checkpoint loading) required for Jetson compatibility.
• Code quality gate implementation needed before publishing to GitHub to prevent broken releases.
• Issues:
  • Code structure inconsistency across developers makes development and bug fixing more difficult.
  • Limited public support available through GitHub Issues, unclear how many users test independently.

Lite version of current E2E models

Previously a team from Alternative Embedded Innovation has successfully revamped a bit of our current E2E models - SceneSeg, Scene3D, EgoLanes - into "lighter/lite" corresponding versions, achieving 3x speed while maintaining fairly the same accuracy.

Models added here: PR #245.

EgoLanes

@TranHuuNhatHuy

• Current accuracy limitations:

  • Lane detection struggles with occlusions, shadows, and non-standard road surfaces.
  • Sometimes the model incorrectly detects road damage, sidewalk edges, and bus stop markings as lane lines.
  • The model also exhibits false positives on curvy roads with extreme lighting and surface conditions.
  • Robustness (speed & accuracy) should be improved a bit further.

• Performance optimization:

  • Auto Speed runs at 30 fps on AMD hardware while Ego Lanes requires optimization.
  • Ego Lanes uses inefficient net backbone and segmentation causing performance bottleneck.
  • Keypoint estimation or detection approach could replace segmentation for faster Ego Lanes.

AutoSpeed

@pranavreddy23 @atanasko

• Integration for AutoSpeed is done by @pranavreddy23 .

Object Finder

@pranavreddy23

• @pranavreddy23 has finished debugging velocity estimation in Object Finder, and successfully integrated it into VisionPilot.

ZENOH Integration

@evshary @m-zain-khawaja

CY's team has completed the Zenoh pipeline for SceneSeg, Scene3D and DomainSeg using the VisionPilot/common folder libs - allowing us to run the models with both ONNX runtime and TensorRT backends using ZENOH with the same shared framework as has been used for ROS2.

Previously, CY's team merged the CARLA-Zenoh vision pipelines and demos (PR #225), as well as latency benchmarking (PR #224).

CARLA camera data is transported through Zenoh and consumed by VisionPilot-style pipelines, including raw video subscription, scene segmentation, domain segmentation, and Scene3D. The PR also includes build/run helpers, and a demo video to illustrate the end-to-end workflow. There are also fixes on latency issues and benchmarkings.

E2E Research and Literature Review

@auo355 and Mrs. Zaynab El Mawas

https://docs.google.com/document/d/1ls-7LhYuCL1SbgiJBoCRg62UZ9FdcgLv5uA2zSYTIkY/edit?usp=sharing

Zoom Meeting Video Recording

Video Meeting Link

Please contact the work group lead (@m-zain-khawaja) to request access to a recording of this meeting.

[TranHuuNhatHuy]

First slot meeting

• Anthony and Thomas Lin successfully reproduced the demo of SodaSim. There is a bit "lag" in the real-time AutoSpeed results.

• Attendance:
  • Tran
  • Anthony
  • Thomas Lin

[ZaynabEM]

@ZaynabEM Attended Slot 2

[pranavreddy23]

@pranavreddy23 Chaired Slot 2

[atanasko]

attend slot 2