Skip to content

Initial TRT bindings#498

Draft
wizenink wants to merge 1 commit intochelsea0x3b:mainfrom
wizenink:tensorrt
Draft

Initial TRT bindings#498
wizenink wants to merge 1 commit intochelsea0x3b:mainfrom
wizenink:tensorrt

Conversation

@wizenink
Copy link
Copy Markdown
Contributor

This PR aims to add support for TensorRT to cudarc.

As TensorRT is a C++ only lib, bindgen automatic generation does not work here, so a cpp wrapper is created to flatten TensorRT into C code.

Right now, we are not downloading redist files from nvidia, that would be a big TODO.

Working ATM:

  • Some basic tests
  • TensorRT linked and working with CUDA 13 and CUDA 12

Implementation details:

  • TensorRT thin wrapper always statically linked (Very basic wrapper, I dont see the benefit of trying to dynamic load it, and CC only supports static lib generation)
  • TensorRT code not being fetched on bindings_generator

Missing:

  • Feature gating TRT code in the cpp wrapper depending on the cuda version, maybe passing some defines onto the header coming from rust feature flags
  • Expand the trt wrapper to expose more TRT features, this is only a small POC
  • Fix some deprecation warnings on the TRT plugin code
  • Safe wrappers (This is a big one)

@chelsea0x3b maybe you can check it out and just check if the implementation logic is right or if maybe another path should be taken to implement this

@chelsea0x3b
Copy link
Copy Markdown
Owner

@wizenink honestly this looks much cleaner than I expected, I think we could totally move forward with this

@wizenink
Copy link
Copy Markdown
Contributor Author

Nice :) Will focus on having feature gates synced between rust and the C wrapper for now, and leave fetching trt redist automatically for the future

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants