Checklist
Background
Loading data in the centralized controller process is extremely expensive.
Potential Solution
Create a DataController and distributed workers for loading and processing data in multiple processes. The worker processes will send RTensors back to the controller. The real tensor data will transfer between the underlying workers.
Checklist
areal/api/. If not, please raise a refactor issue first.Background
Loading data in the centralized controller process is extremely expensive.
Potential Solution
Create a
DataControllerand distributed workers for loading and processing data in multiple processes. The worker processes will sendRTensors back to the controller. The real tensor data will transfer between the underlying workers.