Calculate the dataset batch size according to available resources #16

New issue

Open

opened 2025-11-02 18:53:24 +00:00 by vascodias23 · 0 comments

vascodias23 commented

2025-11-02 18:53:24 +00:00

(Migrated from github.com)

To better adapt the data pipeline to the memory and computing power constraints, the batch size for loading data from storage should be calculated based on the available memory.

This allows for processing data in larger batches when conditions allow it, and shrink the batch size if the system is heavily resource-constrained.

To better adapt the data pipeline to the memory and computing power constraints, the batch size for loading data from storage should be calculated based on the available memory. This allows for processing data in larger batches when conditions allow it, and shrink the batch size if the system is heavily resource-constrained.