optimize resource usage on prod #20

New issue

Open

opened 2025-11-22 13:48:53 +00:00 by AndreFerreira5 · 0 comments

AndreFerreira5 commented

2025-11-22 13:48:53 +00:00

(Migrated from github.com)

our deployment servers are really tight on resources, so we need to optimize as much as we can.

here are some optimization suggestions (feel free to add some if you want):

scale the default batch size value (on data loader) according to available system ram
batch logs to prevent I/O overhead
when downloading another file from the dataset, check if it already is on disk and if yes check if it is complete (to make sure another thread is still downloading it)

our deployment servers are really tight on resources, so we need to optimize as much as we can. here are some optimization suggestions (feel free to add some if you want): - [ ] scale the default batch size value (on data loader) according to available system ram - [ ] batch logs to prevent I/O overhead - [ ] when downloading another file from the dataset, check if it already is on disk and if yes check if it is complete (to make sure another thread is still downloading it)