Data preprocessing rework #23

Merged

vascodias23 merged 5 commits from data-preprocessing-rework into main

2025-11-24 12:08:33 +00:00

vascodias23 commented

2025-11-23 23:16:31 +00:00

(Migrated from github.com)

The restructuring includes pipeline reordering, adding more customizable and default parameters and reorganizing their use in the pipeline.

Some logic inside the pipeline itself was also removed or changed as a result of the new requirements.

Note: Due to obvious reasons, the previous processer unit tests in #19 are outdated as of now, the ones implemented here should serve as the starting point.

The data preprocessor was reworked as a result of the changed project requirements. In summary, a large portion of redundant or otherwise useless code was removed, and many remaining features were adjusted or enhanced. The code was also restructured and is now more uniform. Tests were also implemented in a basic form. The restructuring includes pipeline reordering, adding more customizable and default parameters and reorganizing their use in the pipeline. Some logic inside the pipeline itself was also removed or changed as a result of the new requirements. Note: Due to obvious reasons, the previous processer unit tests in #19 are outdated as of now, the ones implemented here should serve as the starting point.

AndreFerreira5 (Migrated from github.com) requested changes

2025-11-24 10:50:48 +00:00

AndreFerreira5 (Migrated from github.com) left a comment

also, i think we could default the creation of certain features to True, like the hours of the day, the day of the week and the month, which you already implemented, because they are essential to improve the training.

really nice work! your changes correctly improve upon the initial version and adapt to the new dataset. however, one tiny error needs to be addressed before merging. the trip duration is calculated from _tpep_pickup_datetime_ and _tpep_dropoff_datetime_, however, these columns are not dropped after this, which leads to target leakage, making the model cheat and adapt to these values. just drop them when it's safe to do so and i think it is ready to merge also, i think we could default the creation of certain features to True, like the hours of the day, the day of the week and the month, which you already implemented, because they are essential to improve the training.

AndreFerreira5 (Migrated from github.com) approved these changes

2025-11-24 12:06:38 +00:00

AndreFerreira5 (Migrated from github.com) left a comment

looks good to me!