Implemented data preprocessing #17

Merged

vascodias23 merged 1 commit from data_preprocessing into main

2025-11-02 23:31:34 +00:00

vascodias23 commented

2025-11-02 19:02:45 +00:00

(Migrated from github.com)

High-level features of the preprocessing module:

Data cleaning handles missing values, invalid records, and outliers based on domain knowledge from the data dictionary and project requirements.

Feature engineering creates temporal features (hour, day, weekend, rush hour), spatial features (Haversine distance, Manhattan distance, bearing), and contextual features (airport trips, night trips) as recommended in the project specifications.

Reproducibility is achieved through the scikit-learn compatible BaseEstimator and TransformerMixin interface, making it pipeline-friendly and compatible with MLflow tracking.

Flexible configuration allows enabling/disabling outlier removal and feature creation for different stages (EDA vs training).

Target Variable

The module computes trip_duration in seconds from the pickup and dropoff timestamps, which is the prediction target for the MLOps project.

Data Validation

Records are filtered to remove impossible values like negative durations, zero distances, invalid passenger counts (outside 1-9), and coordinates outside valid ranges.

High-level features of the preprocessing module: Data cleaning handles missing values, invalid records, and outliers based on domain knowledge from the data dictionary and project requirements. Feature engineering creates temporal features (hour, day, weekend, rush hour), spatial features (Haversine distance, Manhattan distance, bearing), and contextual features (airport trips, night trips) as recommended in the project specifications. Reproducibility is achieved through the scikit-learn compatible BaseEstimator and TransformerMixin interface, making it pipeline-friendly and compatible with MLflow tracking. Flexible configuration allows enabling/disabling outlier removal and feature creation for different stages (EDA vs training). Target Variable The module computes trip_duration in seconds from the pickup and dropoff timestamps, which is the prediction target for the MLOps project. Data Validation Records are filtered to remove impossible values like negative durations, zero distances, invalid passenger counts (outside 1-9), and coordinates outside valid ranges.

AndreFerreira5 (Migrated from github.com) approved these changes

2025-11-02 23:31:29 +00:00