Three data professionals working at their desks reviewing datasets and feature pipelines on monitors in a bright office

Data Preparation and Feature Engineering Checklist for Machine Learning

A data preparation and feature engineering checklist should protect the validity of a machine-learning decision, not simply make a dataset tidy. The crucial questions are whether each row represents the right unit, whether each feature would have existed at prediction time, whether transformations are learned only from training data and whether exactly the same logic can run in production.

Missing these controls can produce an impressive validation score that collapses after deployment. This checklist gives data scientists, analysts and machine-learning engineers ten evidence-based gates from problem definition to production monitoring. It is designed for structured organisational data, but the principles also apply to text, images and other model inputs.

Key takeaways

  • Define the prediction moment before selecting or engineering features.
  • Split data before fitting imputers, encoders, scalers or feature selectors.
  • Investigate missingness and outliers; do not treat them as automatic deletion tasks.
  • Build transformations as one reproducible pipeline for training and inference.
  • Approve features with evidence about lineage, leakage, stability, cost and monitoring.

Why data preparation determines model reliability

A model learns the relationships present in its training inputs. It cannot repair an incorrectly defined target, an unreliable join or information that would not exist when a real prediction is made. Data preparation therefore begins with the decision and the operating process, not with a cleaning function.

The scikit-learn common-pitfalls guidance identifies two particularly important failures: inconsistent preprocessing and data leakage. It recommends applying the same transformations to later data and learning preprocessing only from the training subset. Pipelines help enforce both requirements.

Gate Question Evidence to retain Failure prevented
1. Decision What will be predicted, for whom and when? Target, unit of analysis, prediction timestamp and action Solving the wrong problem
2. Provenance Where did each field originate? Source owner, definitions, extraction logic and lineage Untraceable or inconsistent inputs
3. Availability Would this value exist at prediction time? Event and availability timestamps Target or temporal leakage
4. Population Does the sample represent intended use? Coverage, exclusions, time range and subgroup profile Biased or obsolete learning data
5. Quality Are missing, invalid, duplicate and extreme values understood? Profiling report and treatment rules Silent corruption or arbitrary cleaning
6. Split Is evaluation data protected from learned transformations? Split strategy, keys, dates and random-state policy Optimistic validation results
7. Features Does every transformation have a valid rationale? Feature definition, owner, cost and expected direction Noise, proxies and unstable complexity
8. Selection Do features add robust out-of-sample value? Ablation, cross-validation and stability results Overfitting and redundancy
9. Pipeline Can identical logic run during training and inference? Versioned pipeline tests and schema contract Training-serving skew
10. Monitoring How will changes and failures be detected? Quality thresholds, drift measures and response owners Undetected production deterioration
EPW TIMECHECK data preparation and feature engineering pipeline
TIMECHECK keeps data meaning, point-in-time validity, evaluation separation and production consistency connected.

The ten-point data preparation and feature engineering checklist

1. Define the target, unit and prediction moment

Write one sentence: “For each [unit], at [time], predict [outcome] within [horizon] so that [owner] can [action].” A row might represent a machine-day, customer-month or service request—not merely whatever the source table contains.

Define how and when the label is observed. If the target is “failure within 30 days”, decide what counts as failure, how repeated events are handled and how long labels take to mature. Without this, features and evaluation windows cannot be judged.

2. Record provenance, definitions and reliable joins

Create a compact data contract for each source: owner, field meaning, key, granularity, update frequency, retention and known quality limits. Check that joins preserve the intended row count. Many-to-many joins can silently duplicate examples and over-weight some entities.

Trace derived fields back to source events. A variable called “open balance” may mean current balance in one system and month-end balance in another. A model cannot resolve conflicting business definitions.

3. Test point-in-time availability

For every candidate feature, ask whether the value would have been recorded and available to the production service at the prediction timestamp. Separate event time from processing or database-update time. A repair code entered after equipment failure is not a valid pre-failure feature even if it appears in the same historical record.

Point-in-time logic matters for aggregates. A 30-day transaction count must use only events preceding each historical prediction. Recalculating all rows from today’s completed data creates a view of the past that the model could never have possessed.

4. Design representative and protected splits

Split according to the way the model will encounter novelty. Random splitting may work for independent records, but time-based splitting is often more realistic for forecasting, and group-based splitting may be necessary when repeated observations belong to the same customer, patient, asset or site.

Create the split before learning any transformation. The test set must not influence imputation values, vocabulary, scaling, principal components or selected features. Use cross-validation within the training data for development, and reserve the holdout for an honest final estimate.

5. Profile quality before choosing treatments

Measure types, ranges, categories, uniqueness, missingness, duplicates and relationships across time and subgroups. TensorFlow Data Validation’s official workflow illustrates how descriptive statistics, schemas and anomaly detection can turn assumptions into executable checks.

Investigate why values are missing. Absence may reflect a process, eligibility rule or system outage, so a missing indicator can sometimes be informative. Likewise, an extreme value may be a genuine rare case, a unit error or a sensor fault. Treatment should follow cause and model sensitivity rather than one universal rule.

6. Encode and transform without hiding meaning

Choose transformations to match data and model requirements. Scaling can help distance-based and gradient methods; robust scaling may reduce sensitivity to extremes. Categorical encoding must consider cardinality, unseen values and ordering. Target encoding requires especially careful out-of-fold construction to avoid leakage.

Logarithmic transforms can stabilise skewed positive variables, but zeros, negatives and interpretability need explicit handling. Preserve units and inverse transformations where stakeholders must understand outputs.

7. Engineer domain-valid features

Good features express information available to the decision: recency, frequency, ratios, lags, rolling windows, interactions or domain states. Each definition should state its window, denominator, treatment of absence and update schedule.

Avoid proxies for protected or unavailable information without governance review. A variable can be statistically powerful and still be operationally invalid, unfair, expensive or unstable. Record why the feature should generalise beyond the development sample.

8. Select features through controlled comparison

Begin with a defensible baseline. Add feature groups through ablation tests and compare performance across folds, time periods and important segments. Filter, wrapper and embedded methods answer different questions; none replaces domain review.

Check redundancy and multicollinearity where interpretation or coefficient stability matters. Dimensionality reduction and embeddings may improve representation, but they increase lineage and explainability requirements. Retain a feature only when its benefit justifies its operational cost.

9. Package transformations as a reproducible pipeline

Version code, dependencies, feature definitions and schemas. The scikit-learn pipeline and composite-estimator documentation shows how transformations can be combined with estimators so that fitting and prediction follow the same sequence. The principle applies beyond one library.

Test training and inference parity with known inputs and expected outputs. Validate column presence, types, categories and ranges before prediction. Decide how unknown categories, missing sources and delayed features are handled; production should fail safely rather than improvise.

10. Monitor data and features after release

Monitor schema violations, missingness, range breaches, category growth, feature distributions and freshness. Compare production inputs with an appropriate reference, but do not assume every distribution shift damages performance. Connect alerts to investigation and outcome evidence.

Assign owners and response thresholds. A feature-quality incident may require fallback values, model rollback, pipeline suspension or source-system correction. Store sufficient lineage to identify which predictions and model versions used the affected data.

Worked example: maintenance-failure prediction

Suppose a maintenance team predicts whether a pump will fail within 30 days. Historical tables include sensor readings, work orders, inspections and repair outcomes. The first version achieves excellent validation results, but a leakage audit finds that “final work-order status” is updated after failure.

Candidate feature Point-in-time finding Decision Replacement or control
Final work-order status Usually completed after the outcome Reject Use status recorded at prediction time
Seven-day vibration mean Available from timestamped sensor events Retain Test sensor freshness and minimum coverage
Days since last inspection Knowable, but missing for new assets Retain with control Add missing indicator and new-asset rule
Site failure rate Risk of using future failures in the aggregate Revise Compute point-in-time expanding estimate from prior events
Asset serial number High-cardinality identifier with weak generalisation Reject as raw field Use governed asset-class and age features
Point-in-time audit of features for maintenance failure prediction
A maintenance feature is valid only if it was available before the prediction and can be reproduced in production.

After removing or rebuilding the leaking features, performance will probably fall. That is a useful correction: the lower result is a more credible estimate of future behaviour. The team can now compare valid feature groups, test recent time periods and design monitoring for sensor freshness and unseen asset classes.

Release checklist for a model-ready dataset

  • The target, unit, horizon and prediction timestamp are approved.
  • Every source and feature has an owner, definition and lineage record.
  • Joins, duplicates, exclusions and label maturity are tested.
  • Feature availability is verified at historical prediction time.
  • Training, validation and holdout separation precedes learned preprocessing.
  • Missing, invalid and extreme values have reasoned treatment rules.
  • Feature groups demonstrate stable out-of-sample value over a baseline.
  • The complete preprocessing pipeline is versioned and reproducible.
  • Training and inference transformations pass parity tests.
  • Production quality thresholds, owners and response actions are documented.

Develop production-ready data capability

EPW’s five-day Data Preparation and Feature Engineering for Machine Learning course covers prediction targets, provenance, sampling, missing data, outliers, leakage, numerical and categorical features, temporal features, selection, dimensionality reduction, reproducible pipelines, point-in-time accuracy and monitoring.

Professionals can also explore EPW’s Artificial Intelligence and Machine Learning Courses and the AI and machine learning article hub. To apply TIMECHECK to a real dataset, review the course outline, available dates and locations, or request tailored course details.

Sources and references

  1. scikit-learn, Common pitfalls and recommended practices, version 1.9 documentation, accessed 2 September 2026.
  2. scikit-learn, Pipelines and composite estimators, version 1.9 documentation, accessed 2 September 2026.
  3. TensorFlow, Get started with TensorFlow Data Validation, accessed 2 September 2026.
  4. National Institute of Standards and Technology, AI RMF Playbook, accessed 2 September 2026.
  5. EPW Training, Data Preparation and Feature Engineering for Machine Learning Course, accessed 2 September 2026.