Feature Engineering & Selection

What it is

Feature engineering transforms raw data into inputs a model can use effectively — encoding categoricals, scaling numerics, extracting date/text features, and creating interaction terms. Feature selection then narrows that set down to what actually improves the model.

Key points

• Encoding: one-hot for low-cardinality categoricals, target/embedding encoding for high-cardinality ones.
• Scaling: standardization or normalization matters for distance-based and gradient-based models (KNN, SVMs, neural nets), largely irrelevant for tree-based models.
• Selection methods: filter methods (correlation, mutual information), wrapper methods (recursive feature elimination), and embedded methods (L1 regularization, tree feature importances).
• More/better features usually beats a fancier model — feature quality is often the highest-leverage lever in a real project.