Part of a series on
Machine learning and data mining
Paradigms Supervised learning Unsupervised learning Online learning Batch learning Meta-learning Semi-supervised learning Self-supervised learning Reinforcement learning Rule-based learning Quantum machine learning
Problems Classification Generative model Regression Clustering dimension reduction density estimation Anomaly detection Data Cleaning AutoML Association rules Semantic analysis Structured prediction Feature engineering Feature learning Learning to rank Grammar induction Ontology learning Multimodal learning
Supervised learning (classification • regression) Apprenticeship learning Decision trees Ensembles Bagging Boosting Random forest k-NN Linear regression Naive Bayes Artificial neural networks Logistic regression Perceptron Relevance vector machine (RVM) Support vector machine (SVM)
Clustering BIRCH CURE Hierarchical k-means Fuzzy Expectation–maximization (EM) DBSCAN OPTICS Mean shift
Dimensionality reduction Factor analysis CCA ICA LDA NMF PCA PGD t-SNE SDL
Structured prediction Graphical models Bayes net Conditional random field Hidden Markov
Anomaly detection RANSAC k-NN Local outlier factor Isolation forest
Artificial neural network Autoencoder Cognitive computing Deep learning DeepDream Multilayer perceptron RNN LSTM GRU ESN reservoir computing Restricted Boltzmann machine GAN SOM Convolutional neural network U-Net Transformer Vision Spiking neural network Memtransistor Electrochemical RAM (ECRAM)
Reinforcement learning Q-learning SARSA Temporal difference (TD) Multi-agent Self-play
Learning with humans Active learning Crowdsourcing Human-in-the-loop
Model diagnostics Learning curve
Theory Kernel machines Bias–variance tradeoff Computational learning theory Empirical risk minimization Occam learning PAC learning Statistical learning VC theory
Machine-learning venues NeurIPS ICML ICLR ML JMLR
Related articles Glossary of artificial intelligence List of datasets for machine-learning research Outline of machine learning
v t e

Feature engineering or feature extraction or feature discovery is the process of using domain knowledge to extract features (characteristics, properties, attributes) from raw data.^[1] The motivation is to use these extra features to improve the quality of results from a machine learning process, compared with supplying only the raw data to the machine learning process.

Process

The feature engineering process is:^[2]

• Brainstorming or testing features^[3]
• Deciding what features to create
• Creating features
• Testing the impact of the identified features on the task
• Improving your features if needed
• Repeat

Typical engineered features

The following list^[4] provides some typical ways to engineer useful features:

• Numerical transformations (like taking fractions or scaling)
• Category encoder like one-hot or target encoder (for categorical data)^[5]
• Clustering
• Group aggregated values
• Principal component analysis (for numerical data)
• Feature construction : building new "physical", knowledge-based parameters relevant to the problem.^[6] For example, in physics, construction of dimensionless numbers such as Reynolds number in fluid dynamics, Nusselt number in heat transfer, Archimedes number in sedimentation, construction of first approximations of the solution such as analytical strength of materials solutions in mechanics, etc.^[7]

Relevance

Features vary in significance.^[8] Even relatively insignificant features may contribute to a model. Feature selection can reduce the number of features to prevent a model from becoming too specific to the training data set (overfitting).^[9]

Explosion

Feature explosion occurs when the number of identified features grows inappropriately. Common causes include:

• Feature templates - implementing feature templates instead of coding new features
• Feature combinations - combinations that cannot be represented by a linear system

Feature explosion can be limited via techniques such as: regularization, kernel methods, and feature selection.^[10]

Automation

Automation of feature engineering is a research topic that dates back to the 1990s.^[11] Machine learning software that incorporates automated feature engineering has been commercially available since 2016.^[12] Related academic literature can be roughly separated into two types:

• Multi-relational decision tree learning (MRDTL) uses a supervised algorithm that is similar to a decision tree.
• Deep Feature Synthesis uses simpler methods.^{[citation needed]}

Multi-relational decision tree learning (MRDTL)

MRDTL generates features in the form of SQL queries by successively adding clauses to the queries.^{[citation needed]} For instance, the algorithm might start out with

SELECT COUNT(*) FROM ATOM t1 LEFT JOIN MOLECULE t2 ON t1.mol_id = t2.mol_id GROUP BY t1.mol_id

The query can then successively be refined by adding conditions, such as "WHERE t1.charge <= -0.392".^{[citation needed]}

However, most MRDTL studies base implementations on relational databases, which results in many redundant operations. These redundancies can be reduced by using techniques such as tuple id propagation.^[13][14] Efficiency can be increased by using incremental updates, which eliminates redundancies.^{[15][promotional source?]}

Open-source implementations

There are a number of open-source libraries and tools that automate feature engineering on relational data and time series:

• featuretools is a Python library for transforming time series and relational data into feature matrices for machine learning.^[16][17][18]
• OneBM or One-Button Machine combines feature transformations and feature selection on relational data with feature selection techniques.^[19]

  [OneBM] helps data scientists reduce data exploration time allowing them to try and error many ideas in short time. On the other hand, it enables non-experts, who are not familiar with data science, to quickly extract value from their data with a little effort, time, and cost.^[20]

• getML community is an open source tool for automated feature engineering on time series and relational data.^[21][22] It is implemented in C/C++ with a Python interface.^[23] It has been shown to be at least 60 times faster than tsflex, tsfresh, tsfel, featuretools or kats.^[24]
• tsfresh is a Python library for feature extraction on time series data.^[25] It evaluates the quality of the features using hypothesis testing.^[26]
• tsflex is an open source Python library for extracting features from time series data.^[27] Despite being 100% written in Python, it has been shown to be faster and more memory efficient than tsfresh, seglearn or tsfel.^[28]
• seglearn is an extension for multivariate, sequential time series data to the scikit-learn Python library.^[29]
• tsfel is a Python package for feature extraction on time series data.^[30]
• kats is a Python toolkit for analyzing time series data.^[31]

Deep feature synthesis

The deep feature synthesis (DFS) algorithm beat 615 of 906 human teams in a competition.^[32][33]

Feature stores

The Feature Store is where the features are stored and organized for the explicit purpose of being used to either train models (by data scientists) or make predictions (by applications that have a trained model). It is a central location where you can either create or update groups of features created from multiple different data sources, or create and update new datasets from those feature groups for training models or for use in applications that do not want to compute the features but just retrieve them when it needs them to make predictions.^[34]

A feature store includes the ability to store code used to generate features, apply the code to raw data, and serve those features to models upon request. Useful capabilities include feature versioning and policies governing the circumstances under which features can be used.^[35]

Feature stores can be standalone software tools or built into machine learning platforms.

Alternatives

Feature engineering can be a time-consuming and error-prone process, as it requires domain expertise and often involves trial and error.^[36][37] Deep learning algorithms may be used to process a large raw dataset without having to resort to feature engineering.^[38] However, it's important to note that deep learning algorithms still require careful preprocessing and cleaning of the input data.^[39] In addition, choosing the right architecture, hyperparameters, and optimization algorithm for a deep neural network can be a challenging and iterative process.^[40]