This article needs to be updated. Please help update this article to reflect recent events or newly available information. (February 2024)

Feature engineering or feature extraction or feature discovery is the process of extracting features (characteristics, properties, attributes) from raw data to support training a downstream statistical model.[1]

Other examples of features in physics include the construction of dimensionless numbers such as the Reynolds number in fluid dynamics, the Nusselt number in heat transfer, the Archimedes number in sedimentation, and construction of first approximations of the solution such as analytical strength of materials solutions in mechanics.[2]

Predictive modelling

Feature engineering in machine learning and statistical modeling involves selecting, creating, transforming, and extracting data features. Key components include feature creation from existing data, transforming and imputing missing or invalid features, reducing data dimensionality through methods like Principal Components Analysis (PCA), Independent Component Analysis (ICA), and Linear Discriminant Analysis (LDA), and selecting the most relevant features for model training based on importance scores and correlation matrices.[3]

Features vary in significance.[4] Even relatively insignificant features may contribute to a model. Feature selection can reduce the number of features to prevent a model from becoming too specific to the training data set (overfitting).[5]

Feature explosion occurs when the number of identified features is too large for effective model estimation or optimization. Common causes include:

Feature explosion can be limited via techniques such as: regularization, kernel methods, and feature selection.[6]

Automation

Automation of feature engineering is a research topic that dates back to the 1990s.[7] Machine learning software that incorporates automated feature engineering has been commercially available since 2016.[8] Related academic literature can be roughly separated into two types:

Multi-relational decision tree learning (MRDTL)

This section's factual accuracy is disputed. Relevant discussion may be found on the talk page. Please help to ensure that disputed statements are reliably sourced. (March 2024) (Learn how and when to remove this template message)

MRDTL generates features in the form of SQL queries by successively adding clauses to the queries.[citation needed] For instance, the algorithm might start out with

SELECT COUNT(*) FROM ATOM t1 LEFT JOIN MOLECULE t2 ON t1.mol_id = t2.mol_id GROUP BY t1.mol_id

The query can then successively be refined by adding conditions, such as "WHERE t1.charge <= -0.392".[citation needed]

However, most MRDTL studies base implementations on relational databases, which results in many redundant operations. These redundancies can be reduced by using techniques such as tuple id propagation.[9][10] Efficiency can be increased by using incremental updates, which eliminates redundancies.[11][promotional source?]

Open-source implementations

There are a number of open-source libraries and tools that automate feature engineering on relational data and time series:

Deep feature synthesis

The deep feature synthesis (DFS) algorithm beat 615 of 906 human teams in a competition.[28][29]

Feature stores

The Feature Store is where the features are stored and organized for the explicit purpose of being used to either train models (by data scientists) or make predictions (by applications that have a trained model). It is a central location where you can either create or update groups of features created from multiple different data sources, or create and update new datasets from those feature groups for training models or for use in applications that do not want to compute the features but just retrieve them when it needs them to make predictions.[30]

A feature store includes the ability to store code used to generate features, apply the code to raw data, and serve those features to models upon request. Useful capabilities include feature versioning and policies governing the circumstances under which features can be used.[31]

Feature stores can be standalone software tools or built into machine learning platforms.

Alternatives

Feature engineering can be a time-consuming and error-prone process, as it requires domain expertise and often involves trial and error.[32][33] Deep learning algorithms may be used to process a large raw dataset without having to resort to feature engineering.[34] However, it's important to note that deep learning algorithms still require careful preprocessing and cleaning of the input data.[35] In addition, choosing the right architecture, hyperparameters, and optimization algorithm for a deep neural network can be a challenging and iterative process.[36]

See also

References

  1. ^ Hastie, Trevor; Tibshirani, Robert; Friedman, Jerome H. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer. ISBN 978-0-387-84884-6.
  2. ^ MacQueron C (2021). SOLID-LIQUID MIXING IN STIRRED TANKS : Modeling, Validation, Design Optimization and Suspension Quality Prediction (Report). doi:10.13140/RG.2.2.11074.84164/1.
  3. ^ "Feature engineering - Machine Learning Lens". docs.aws.amazon.com. Retrieved 2024-03-01.
  4. ^ "Feature Engineering" (PDF). 2010-04-22. Retrieved 12 November 2015.
  5. ^ "Feature engineering and selection" (PDF). Alexandre Bouchard-Côté. October 1, 2009. Retrieved 12 November 2015.
  6. ^ "Feature engineering in Machine Learning" (PDF). Zdenek Zabokrtsky. Archived from the original (PDF) on 4 March 2016. Retrieved 12 November 2015.
  7. ^ Knobbe AJ, Siebes A, Van Der Wallen D (1999). "Multi-relational Decision Tree Induction" (PDF). Principles of Data Mining and Knowledge Discovery. Lecture Notes in Computer Science. Vol. 1704. pp. 378–383. doi:10.1007/978-3-540-48247-5_46. ISBN 978-3-540-66490-1.
  8. ^ "Its all about the features". Reality AI Blog. September 2017.
  9. ^ Yin X, Han J, Yang J, Yu PS (2004). "CrossMine: Efficient classification across multiple database relations". Proceedings. 20th International Conference on Data Engineering. pp. 399–410. doi:10.1109/ICDE.2004.1320014. ISBN 0-7695-2065-0. S2CID 1183403.
  10. ^ Frank R, Moser F, Ester M (2007). "A Method for Multi-relational Classification Using Single and Multi-feature Aggregation Functions". Knowledge Discovery in Databases: PKDD 2007. Lecture Notes in Computer Science. Vol. 4702. pp. 430–437. doi:10.1007/978-3-540-74976-9_43. ISBN 978-3-540-74975-2.
  11. ^ "How automated feature engineering works - The most efficient feature engineering solution for relational data and time series". Retrieved 2019-11-21.
  12. ^ "What is Featuretools?". Retrieved September 7, 2022.
  13. ^ "Featuretools - An open source python framework for automated feature engineering". Retrieved September 7, 2022.
  14. ^ "github: alteryx/featuretools". GitHub. Retrieved September 7, 2022.
  15. ^ Thanh Lam, Hoang; Thiebaut, Johann-Michael; Sinn, Mathieu; Chen, Bei; Mai, Tiep; Alkan, Oznur (2017-06-01). "One button machine for automating feature engineering in relational databases". arXiv:1706.00327 [cs.DB].
  16. ^ Thanh Lam, Hoang; Thiebaut, Johann-Michael; Sinn, Mathieu; Chen, Bei; Mai, Tiep; Alkan, Oznur (2017-06-01). "One button machine for automating feature engineering in relational databases". arXiv:1706.00327 [cs.DB].
  17. ^ "getML documentation". Retrieved September 7, 2022.
  18. ^ "github: getml/getml-community". GitHub. Retrieved September 7, 2022.
  19. ^ "github: getml/getml-community". GitHub. Retrieved September 7, 2022.
  20. ^ "github: getml/getml-community". GitHub. Retrieved September 7, 2022.
  21. ^ "tsfresh documentation". Retrieved September 7, 2022.
  22. ^ "Time Series FeatuRe Extraction on basis of Scalable Hypothesis tests (tsfresh – A Python package)". Retrieved September 7, 2022.
  23. ^ "predict-idlab/tsflex". GitHub. Retrieved September 7, 2022.
  24. ^ Van Der Donckt, Jonas; Van Der Donckt, Jeroen; Deprost, Emiel; Van Hoecke, Sofie (2022). "tsflex: Flexible time series processing & feature extraction". SoftwareX. 17: 100971. arXiv:2111.12429. Bibcode:2022SoftX..1700971V. doi:10.1016/j.softx.2021.100971. S2CID 244527198. Retrieved September 7, 2022.
  25. ^ "seglearn user guide". Retrieved September 7, 2022.
  26. ^ "Welcome to TSFEL documentation!". Retrieved September 7, 2022.
  27. ^ "github: facebookresearch/Kats". GitHub. Retrieved September 7, 2022.
  28. ^ "Automating big-data analysis". 16 October 2015.
  29. ^ Kanter, James Max; Veeramachaneni, Kalyan (2015). "Deep feature synthesis: Towards automating data science endeavors". 2015 IEEE International Conference on Data Science and Advanced Analytics (DSAA). pp. 1–10. doi:10.1109/DSAA.2015.7344858. ISBN 978-1-4673-8272-4. S2CID 206610380.
  30. ^ "What is a feature store". Retrieved 2022-04-19.
  31. ^ "An Introduction to Feature Stores". Retrieved 2021-04-15.
  32. ^ "Feature Engineering in Machine Learning". Engineering Education (EngEd) Program | Section. Retrieved 2023-03-21.
  33. ^ explorium_admin (2021-10-25). "5 Reasons Why Feature Engineering is Challenging". Explorium. Retrieved 2023-03-21.
  34. ^ Spiegelhalter, D. J. (2019). The art of statistics : learning from data. [London] UK. ISBN 978-0-241-39863-0. OCLC 1064776283.((cite book)): CS1 maint: location missing publisher (link)
  35. ^ Sarker IH (November 2021). "Deep Learning: A Comprehensive Overview on Techniques, Taxonomy, Applications and Research Directions". SN Computer Science. 2 (6): 420. doi:10.1007/s42979-021-00815-1. PMC 8372231. PMID 34426802.
  36. ^ Bengio, Yoshua (2012), "Practical Recommendations for Gradient-Based Training of Deep Architectures", Neural Networks: Tricks of the Trade, Lecture Notes in Computer Science, vol. 7700, Berlin, Heidelberg: Springer Berlin Heidelberg, pp. 437–478, arXiv:1206.5533, doi:10.1007/978-3-642-35289-8_26, ISBN 978-3-642-35288-1, S2CID 10808461, retrieved 2023-03-21

Further reading

  • Boehmke B, Greenwell B (2019). "Feature & Target Engineering". Hands-On Machine Learning with R. Chapman & Hall. pp. 41–75. ISBN 978-1-138-49568-5.
  • Zheng A, Casari A (2018). Feature Engineering for Machine Learning: Principles and Techniques for Data Scientists. O'Reilly. ISBN 978-1-4919-5324-2.
  • Zumel N, Mount (2020). "Data Engineering and Data Shaping". Practical Data Science with R (2nd ed.). Manning. pp. 113–160. ISBN 978-1-61729-587-4.