You do not have permission to edit this page, for the following reasons:

This IP address has been blocked from editing Wikipedia.
This does not affect your ability to read Wikipedia pages.
Most people who see this message have done nothing wrong. Some kinds of blocks restrict editing from specific service providers or telecom companies in response to recent abuse or vandalism, and can sometimes affect other users who are unrelated to that abuse. Review the information below for assistance if you do not believe that you have done anything wrong.

The IP address or range 3.239.0.0/16 has been blocked (disabled) by ‪Blablubbs‬ for the following reason(s):

The IP address that you are currently using has been blocked because it is believed to be a web host provider or colocation provider. To prevent abuse, web hosts and colocation providers may be blocked from editing Wikipedia.
You will not be able to edit Wikipedia using a web host or colocation provider because it hides your IP address, much like a proxy or VPN.
We recommend that you attempt to use another connection to edit. For example, if you use a proxy or VPN to connect to the internet, turn it off when editing Wikipedia. If you edit using a mobile connection, try using a Wi-Fi connection, and vice versa. If you are using a corporate internet connection, switch to a different Wi-Fi network. If you have a Wikipedia account, please log in.
If you do not have any other way to edit Wikipedia, you will need to request an IP block exemption.

How to appeal if you are confident that your connection does not use a colocation provider's IP address:
If you are confident that you are not using a web host, you may appeal this block by adding the following text on your talk page: ((unblock|reason=Caught by a colocation web host block but this host or IP is not a web host. My IP address is _______. Place any further information here. ~~~~)). You must fill in the blank with your IP address for this block to be investigated. Your IP address can be determined here. Alternatively, if you wish to keep your IP address private you can use the unblock ticket request system. There are several reasons you might be editing using the IP address of a web host or colocation provider (such as if you are using VPN software or a business network); please use this method of appeal only if you think your IP address is in fact not a web host or colocation provider.

Administrators: The IP block exemption user right should only be applied to allow users to edit using web host in exceptional circumstances, and requests should usually be directed to the functionaries team via email. If you intend to give the IPBE user right, a CheckUser needs to take a look at the account. This can be requested most easily at SPI Quick Checkuser Requests. Unblocking an IP or IP range with this template is highly discouraged without at least contacting the blocking administrator.

This block will expire on 07:38, 2 November 2024.

Even when blocked, you will usually still be able to edit your user talk page, as well as email administrators and other editors.

For information on how to proceed, please read the FAQ for blocked users and the guideline on block appeals. The guide to appealing blocks may also be helpful.

Other useful links: Blocking policy · Help:I have been blocked
Your IP address is in a range that has been blocked on all Wikimedia Foundation wikis. The block was made by ‪Elton‬. The reason given is Open proxy/Webhost: See the help page if you are affected: .
- Start of block: 18:08, 17 February 2024
- Expiry of block: 18:08, 17 February 2029
Your current IP address is 3.239.171.112. The blocked range is 3.239.0.0/16. Please include all above details in any queries you make. If you believe you were blocked by mistake, you can find additional information and instructions in the No open proxies global policy. Otherwise, to discuss the block please post a request for review on Meta-Wiki. You could also send an email to the stewards VRT queue at stewards@wikimedia.org including all above details.

You can view and copy the source of this page:

((More citations needed|date=January 2021))
A '''document-term matrix''' is a mathematical [[Matrix (mathematics)|matrix]] that describes the frequency of terms that occur in each document in a collection. In a document-term matrix, rows correspond to documents in the collection and columns correspond to terms. This matrix is a specific instance of a '''document-feature matrix''' where "features" may refer to other properties of a document besides terms.<ref>((Cite web|title=Document-feature matrix :: Tutorials for quanteda|url=https://tutorials.quanteda.io/basic-operations/dfm/|access-date=2021-01-02|website=tutorials.quanteda.io))</ref> It is also common to encounter the transpose, or '''term-document matrix''' where documents are the columns and terms are the rows. They are useful in the field of [[natural language processing]] and [[computational text analysis]].<ref>((Cite web|title=15 Ways to Create a Document-Term Matrix in R|url=https://www.dustinstoltz.com/blog/2020/12/1/creating-document-term-matrix-comparison-in-r|access-date=2021-01-02|website=Dustin S. Stoltz|language=en-US))</ref>

While the value of the cells is commonly the raw count of a given term, there are various schemes for weighting the raw counts such as, row normalizing (i.e. relative frequency/proportions) and [[tf-idf]].

Terms are commonly single words separated by whitespace or punctuation on either side (a.k.a. unigrams). In such a case, this is also referred to as "bag of words" representation because the counts of individual words is retained, but not the order of the words in the document.

==General concept==
When creating a data-set of [[term (language)|terms]] that appear in a corpus of [[document]]s, the document-term matrix contains rows corresponding to the documents and columns corresponding to the terms. Each ''ij'' cell, then, is the number of times word ''j'' occurs in document ''i''. As such, each row is a vector of term counts that represents the content of the document corresponding to that row. For instance if one has the following two (short) documents:
*D1 = "I like databases"
*D2 = "I dislike databases",
then the document-term matrix would be:
{| border="1" cellspacing="0"
! ||I||like||dislike||databases
|- align=center
|'''D1'''||1||1||0||1
|- align=center
|'''D2'''||1||0||1||1
|}
which shows which documents contain which terms and how many times they appear. Note that, unlike representing a document as just a token-count list, the document-term matrix includes all terms in the corpus (i.e. the corpus vocabulary), which is why there are zero-counts for terms in the corpus which do not also occur in a specific document. For this reason, document-term matrices are usually stored in a sparse matrix format.

As a result of the power-law distribution of tokens in nearly every corpus (see [[Zipf's law]]), it is common to weight the counts. This can be as simple as dividing counts by the total number of tokens in a document (called relative frequency or proportions), dividing by the maximum frequency in each document (called prop max), or taking the log of frequencies (called log count). If one desires to weight the words most unique to an individual document as compared to the corpus as a whole, it is common to use [[tf-idf]], which divides the term frequency by the term's document frequency.

== History of the concept ==
The document-term matrix emerged in the earliest years of the computerization of text. The increasing capacity for storing documents created the problem of retrieving a given document in an efficient manner. While previously the work of classifying and indexing was accomplished by hand, researchers explored the possibility of doing this automatically using word frequency information.

One of the first published document-term matrices was in [[Harold Borko]]'s 1962 article "The construction of an empirically based mathematically derived classification system" (page 282, see also his 1965 article<ref>((Cite journal|last=Borko|first=Harold|date=1965|title=A Factor Analytically Derived Classification System for Psychological Reports|url=http://dx.doi.org/10.2466/pms.1965.20.2.393|journal=Perceptual and Motor Skills|volume=20|issue=2|pages=393–406|doi=10.2466/pms.1965.20.2.393|pmid=14279310|s2cid=34230652|issn=0031-5125))</ref>). Borko references two computer programs, "FEAT" which stood for "Frequency of Every Allowable Term," written by John C. Olney of the System Development Corporation and the Descriptor Word Index Program, written by [[Eileen Stone]] also of the System Development Corporation: <blockquote>Having selected the documents which were to make up the experimental library, the next step consisted of keypunching the entire body of text preparatory to computer processing.  The program used for this analysis was FEAT (Frequency of Every Allowable Term).  it was written by John C. Olney of the System Development Corporation and is designed to perform frequency and summary counts of individual words and of word pairs.  The  output of this program is an alphabetical listing, by frequency of occurrence, of all word types which appeared in the text.  Certain function words such as and, the,  at, a, etc., were placed in a "forbidden word list" table, and the frequency of these words was recorded  in a separate listing... A special computer program, called the Descriptor Word Index Program, was written to provide this information and to prepare a document-term matrix in a form suitable for in-put to the Factor Analysis Program.  The Descriptor Word Index program was prepared by Eileen Stone of the System Development Corporation.<ref>((Cite book|last=Borko|first=Harold|title=Proceedings of the May 1-3, 1962, spring joint computer conference on - AIEE-IRE '62 (Spring) |chapter=The construction of an empirically based mathematically derived classification system |date=1962|series=AIEE-IRE '62 (Spring)|pages=279–289|location=New York, New York, USA|publisher=ACM Press|doi=10.1145/1460833.1460865|isbn=9781450378758|s2cid=6483337|doi-access=free))</ref></blockquote>Shortly thereafter, [[Gerard Salton]] published "Some hierarchical models for automatic document retrieval" in 1963 which also included a visual depiction of a document-term matrix.<ref name=":0">((Cite journal|last=Salton|first=Gerard|date=July 1963|title=Some hierarchical models for automatic document retrieval|url=http://dx.doi.org/10.1002/asi.5090140307|journal=American Documentation|volume=14|issue=3|pages=213–222|doi=10.1002/asi.5090140307|issn=0096-946X))</ref> Salton was at Harvard University at the time and his work was supported by the Air Force Cambridge Research Laboratories and Sylvania Electric Products, Inc. In this paper, Salton introduces the document-term matrix by comparison to a kind of term-context matrix used to measure similarities between words:<blockquote>If it is desired to generate document associations or document clusters instead of word associations, the same procedures can be used with slight modifications. Instead of starting with a word-sentence matrix ''C'',... it is now convenient to construct a word-document matrix ''F,'' listing frequency of occurrence of word W<sub>i</sub> in Document D<sub>j</sub>... Document similarities can now be computed as before by comparing pairs of rows and by obtaining similarity coefficients based on the frequency of co-occurrences of the content words included in the given document. This procedure produces a document-document similarity matrix which can in turn be used for the generation of document clusters...<ref name=":0" /></blockquote>In addition to Borko and Salton, in 1964, F.W. Lancaster published a comprehensive review of automated indexing and retrieval. While the work was published while he worked at the Herner and Company in Washington D.C., the paper was written while he was "employed in research work at Aslib, on the Aslib Cranfield Project."<ref>((Cite journal|last=LANCASTER|first=F.W.|date=1964-01-01|title=MECHANIZED DOCUMENT CONTROL: A Review of Some Recent Research|url=https://doi.org/10.1108/eb049960|journal=ASLIB Proceedings|volume=16|issue=4|pages=132–152|doi=10.1108/eb049960|issn=0001-253X))</ref> Lancaster credits Borko with the document-term matrix:<blockquote>Harold  Borko,  of  the  System  Development  Corporation,  has  carried  this  operation  a little  further.  A  significant  group  of clue words  is chosen  from  the vocabulary of an experimental collection.  These are arranged in a document/term matrix to show the frequency  of occurrence of each term in each document....  A correlation  coefficient  for  each word  pair  is then  computed,  based  on  their  co-occurrence  in  the  document  set.   The  resulting  term/term  matrix... is then  factor  analysed  and  a series  of  factors  are  isolated.  These factors,  when  interpreted  and  named  on the  basis  of the  terms with  high  loadings which  appear in  each  of the factors, become the classes  of an  empirical  classification.  The terms with high loadings in each factor are the clue words  or  predictors  of  the  categories.  </blockquote>

==Choice of terms==
A point of view on the matrix is that each row represents a document. In the [[Vector space model|vectorial semantic model]], which is normally the one used to compute a document-term matrix, the goal is to represent the topic of a document by the frequency of semantically significant terms. The terms are semantic units of the documents. It is often assumed, for [[Indo-European languages]], that nouns, verbs and adjectives are the more significant [[syntactic category|categories]], and that words from those categories should be kept as terms.
Adding [[collocation]] as terms improves the quality of the vectors, especially when computing similarities between documents.

==Applications==
===Improving search results===
[[Latent semantic analysis]] (LSA, performing [[singular-value decomposition]] on the document-term matrix) can improve search results by [[word-sense disambiguation|disambiguating]] [[polysemy|polysemous words]] and searching for [[synonym]]s of the query. However, searching in the high-dimensional continuous space is much slower than searching the standard [[trie]] data structure of search engines.

===Finding topics===
[[Multivariate analysis]] of the document-term matrix can reveal topics/themes of the corpus. Specifically, [[latent semantic analysis]] and [[data clustering]] can be used, and, more recently, [[probabilistic latent semantic analysis]] with its generalization [[Latent Dirichlet allocation]], and [[non-negative matrix factorization]], have been found to perform well for this task.

==See also==
* [[Bag of words model]]

===Implementations===

* [http://nlp.fi.muni.cz/projekty/gensim Gensim]: Open source Python framework for Vector Space modelling. Contains memory-efficient algorithms for constructing term-document matrices from text plus common transformations ([[tf-idf]], [[Latent semantic analysis|LSA]], [[Latent Dirichlet allocation|LDA]]).

== References ==
((Reflist))

((Natural Language Processing))

((DEFAULTSORT:Document-Term Matrix))
[[Category:Natural language processing]]

Pages transcluded onto the current version of this page (help):

Document-term matrix (edit)
Template:Ambox (view source) (template editor protected)
Template:Cite book (view source) (protected)
Template:Cite journal (view source) (protected)
Template:Cite web (view source) (protected)
Template:Find sources mainspace (view source) (template editor protected)
Template:Hlist/styles.css (view source) (protected)
Template:Main other (view source) (protected)
Template:More citations needed (view source) (template editor protected)
Template:Natural Language Processing (edit)
Template:Natural language processing (edit)
Template:Navbox (view source) (template editor protected)
Template:Reflist (view source) (protected)
Template:Reflist/styles.css (view source) (protected)
Module:Arguments (view source) (protected)
Module:Category handler (view source) (protected)
Module:Category handler/blacklist (view source) (protected)
Module:Category handler/config (view source) (protected)
Module:Category handler/data (view source) (protected)
Module:Category handler/shared (view source) (protected)
Module:Check for unknown parameters (view source) (protected)
Module:Citation/CS1 (view source) (protected)
Module:Citation/CS1/COinS (view source) (protected)
Module:Citation/CS1/Configuration (view source) (protected)
Module:Citation/CS1/Date validation (view source) (protected)
Module:Citation/CS1/Identifiers (view source) (protected)
Module:Citation/CS1/Utilities (view source) (protected)
Module:Citation/CS1/Whitelist (view source) (protected)
Module:Citation/CS1/styles.css (view source) (protected)
Module:Find sources (view source) (template editor protected)
Module:Find sources/config (view source) (template editor protected)
Module:Find sources/links (view source) (template editor protected)
Module:Find sources/templates/Find sources mainspace (view source) (template editor protected)
Module:Message box (view source) (protected)
Module:Message box/ambox.css (view source) (protected)
Module:Message box/configuration (view source) (protected)
Module:Namespace detect/config (view source) (protected)
Module:Namespace detect/data (view source) (protected)
Module:Navbar (view source) (protected)
Module:Navbar/configuration (view source) (protected)
Module:Navbar/styles.css (view source) (protected)
Module:Navbox (view source) (template editor protected)
Module:Navbox/configuration (view source) (template editor protected)
Module:Navbox/styles.css (view source) (template editor protected)
Module:Unsubst (view source) (protected)
Module:Yesno (view source) (protected)

Return to Document-term matrix.

Retrieved from "https://en.wikipedia.org/wiki/Document-term_matrix"