Topic Modeling

How Topic Modeling Works

[Corpus of Documents]
        |
        | (Text Pre-processing: tokenization, stop-word removal, stemming)
        v
[Document-Term Matrix]
        |
        | (Algorithm, e.g., LDA)
        |--> [Topic 1: word_A, word_B, ...]
        |--> [Topic 2: word_C, word_D, ...]
        |--> [Topic K: word_X, word_Y, ...]
        v
[Document-Topic Distribution]
(e.g., Doc1: 70% Topic 1, 30% Topic 2)

Core Formulas and Applications

Example 1: Latent Dirichlet Allocation (LDA)

LDA is a generative probabilistic model that assumes documents are a mixture of topics and topics are a mixture of words. The joint distribution is used to infer the hidden topic structure from the observed words.

p(W, Z, θ, φ | α, β) = Π(k=1 to K) p(φ_k | β) * Π(d=1 to M) p(θ_d | α) * Π(n=1 to N_d) p(z_{d,n} | θ_d) * p(w_{d,n} | φ_{z_{d,n}})

Example 2: Probabilistic Latent Semantic Analysis (pLSA)

pLSA models the probability of a word appearing in a document as a mixture of topic-specific distributions. It is used for discovering latent topics in document collections and is a precursor to LDA.

P(d, w) = P(d) * Σ(z in Z) P(w | z) * P(z | d)

Example 3: Non-Negative Matrix Factorization (NMF)

NMF is a matrix factorization technique that decomposes the document-term matrix (V) into two non-negative matrices: one representing document-topic relationships (W) and another for topic-word relationships (H). It’s used for dimensionality reduction and topic extraction.

V ≈ W * H

Practical Use Cases for Businesses Using Topic Modeling

• Customer Feedback Analysis. Automatically sift through thousands of customer reviews, survey responses, or support tickets to identify recurring themes like “product defects,” “shipping delays,” or “positive user experience,” allowing businesses to prioritize improvements and address concerns at scale.
• Content Recommendation and Personalization. Analyze user reading habits or content libraries to discover topics of interest. This enables personalized recommendations for articles, products, or media, improving user engagement and retention on platforms like news sites or e-commerce stores.
• Market Trend Detection. Monitor social media, news articles, and industry reports to detect emerging trends and shifts in consumer conversation. This helps businesses stay ahead of the competition by identifying new market needs or changing sentiment.
• Intelligent Document Management. Automatically categorize and tag large volumes of internal documents, such as contracts, reports, and emails. This improves information retrieval, ensuring employees can find relevant information quickly and efficiently.

Example 1: Customer Support Ticket Routing

Input: [List of Unassigned Support Tickets]
Process:
1. Pre-process text data (clean, tokenize).
2. Apply trained LDA model to each ticket.
3. Get Topic Distribution for each ticket (e.g., Ticket #123: {Topic_A: 0.85, Topic_B: 0.15}).
4. Route ticket based on highest probability topic.
   IF Topic_A == "Billing_Inquiries" -> Route to Finance Dept.
   IF Topic_B == "Technical_Issues" -> Route to IT Support.
Business Use Case: A software company can automatically route incoming support tickets to the correct department (e.g., Billing, Technical Support, Sales) without manual sorting, reducing response times and improving customer satisfaction.

Example 2: Analyzing Product Reviews

Input: [Dataset of 10,000 product reviews]
Process:
1. Run NMF to decompose the review corpus into 5 topics.
2. Analyze Topic-Word Matrix (H) to interpret topics.
   - Topic 1: 'battery', 'life', 'charge', 'short'
   - Topic 2: 'screen', 'display', 'bright', 'pixel'
3. Analyze Document-Topic Matrix (W) to score reviews against topics.
Business Use Case: An electronics retailer can analyze thousands of reviews for a new smartphone to quickly identify that the main points of discussion are "short battery life" and "screen quality," guiding future product development and marketing messages.

🐍 Python Code Examples

This example demonstrates how to perform topic modeling using Latent Dirichlet Allocation (LDA) with the scikit-learn library. It takes a small corpus of documents, vectorizes it using a CountVectorizer, and then fits an LDA model to discover two topics.

from sklearn.feature_extraction.text import CountVectorizer
from sklearn.decomposition import LatentDirichletAllocation

# Sample documents
documents = [
    "The stock market is performing well with new technology stocks.",
    "Investors are looking into tech stocks and financial markets.",
    "The new software update improves the performance and security of the system.",
    "Data security and software engineering are key parts of modern technology.",
    "Financial planning and market analysis are crucial for investment."
]

# Create a document-term matrix
vectorizer = CountVectorizer(stop_words='english')
X = vectorizer.fit_transform(documents)

# Apply Latent Dirichlet Allocation
lda = LatentDirichletAllocation(n_components=2, random_state=42)
lda.fit(X)

# Display the topics
feature_names = vectorizer.get_feature_names_out()
for topic_idx, topic in enumerate(lda.components_):
    print(f"Topic #{topic_idx + 1}:")
    print(" ".join([feature_names[i] for i in topic.argsort()[:-6:-1]]))

This code snippet showcases topic modeling using Non-Negative Matrix Factorization (NMF). NMF is another technique that can be used for topic discovery. The process is similar: vectorize the text and then apply the NMF model to find the topics.

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.decomposition import NMF

# Sample documents (can reuse from the previous example)
documents = [
    "The stock market is performing well with new technology stocks.",
    "Investors are looking into tech stocks and financial markets.",
    "The new software update improves the performance and security of the system.",
    "Data security and software engineering are key parts of modern technology.",
    "Financial planning and market analysis are crucial for investment."
]

# Create a TF-IDF matrix
tfidf_vectorizer = TfidfVectorizer(stop_words='english')
X_tfidf = tfidf_vectorizer.fit_transform(documents)

# Apply Non-Negative Matrix Factorization
nmf = NMF(n_components=2, random_state=42)
nmf.fit(X_tfidf)

# Display the topics
feature_names = tfidf_vectorizer.get_feature_names_out()
for topic_idx, topic in enumerate(nmf.components_):
    print(f"Topic #{topic_idx + 1}:")
    print(" ".join([feature_names[i] for i in topic.argsort()[:-6:-1]]))

Types of Topic Modeling

• Latent Dirichlet Allocation (LDA). A probabilistic generative model assuming documents are mixtures of topics, and topics are mixtures of words. It is one of the most popular and widely used topic modeling algorithms for discovering underlying thematic structures in large text corpora.
• Probabilistic Latent Semantic Analysis (pLSA). A statistical technique that models topics as a latent variable to find co-occurrence patterns. pLSA models each document as a mixture of topics but has limitations in generalizing to new, unseen documents, which led to the development of LDA.
• Non-Negative Matrix Factorization (NMF). An unsupervised learning algorithm that factorizes the high-dimensional document-term matrix into two lower-dimensional, non-negative matrices. These matrices represent document-topic and topic-word distributions, offering an alternative approach to probabilistic methods for topic extraction.
• Latent Semantic Analysis (LSA). An earlier technique that uses linear algebra, specifically Singular Value Decomposition (SVD), to reduce the dimensionality of the document-term matrix. It identifies latent relationships between terms and documents to uncover topics but lacks a clear probabilistic interpretation.
• Correlated Topic Models (CTM). An extension of LDA that models correlations between topics, addressing a limitation of LDA which assumes topics are independent. This is useful for corpora where themes are naturally interrelated, providing a more realistic representation of the topic structure.