{"id":238,"date":"2025-11-30T01:39:35","date_gmt":"2025-11-30T01:39:35","guid":{"rendered":"http:\/\/ijeesoo.com\/?page_id=238"},"modified":"2025-11-30T02:19:10","modified_gmt":"2025-11-30T02:19:10","slug":"clustering-for-information-retrieval-recommender-systems","status":"publish","type":"page","link":"http:\/\/ijeesoo.com\/?page_id=238","title":{"rendered":"\u00a0Clustering for Information Retrieval, Recommender Systems"},"content":{"rendered":"\n
\n
\n
\n
\n


What is Clustering?<\/p>\n<\/div>\n<\/div>\n\n\n\n

\n

\u2022Definition:<\/strong>The fundamental task of grouping a set of objects (documents) such that objects in the same group (cluster) are more similar<\/strong>to each other than to those in other groups.<\/p>\n\n\n\n

\u2022It is the most prevalent form of unsupervised learning<\/strong>in IR.<\/p>\n\n\n\n

\u2022Goal:<\/strong>To achieve high intra-cluster similarity<\/strong>(documents inside a cluster are highly similar) and high inter-cluster dissimilarity<\/strong>(documents in different clusters are highly dissimilar).<\/p>\n\n\n\n

\u2022Contrast with Classification:<\/strong><\/p>\n\n\n\n

\u2022Classification<\/strong>is supervised: documents are assigned to predefined, known classes (labels).<\/p>\n\n\n\n

\u2022Clustering<\/strong>is unsupervised: the classes (clusters) are discovered from<\/em>the data.<\/p>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n

\n
\n
\n\n\n\n

What is Clustering?<\/h3>\n\n\n\n
    \n
  • Definition:<\/strong>The fundamental task of grouping a set of objects (documents) such that objects in the same group (cluster) are more similar<\/strong> to each other than to those in other groups.<\/li>\n\n\n\n
  • It is the most prevalent form of unsupervised learning<\/strong> in IR.<\/li>\n\n\n\n
  • Goal:<\/strong>To achieve high intra-cluster similarity<\/strong>(documents inside a cluster are highly similar) and high inter-cluster dissimilarity<\/strong>(documents in different clusters are highly dissimilar).<\/li>\n\n\n\n
  • Contrast with Classification:<\/strong><\/li>\n\n\n\n
  • Classification<\/strong> is supervised: documents are assigned to predefined, known classes (labels).<\/li>\n\n\n\n
  • Clustering<\/strong> is unsupervised: the classes (clusters) are discovered from<\/em> the data.<\/li>\n<\/ul>\n\n\n\n
    \n\n\n\n

    The Cluster Hypothesis<\/h3>\n\n\n\n
    \"\"<\/figure>\n\n\n\n

    <\/p>\n\n\n\n

    \n\n\n\n

    Why Cluster Documents? (I: User Interface)<\/h3>\n\n\n\n

    1. Organizing Search Results (Post-Retrieval Clustering):<\/strong><\/p>\n\n\n\n

      \n
    • Clustering is performed on the top 50\u2013100 documents retrieved by a standard ranking function.<\/li>\n\n\n\n
    • Purpose:<\/strong>To provide a summary<\/strong> or topical categorization<\/strong> of the result set.<\/li>\n\n\n\n
    • Benefit:<\/strong>Users can quickly grasp the diverse themes\/topics present and navigate to the most relevant group.<\/li>\n\n\n\n
    • Example:<\/strong>Tools like Scatter\/Gather<\/strong> or modern clustered search engines (e.g., Viv\u00edsimo) group results with auto-generated labels (e.g., \u201cAmazon River,\u201d \u201cAmazon E-Commerce,\u201d \u201cAmazon Rainforest\u201d<\/em>for the query \u201cAmazon\u201d).<\/li>\n<\/ul>\n\n\n\n
      \n\n\n\n

      Why Cluster Documents? (II: System Efficiency & Effectiveness)<\/h3>\n\n\n\n
      \"\"<\/figure>\n\n\n\n

      <\/p>\n\n\n\n

      <\/div>\n<\/div>\n\n\n\n
      \n
      \n\n\n\n

      Document Representation for Clustering<\/h3>\n\n\n\n
      \"\"<\/figure>\n\n\n\n

      <\/p>\n\n\n\n

      \n\n\n\n

      Measuring Document Similarity<\/h3>\n\n\n\n
      \"\"<\/figure>\n\n\n\n

      <\/p>\n\n\n\n

      \n\n\n\n

      Types of Clustering Structures<\/h3>\n\n\n\n

      Clustering methods can be broadly categorized based on the structure of the resulting partition:<\/p>\n\n\n\n

      1. Partitional (Flat) Clustering:<\/strong><\/p>\n\n\n\n

        \n
      • Creates a single level of K<\/strong> non-overlapping, exhaustive clusters.<\/li>\n\n\n\n
      • Every document belongs to exactly one cluster.<\/li>\n\n\n\n
      • Requires specifying the number of clusters K<\/strong>a priori<\/em>.<\/li>\n\n\n\n
      • Algorithms:K-Means<\/strong>, EM (Expectation-Maximization).<\/li>\n<\/ul>\n\n\n\n
          <\/ul>\n\n\n\n

          2. Hierarchical Clustering:<\/strong><\/p>\n\n\n\n

            \n
          • Creates a nested sequence of partitions, organized into a tree structure called a Dendrogram<\/strong>.<\/li>\n\n\n\n
          • Does not<\/em>require specifying K<\/strong>. The final partition is chosen by cutting the dendrogramat a specific level.<\/li>\n\n\n\n
          • Algorithms:Agglomerative (Bottom-Up)<\/strong>and Divisive (Top-Down)<\/strong>.<\/li>\n<\/ul>\n\n\n\n
            \n\n\n\n

            Partitional Clustering: K-Means Introduction<\/h3>\n\n\n\n
            \"\"<\/figure>\n\n\n\n

            <\/p>\n\n\n\n

            <\/div>\n\n\n\n
            \n\n\n\n

            The K-Means Algorithm Steps<\/h3>\n\n\n\n
            \"\"<\/figure>\n\n\n\n

            <\/p>\n\n\n\n

            \n\n\n\n

            Challenges in K-Means Clustering<\/h3>\n\n\n\n

            While fast and simple, K-Means has significant drawbacks, especially in the context of high-dimensional IR data:<\/p>\n\n\n\n

            1. Sensitivity to Initial Centroids:<\/strong>The final clustering result often depends heavily on the initial random selection of K<\/strong>centroids. Poor initialization can lead to a locally optimal but globally suboptimal solution (high SSE).<\/p>\n\n\n\n

              \n
            • SSE = sum of squared distances between each point and its assigned cluster centroid.<\/li>\n\n\n\n
            • High SSE \u2192 clusters are not tight, points are far from their centroids \u2192 poor clustering quality.<\/li>\n\n\n\n
            • Mitigation:<\/em>Use more sophisticated initialization methods like K-Means++<\/strong>.<\/li>\n<\/ul>\n\n\n\n

              2. Requirement to Pre-specify K:<\/strong>There is no a priori<\/em>method to determine the “correct” number of clusters, K<\/strong>.<\/p>\n\n\n\n

              \u2022Mitigation:<\/em>Use techniques like the Elbow Method<\/strong>or Silhouette Analysis<\/strong>.<\/p>\n\n\n\n

              3. Assumes Spherical Clusters:<\/strong>K-Means minimizes squared Euclidean distance, inherently assuming clusters are convex and roughly spherical (isotropicallydistributed). It struggles with irregularly shaped or intertwined clusters.<\/p>\n\n\n\n

              4. Outlier Sensitivity:<\/strong>Centroids are the mean of the data points, making them highly sensitive to outliers, which can significantly skew the cluster center.<\/p>\n\n\n\n

              <\/p>\n\n\n\n

              \n\n\n\n

              Hierarchical Clustering: Introduction<\/h3>\n\n\n\n

              \u2022Concept:<\/strong>Creates a nested sequence of clusters, resulting in a tree-like structure known as a Dendrogram<\/strong>.<\/p>\n\n\n\n

              \u2022Advantage:<\/strong>Does not require specifying the number of clusters (K<\/strong>) beforehand. The resulting clusters are viewed at various levels of granularity by “cutting” the dendrogram.<\/p>\n\n\n\n

              \u2022Two Main Approaches:<\/strong><\/p>\n\n\n\n

                \n
              1. Agglomerative (Bottom-Up):<\/strong>Starts with individual documents as clusters and merges the most similar clusters iteratively.<\/li>\n\n\n\n
              2. Divisive (Top-Down):<\/strong>Starts with all documents in one cluster and recursively splits the cluster into two until only individual documents remain.<\/li>\n<\/ol>\n\n\n\n
                \n\n\n\n

                Agglomerative Hierarchical Clustering (HAC)<\/h3>\n\n\n\n
                \"\"<\/figure>\n\n\n\n

                <\/p>\n\n\n\n

                <\/div>\n\n\n\n
                \n\n\n\n

                Defining Cluster Proximity: Linkage Criteria<\/h3>\n\n\n\n
                \"\"<\/figure>\n\n\n\n

                <\/p>\n\n\n\n

                \n\n\n\n

                The Dendrogram<\/h3>\n\n\n\n

                \u2022Visualization Tool:<\/strong>The Dendrogramvisually represents the entire history of merges (in HAC) or splits (in Divisive) that created the hierarchy.<\/p>\n\n\n\n

                \u2022Interpretation:<\/strong><\/p>\n\n\n\n

                \u2022X-axis:<\/strong>Represents the documents.<\/p>\n\n\n\n

                \u2022Y-axis:<\/strong>Represents the distance\/dissimilarity<\/strong>(or time\/iteration step) at which the merges occurred.<\/p>\n\n\n\n

                \u2022Determining K:<\/strong>To obtain K<\/strong>flat clusters, one “cuts” the dendrogramhorizontally at a height that intersects exactly K<\/strong>vertical lines. Clusters joined at lower heights are more similar.<\/p>\n\n\n\n

                \n\n\n\n

                Divisive Hierarchical Clustering (DHC)<\/h3>\n\n\n\n
                \"\"<\/figure>\n\n\n\n
                \n\n\n\n

                Density-Based Clustering (DBSCAN)<\/h3>\n\n\n\n
                \"\"<\/figure>\n\n\n\n

                <\/p>\n\n\n\n

                \n\n\n\n
                <\/div>\n\n\n\n


                Model-Based Clustering (GMM)<\/h3>\n\n\n\n
                \"\"<\/figure>\n\n\n\n

                <\/p>\n\n\n\n

                \n\n\n\n

                Clustering of Terms<\/h3>\n\n\n\n
                \"\"<\/figure>\n\n\n\n

                <\/p>\n\n\n\n

                \n\n\n\n

                Dimensionality Reduction for Clustering<\/h3>\n\n\n\n

                \u2022Document vectors are extremely high-dimensional (vocabulary size M <\/strong>is typically tens of thousands). This presents a challenge:<\/p>\n\n\n\n

                \u2022The Curse of Dimensionality:<\/strong>In high dimensions, all distances tend to become equal, making the concept of “nearness” and “farness” less meaningful for clustering.<\/p>\n\n\n\n

                \u2022Computational Cost:<\/strong>Clustering algorithms slow down considerably as M<\/strong>increases.<\/p>\n\n\n\n

                \u2022Common Techniques:<\/strong><\/p>\n\n\n\n

                \u20221. Latent Semantic Indexing (LSI) \/ Singular Value Decomposition (SVD):<\/strong>Decomposes the Term-Document matrix to project data into a lower-dimensional latent space (k \u2248 300).<\/p>\n\n\n\n

                \u20222. Principal Component Analysis (PCA):<\/strong>Finds the directions (components) in the data that account for the most variance.<\/p>\n\n\n\n

                \u2022Goal:<\/strong>Preserve the underlying cluster structure while removing noise and improving computational efficiency.<\/p>\n\n\n\n

                \n\n\n\n

                Scalability Challenges in IR Clustering<\/h3>\n\n\n\n
                \"\"<\/figure>\n\n\n\n

                <\/p>\n\n\n\n

                <\/div>\n\n\n\n
                \n\n\n\n

                Scalability Solutions: Sampling and Bisection<\/h3>\n\n\n\n

                \u20221. Random Sampling:<\/strong><\/p>\n\n\n\n

                \u2022Cluster only a representative sample<\/strong>of the documents (e.g., 1% of the corpus).<\/p>\n\n\n\n

                \u2022Use the resulting centroids\/structure to guide the assignment of the remaining 99%.<\/p>\n\n\n\n

                \u2022Risk:<\/strong>The sample may not perfectly capture the true distribution of topics.<\/p>\n\n\n\n

                \u20222. Bisecting K-Means:<\/strong><\/p>\n\n\n\n

                \u2022A hybrid approach blending partitionaland divisive hierarchical concepts.<\/p>\n\n\n\n

                \u2022Method:<\/strong>Start with one cluster containing all documents. In each step, choose a cluster to split and use K-Means with K=2<\/strong>to divide it into two sub-clusters.<\/p>\n\n\n\n

                \u2022Advantage:<\/strong>Faster than traditional K-Means because it runs K-Means on smaller subsets, and it is deterministic (produces a single hierarchy).<\/p>\n\n\n\n

                <\/p>\n\n\n\n

                \n\n\n\n

                Scalability Solutions: Centroid-Based Optimization<\/h3>\n\n\n\n
                \"\"<\/figure>\n\n\n\n

                <\/p>\n\n\n\n

                \n\n\n\n

                Soft Clustering and Multilingual IR<\/h3>\n\n\n\n

                \u2022Soft (Fuzzy) Clustering:<\/strong>Documents are assigned a degree of membership<\/strong>(a probability or weight) to every<\/em>cluster.<\/p>\n\n\n\n

                \u2022Benefit in IR:<\/strong>A document covering multiple topics (e.g., “AI ethics and law”) is appropriately represented in both the “AI” and “Law” clusters.<\/p>\n\n\n\n

                \u2022Common Algorithm:Expectation-Maximization (EM) for Gaussian Mixture Models (GMMs)<\/strong>.<\/p>\n\n\n\n

                \u2022Multilingual Clustering:<\/strong><\/p>\n\n\n\n

                \u2022Documents in different languages covering the same topic should cluster together.<\/p>\n\n\n\n

                \u2022Requirement:<\/strong>Needs a language-independent representation<\/strong>, such as Cross-Lingual LSI (CL-LSI) or machine translation to a single intermediary language <\/p>\n\n\n\n

                \n\n\n\n

                Application I: Search Result Clustering (SRC)<\/h3>\n\n\n\n

                \u2022Type:Post-Retrieval Clustering<\/strong><\/p>\n\n\n\n

                \u2022Process:<\/strong>The system first executes the query using a standard ranking model (e.g., BM25) and retrieves the top N<\/strong>documents (N <\/strong>\u224850-200<\/strong>). It then clusters only this small result set.<\/p>\n\n\n\n

                \u2022User Benefit:<\/strong>Improves the browsing<\/strong>experience. Instead of a long ranked list, the user sees a few meaningful cluster labels.<\/p>\n\n\n\n

                \u2022Challenges:<\/strong><\/p>\n\n\n\n

                \u2022Speed:<\/strong>Must be executed in real-time<\/strong>(milliseconds) to avoid user latency. Requires very fast algorithms.<\/p>\n\n\n\n

                \u2022Labeling:<\/strong>Clusters must be accurately and concisely labeled<\/strong>using terms extracted from the cluster documents (e.g., using phrases or named entities).<\/p>\n\n\n\n

                <\/div>\n\n\n\n
                \n\n\n\n

                Application II: Pre-Retrieval Clustering (Corpus-Based)<\/h3>\n\n\n\n
                \"\"<\/figure>\n\n\n\n

                <\/p>\n\n\n\n

                \n\n\n\n

                The Cluster Labeling Problem<\/h3>\n\n\n\n

                \u2022Crucial for SRC:<\/strong>A cluster is useless to a user without a descriptive, human-readable label.<\/p>\n\n\n\n

                \u2022Goal:<\/strong>Generate a short phrase (2\u20134 words) that accurately summarizes the topic of all documents in the cluster.<\/p>\n\n\n\n

                \u2022Methods:<\/strong><\/p>\n\n\n\n

                \u20221. Centroid Terms:<\/strong>Use the terms with the highest weight (TF-IDF) in the cluster centroid vector.<\/p>\n\n\n\n

                \u20222. Frequent Phrases:<\/strong>Extract frequent, non-stop-word phrases (N-grams) that appear across the cluster documents.<\/p>\n\n\n\n

                \u20223. Named Entities:<\/strong>Utilize recognized names (people, organizations, locations) as labels.<\/p>\n\n\n\n

                \u2022Filtering:<\/strong>Labels must be filtered to remove overly general or non<\/p>\n\n\n\n

                <\/p>\n\n\n\n

                \n\n\n\n

                Determining the Optimal Number of Clusters (K)<\/h3>\n\n\n\n

                The K Problem:<\/strong>Only some partitionalalgorithms (like K-Means) require K<\/strong>as input. Choosing K<\/strong>is non-trivial.<\/p>\n\n\n\n

                \u20221. Elbow Method:<\/strong><\/p>\n\n\n\n

                \u2022Plot the SSE<\/strong>(Sum of Squared Errors) against the number of clusters K<\/strong>.<\/p>\n\n\n\n

                \u2022Rationale:<\/strong>As K<\/strong>increases, the SSE naturally drops. The optimal K<\/strong>is often found at the “elbow”\u2014where the rate of decrease dramatically slows down.<\/p>\n\n\n\n

                \u20222. Silhouette Analysis:<\/strong><\/p>\n\n\n\n

                \u2022Measures how similar a document is to its own cluster compared to other clusters.<\/p>\n\n\n\n

                \u2022Goal:<\/strong>Maximize the average Silhouette Coefficient<\/strong>over all documents.<\/p>\n\n\n\n

                \u20223. Domain Knowledge:<\/strong>Sometimes, K<\/strong>is fixed based on known categories or user interface constraints (e.g., “show 10 clusters”).<\/p>\n\n\n\n

                \n\n\n\n

                Evaluation of Clustering: Internal Metrics<\/h3>\n\n\n\n

                \u2022Concept:<\/strong>Assess the quality of the clustering structure without<\/em>reference to external, pre-defined class labels. Based purely on the data and the distance metric.<\/p>\n\n\n\n

                \u20221. Cohesion (Intra-Cluster Similarity):<\/strong><\/p>\n\n\n\n

                \u2022Measures how tightly related the documents within a cluster are.<\/p>\n\n\n\n

                \u2022Metrics:<\/strong>Average similarity of all documents to the cluster centroid (or the mean pairwise distance). We want high cohesion.<\/strong><\/p>\n\n\n\n

                \u20222. Separation (Inter-Cluster Dissimilarity):<\/strong><\/p>\n\n\n\n

                \u2022Measures how distinct clusters are from each other.<\/p>\n\n\n\n

                \u2022Metrics:<\/strong>Distance between cluster centroids (e.g., Euclidean distance). We want high separation.<\/strong><\/p>\n\n\n\n

                \u20223. Silhouette Coefficient:<\/strong>Combines cohesion and separation into a single score.<\/p>\n\n\n\n

                <\/p>\n\n\n\n

                <\/div>\n\n\n\n
                \n\n\n\n

                Evaluation of Clustering: External Metrics<\/h3>\n\n\n\n
                \"\"<\/figure>\n\n\n\n

                <\/p>\n\n\n\n

                \n\n\n\n

                Evaluation Pitfall: The Subjectivity of Clustering<\/h3>\n\n\n\n

                \u2022Clustering is Subjective:<\/strong>Unlike classification, there is no single, objective “correct” clustering for a dataset. Different goals require different clustering structures.<\/p>\n\n\n\n

                \u2022For efficiency<\/strong>, a flat, coarse clustering (low K<\/strong>) might be best.<\/p>\n\n\n\n

                \u2022For user browsing<\/strong>, a fine-grained, meaningful clustering (higher K<\/strong>) is better.<\/p>\n\n\n\n

                \u2022Impact of Metrics:<\/strong>Internal metrics (cohesion\/separation) and external metrics (purity\/F-measure) may disagree.<\/p>\n\n\n\n

                \u2022A clustering that optimizes Purity<\/strong>might fail to be useful in a search interface if the clusters are too small or lack good labels.<\/p>\n\n\n\n

                \u2022Conclusion:<\/strong>Clustering evaluation must consider the application context<\/strong>and human judgment (user studies) alongside numerical <\/p>\n\n\n\n

                \n\n\n\n

                Improving Retrieval Effectiveness via Clustering<\/h3>\n\n\n\n
                \"\"<\/figure>\n\n\n\n
                \n\n\n\n

                Case Study: Scatter\/Gather Interface<\/h3>\n\n\n\n

                \u2022Concept:<\/strong>A user-driven interactive clustering interface for browsing large document collections.<\/p>\n\n\n\n

                \u2022Steps:<\/strong><\/p>\n\n\n\n

                1.The user selects an initial set of documents or submits a query.<\/p>\n\n\n\n

                2.The result set is quickly clustered (the Scatter<\/strong>step).<\/p>\n\n\n\n

                3.The system presents the cluster labels and sizes.<\/p>\n\n\n\n

                4.The user selects one or more clusters to merge, refine, or drill down into (the Gather<\/strong>step).<\/p>\n\n\n\n

                5.The system re-clusters the gathered subset, repeating the process.<\/p>\n\n\n\n

                \u2022Advantage:<\/strong>Provides an effective topical overview and allows the user to dynamically navigate the space without multiple query reformulations.<\/p>\n\n\n\n

                <\/p>\n\n\n\n

                <\/div>\n\n\n\n


                Clustering and Personalization<\/h3>\n\n\n\n

                \u2022User Profiles:<\/strong>Documents retrieved by a system are clustered based not just on content, but also on the user’s past interaction history.<\/p>\n\n\n\n

                \u2022Method:<\/strong>Documents are weighted by a user-interest vector. Clustering then groups documents that are topically similar and<\/em>relevant to the user’s inferred interests.<\/p>\n\n\n\n

                \u2022Application: News and Recommendation Systems:<\/strong><\/p>\n\n\n\n

                \u2022Grouping news articles by topic, then prioritizing clusters where the user has shown previous interest.<\/p>\n\n\n\n

                \u2022Helps filter out irrelevant content and improve the perceived <\/p>\n\n\n\n

                \n\n\n\n

                Clustering as an Alternative to Indexing<\/h3>\n\n\n\n

                \u2022Inverted Indices<\/strong>are the standard method for fast retrieval in IR.<\/p>\n\n\n\n

                \u2022Clustering<\/strong>offers an alternative access structure.<\/p>\n\n\n\n

                \u2022Comparison:<\/strong><\/p>\n\n\n\n

                \u2022Indexing:High Precision\/High Recall<\/strong>. Comparison cost is O(query terms)<\/em><\/strong>. Highly accurate.<\/p>\n\n\n\n

                \u2022Clustering:Reduced Cost\/Potentially Lower Recall<\/strong>. Comparison cost is O(number of clusters)<\/em><\/strong>. Fast, but lossy.<\/p>\n\n\n\n

                \u2022Hybrid Systems:<\/strong>The most common approach is to use clustering in conjunction<\/strong>with an inverted index\u2014e.g., using clustering to find the <\/p>\n\n\n\n

                \n\n\n\n

                The Impact of Sparsity<\/h3>\n\n\n\n

                \u2022Sparsity:<\/strong>Document vectors are extremely sparse<\/strong>(most dimensions\/terms have a zero weight) in IR.<\/p>\n\n\n\n

                \u2022Consequence for Distance:<\/strong>Sparse vectors often have few co-occurring non-zero dimensions, leading to small or zero dot products (high Cosine Similarity if normalized, but the space is complex).<\/p>\n\n\n\n

                \u2022Mitigation:<\/strong><\/p>\n\n\n\n

                \u2022Dimensionality Reduction (LSI):<\/strong>Creating dense vectors in a latent space mitigates the sparsity problem.<\/p>\n\n\n\n

                \u2022Careful Metric Selection:<\/strong>Use metrics that handle high-dimensional sparsity well (like Cosine Similarity) over Euclidean distance.<\/p>\n\n\n\n

                \n\n\n\n

                Challenges: Noise and Outliers<\/h3>\n\n\n\n

                \u2022Noise:<\/strong>Documents that are short, contain little content, or have irrelevant content (e.g., boilerplate text, spam) can distort cluster centers.<\/p>\n\n\n\n

                \u2022Outliers:<\/strong>Documents that are entirely dissimilar to the rest of the collection.<\/p>\n\n\n\n

                \u2022K-Means Problem:<\/strong>Outliers can hijack a cluster centroid, pulling it away from the true cluster mean.<\/p>\n\n\n\n

                \u2022DBSCAN Solution:<\/strong>Density-based methods, explicitly mark outliers as “noise” points that do not belong to any cluster, making the resulting clusters cleaner.<\/p>\n\n\n\n

                <\/div>\n<\/div>\n\n\n\n
                \n
                \n\n\n\n

                Summary: Clustering in IR<\/h3>\n\n\n\n

                \u2022Objective:<\/strong>Group similar documents to improve organization, user browsing, efficiency, and retrieval effectiveness (Recall).<\/p>\n\n\n\n

                \u2022Foundation:<\/strong>The Cluster Hypothesis<\/strong>\u2014similar documents are relevant to similar queries.<\/p>\n\n\n\n

                \u2022Key Algorithms:<\/strong><\/p>\n\n\n\n

                \u2022K-Means:<\/strong>Fast, scalable, requires pre-defined K<\/strong>, assumes spherical clusters.<\/p>\n\n\n\n

                \u2022Hierarchical (HAC\/DHC):<\/strong>Creates a rich structural hierarchy (Dendrogram), more complex\/slower.<\/p>\n\n\n\n

                \u2022Evaluation:<\/strong>Requires both internal metrics<\/strong>(Cohesion\/Separation) and external metrics<\/strong>(Purity\/F-Measure) relative to the final application goal.<\/p>\n\n\n\n

                \n\n\n\n

                Future Directions in IR Clustering<\/h3>\n\n\n\n

                \u20221. Dynamic and Streaming Clustering:<\/strong><\/p>\n\n\n\n

                \u2022Clustering documents (e.g., news feeds, social media posts) that arrive continuously and rapidly. The cluster structure must be maintained without re-clustering the entire history.<\/p>\n\n\n\n

                \u20222. Integrated Topic Modeling:<\/strong><\/p>\n\n\n\n

                \u2022Moving beyond traditional geometric clustering by using probabilistic models<\/strong>(like Latent DirichletAllocation -LDA) to define clusters based on latent topics, leading to more semantically meaningful cluster labels.<\/p>\n\n\n\n

                \u20223. GPU-Accelerated Clustering:<\/strong><\/p>\n\n\n\n

                \u2022Leveraging parallel processing hardware to achieve real-time clustering performance necessary for high-volume, real-time web search result organization.<\/p>\n\n\n\n

                \n\n\n\n

                Conclusion<\/h3>\n\n\n\n

                \u2022Clustering is a vital component of modern Information Retrieval systems, transforming vast, undifferentiated document collections into organized, navigable topical structures.<\/p>\n\n\n\n

                \u2022Key Takeaway:<\/strong>The choice of clustering algorithm, similarity metric, and evaluation method must always be dictated by the specific IR goal: efficiency, effectiveness, or user experience.<\/p>\n\n\n\n

                \n\n\n\n
                <\/div>\n<\/div>\n<\/div>\n\n\n\n
                \n
                \n
                \n


                Recommender Systems<\/h2>\n<\/div>\n\n\n\n
                <\/div>\n<\/div>\n<\/div>\n\n\n\n
                \n
                \n
                \n\n\n\n

                Introduction to Recommender Systems<\/h3>\n\n\n\n
                \"\"<\/figure>\n\n\n\n

                <\/p>\n\n\n\n

                \n\n\n\n

                Types of Feedback and Data<\/h3>\n\n\n\n

                \u2022Explicit Feedback:<\/strong><\/p>\n\n\n\n

                \u2022Direct input from users regarding their interest in items.<\/p>\n\n\n\n

                \u2022Examples:<\/strong>Star ratings (1-5 stars on Amazon), Like\/Dislike (YouTube), Numerical scores.<\/p>\n\n\n\n

                \u2022Pros:<\/strong>High precision; clear signal of user preference.<\/p>\n\n\n\n

                \u2022Cons:<\/strong>Sparse data (users rarely rate); reporting bias (users usually rate only things they love or hate).<\/p>\n\n\n\n

                \u2022Implicit Feedback:<\/strong><\/p>\n\n\n\n

                \u2022Inferences made from user behavior.<\/p>\n\n\n\n

                \u2022Examples:<\/strong>Purchase history, video watch time, page views, click-through rate, listening history (Spotify).<\/p>\n\n\n\n

                \u2022Pros:<\/strong>Abundant data; no extra effort required from the user.<\/p>\n\n\n\n

                \u2022Cons:<\/strong>Noisy data (a click doesn’t guarantee satisfaction); binary interpretation (interacted vs. didn’t interact) is often required.<\/p>\n\n\n\n

                \u2022The Utility Matrix:<\/strong><\/p>\n\n\n\n

                \u2022The central data structure is a matrix with Users as rows and Items as columns.<\/p>\n\n\n\n

                \u2022The matrix is extremely sparse<\/strong>(often >99% empty). The goal of RS is effectively “Matrix Completion.”<\/p>\n\n\n\n

                \n\n\n\n

                Taxonomy of Recommendation Approaches<\/h3>\n\n\n\n

                \u20221. Content-Based Filtering (CBF):<\/strong><\/p>\n\n\n\n

                \u2022Recommends items similar to those the user Liked in the past.<\/p>\n\n\n\n

                \u2022Relies on Item Features<\/strong>(keywords, genre, director, etc.) and a User Profile<\/strong>built from those features.<\/p>\n\n\n\n

                \u20222. Collaborative Filtering (CF):<\/strong><\/p>\n\n\n\n

                \u2022Recommends items based on the preferences of similar users.<\/p>\n\n\n\n

                \u2022Relies on the Interaction Matrix<\/strong>(ratings\/behavior) rather than item metadata.<\/p>\n\n\n\n

                \u2022Key Idea:<\/strong>“Users who agreed in the past will agree in the future.”<\/p>\n\n\n\n

                \u20223. Hybrid Methods:<\/strong><\/p>\n\n\n\n

                \u2022Combines CBF and CF to overcome limitations like the Cold Start problem.<\/p>\n\n\n\n

                \u20224. Knowledge-Based:<\/strong><\/p>\n\n\n\n

                \u2022Uses explicit domain knowledge and requirements (e.g., “I need a laptop with 16GB RAM for under $1000”). Common in complex domains like real estate or electronics.<\/p>\n\n\n\n

                \n\n\n\n

                Content-Based Filtering: Mechanics<\/h3>\n\n\n\n
                \"\"<\/figure>\n\n\n\n

                <\/p>\n\n\n\n

                <\/div>\n<\/div>\n\n\n\n
                \n
                \n\n\n\n

                Collaborative Filtering: User-Based<\/h3>\n\n\n\n
                \"\"<\/figure>\n\n\n\n

                <\/p>\n\n\n\n

                \n\n\n\n

                Collaborative Filtering: Item-Based<\/h3>\n\n\n\n
                \"\"<\/figure>\n\n\n\n

                <\/p>\n\n\n\n

                \n\n\n\n

                Model-Based CF: Matrix Factorization<\/h3>\n\n\n\n
                \"\"<\/figure>\n\n\n\n

                <\/p>\n\n\n\n

                \n\n\n\n

                The Cold Start Problem & Hybridization<\/h3>\n\n\n\n
                \"\"<\/figure>\n\n\n\n

                <\/p>\n\n\n\n

                <\/div>\n\n\n\n
                \n\n\n\n

                Evaluation Metrics (Prediction vs. Ranking)<\/h3>\n\n\n\n
                \"\"<\/figure>\n\n\n\n

                <\/p>\n\n\n\n

                \n\n\n\n

                Challenges and Advanced Topics<\/h3>\n\n\n\n

                \u2022Scalability:<\/strong><\/p>\n\n\n\n

                \u2022Real-world systems have millions of users and items. O(N2) neighbor calculations are too slow.<\/p>\n\n\n\n

                \u2022Solutions:<\/strong>Clustering, Dimensionality Reduction (SVD), Approximate Nearest Neighbors (ANN) searching (e.g., Faiss).<\/p>\n\n\n\n

                \u2022Data Sparsity:<\/strong><\/p>\n\n\n\n

                \u2022When the interaction matrix is < 0.1% full, finding overlaps between users is difficult.<\/p>\n\n\n\n

                \u2022Solutions:<\/strong>Matrix Factorization, implicating trust networks (Trust-aware Recommenders).<\/p>\n\n\n\n

                \u2022The “Filter Bubble”:<\/strong><\/p>\n\n\n\n

                \u2022System keeps recommending what the user already agrees with\/likes, reducing diversity.<\/p>\n\n\n\n

                \u2022Evaluation:<\/strong>Need metrics for Novelty<\/strong>(recommending unknown items) and Serendipity<\/strong>(recommending surprisingly interesting items).<\/p>\n\n\n\n

                \u2022Shilling Attacks:<\/strong><\/p>\n\n\n\n

                \u2022Malicious users creating fake profiles to rate items highly (push attacks) or poorly (nuke attacks) to manipulate the system.<\/p>\n\n\n\n

                \u2022Robustness:<\/strong>Designing algorithms resistant to outliers and fake profiles.<\/p>\n\n\n\n

                <\/p>\n\n\n\n

                \n\n\n\n
                <\/div>\n<\/div>\n<\/div>\n","protected":false},"excerpt":{"rendered":"

                What is Clustering? \u2022Definition:The fundamental task of grouping a set of objects (documents) such that objects in the same group (cluster) are more similarto each other than to those in other groups. \u2022It is the most prevalent form of unsupervised learningin IR. \u2022Goal:To achieve high intra-cluster similarity(documents inside a cluster are highly similar) and high […]<\/p>\n","protected":false},"author":1,"featured_media":0,"parent":113,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-238","page","type-page","status-publish","hentry"],"_links":{"self":[{"href":"http:\/\/ijeesoo.com\/index.php?rest_route=\/wp\/v2\/pages\/238","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/ijeesoo.com\/index.php?rest_route=\/wp\/v2\/pages"}],"about":[{"href":"http:\/\/ijeesoo.com\/index.php?rest_route=\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"http:\/\/ijeesoo.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/ijeesoo.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=238"}],"version-history":[{"count":73,"href":"http:\/\/ijeesoo.com\/index.php?rest_route=\/wp\/v2\/pages\/238\/revisions"}],"predecessor-version":[{"id":336,"href":"http:\/\/ijeesoo.com\/index.php?rest_route=\/wp\/v2\/pages\/238\/revisions\/336"}],"up":[{"embeddable":true,"href":"http:\/\/ijeesoo.com\/index.php?rest_route=\/wp\/v2\/pages\/113"}],"wp:attachment":[{"href":"http:\/\/ijeesoo.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=238"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}