Understanding Cluster Analysis: Types, Methods, and Applications, Study notes of Computer Science

Download Understanding Cluster Analysis: Types, Methods, and Applications and more Study notes Computer Science in PDF only on Docsity! Cluster Analysis Cluster Analysis  What is Cluster Analysis?  Types of Data in Cluster Analysis  A Categorization of Major Clustering Methods  Partitioning Methods  Hierarchical Methods  Density-Based Methods  Grid-Based Methods  Model-Based Clustering Methods  Outlier Analysis  Summary What Is Good Clustering?  A good clustering method will produce high quality clusters with  high intra-class similarity  low inter-class similarity  The quality of a clustering result depends on both the similarity measure used by the method and its implementation.  The quality of a clustering method is also measured by its ability to discover some or all of the hidden patterns. Requirements of Clustering in Data Mining  Scalability  Ability to deal with different types of attributes  Discovery of clusters with arbitrary shape  Minimal requirements for domain knowledge to determine input parameters  Able to deal with noise and outliers  Insensitive to order of input records  High dimensionality  Incorporation of user-specified constraints  Interpretability and usability Outliers  Outliers are objects that do not belong to any cluster or form clusters of very small cardinality  In some applications we are interested in discovering outliers, not clusters (outlier analysis) cluster outliers Measuring Similarity in Clustering  Dissimilarity/Similarity metric:  The dissimilarity d(i, j) between two objects i and j is expressed in terms of a distance function, which is typically a metric:  d(i, j)0 (non-negativity)  d(i, i)=0 (isolation)  d(i, j)= d(j, i) (symmetry)  d(i, j) ≤ d(i, h)+d(h, j) (triangular inequality)  The definitions of distance functions are usually different for interval-scaled, boolean, categorical, ordinal and ratio-scaled variables.  Weights may be associated with different variables based on applications and data semantics. Type of data in cluster analysis  Interval-scaled variables  e.g., salary, height  Binary variables  e.g., gender (M/F), has_cancer(T/F)  Nominal (categorical) variables  e.g., religion (Christian, Muslim, Buddhist, Hindu, etc.)  Ordinal variables  e.g., military rank (soldier, sergeant, lutenant, captain, etc.)  Ratio-scaled variables  population growth (1,10,100,1000,...)  Variables of mixed types  multiple attributes with various types Similarity and Dissimilarity Between Objects  Distance metrics are normally used to measure the similarity or dissimilarity between two data objects  The most popular conform to Minkowski distance: where i = (xi1, xi2, …, xin) and j = (xj1, xj2, …, xjn) are two n-dimensional data objects, and p is a positive integer  If p = 1, L1 is the Manhattan (or city block) distance: pp jn x in x p j x i x p j x i xjipL /1 ||...| 22 || 11 |),(            ||...||||),( 1 2211 nn j x i x j x i x j x i xjiL  Binary Variables  Another approach is to define the similarity of two objects and not their distance.  In that case we have the following:  Simple matching coefficient similarity:  Jaccard coefficient similarity: dcba da jis  ),( cba a jis  ),( Note that: s(i,j) = 1 – d(i,j) Dissimilarity between Binary Variables  Example (Jaccard coefficient)  all attributes are asymmetric binary  1 denotes presence or positive test  0 denotes absence or negative test Name Fever Cough Test-1 Test-2 Test-3 Test-4 Jack 1 0 1 0 0 0 Mary 1 0 1 0 1 0 Jim 1 1 0 0 0 0 75.0 211 21 ),( 67.0 111 11 ),( 33.0 102 10 ),(             maryjimd jimjackd maryjackd  Each variable is mapped to a bitmap (binary vector)  Jack: 101000  Mary: 101010  Jim: 110000  Simple match distance:  Jaccard coefficient: A simpler definition Name Fever Cough Test-1 Test-2 Test-3 Test-4 Jack 1 0 1 0 0 0 Mary 1 0 1 0 1 0 Jim 1 1 0 0 0 0 bits ofnumber total positionsbit common -non ofnumber ),( jid in s1' ofnumber in s1' ofnumber 1),( ji ji jid    Major Clustering Approaches  Partitioning algorithms: Construct random partitions and then iteratively refine them by some criterion  Hierarchical algorithms: Create a hierarchical decomposition of the set of data (or objects) using some criterion  Density-based: based on connectivity and density functions  Grid-based: based on a multiple-level granularity structure  Model-based: A model is hypothesized for each of the clusters and the idea is to find the best fit of that model to each other Cluster Analysis  What is Cluster Analysis?  Types of Data in Cluster Analysis  A Categorization of Major Clustering Methods  Partitioning Methods  Hierarchical Methods  Density-Based Methods  Grid-Based Methods  Model-Based Clustering Methods  Outlier Analysis  Summary Partitioning Algorithms: Basic Concepts  Partitioning method: Construct a partition of a database D of n objects into a set of k clusters  Given a k, find a partition of k clusters that optimizes the chosen partitioning criterion  Global optimal: exhaustively enumerate all partitions  Heuristic methods: k-means and k-medoids algorithms  k-means (MacQueen’67): Each cluster is represented by the center of the cluster  k-medoids or PAM (Partition around medoids) (Kaufman & Rousseeuw’87): Each cluster is represented by one of the objects in the cluster Comments on the k-means Method  Strength  Relatively efficient: O(tkn), where n is # objects, k is # clusters, and t is # iterations. Normally, k, t << n.  Often terminates at a local optimum.  Weaknesses  Applicable only when mean is defined, then what about categorical data?  Need to specify k, the number of clusters, in advance  Unable to handle noisy data and outliers  Not suitable to discover clusters with non-convex shapes The K-Medoids Clustering Method  Find representative objects, called medoids, in clusters  PAM (Partitioning Around Medoids, 1987)  starts from an initial set of medoids and iteratively replaces one of the medoids by one of the non-medoids if it improves the total distance of the resulting clustering  PAM works effectively for small data sets, but does not scale well for large data sets  CLARA (Kaufmann & Rousseeuw, 1990)  CLARANS (Ng & Han, 1994): Randomized sampling PAM (Partitioning Around Medoids) (1987)  PAM (Kaufman and Rousseeuw, 1987), built in statistical package S+  Use real object to represent the cluster 1. Select k representative objects arbitrarily 2. For each pair of non-selected object h and selected object i, calculate the total swapping cost TCih 3. For each pair of i and h,  If TCih < 0, i is replaced by h  Then assign each non-selected object to the most similar representative object 4. repeat steps 2-3 until there is no change CLARA (Clustering Large Applications)  CLARA (Kaufmann and Rousseeuw in 1990)  Built in statistical analysis packages, such as S+  It draws multiple samples of the data set, applies PAM on each sample, and gives the best clustering as the output  Strength: deals with larger data sets than PAM  Weakness:  Efficiency depends on the sample size  A good clustering based on samples will not necessarily represent a good clustering of the whole data set if the sample is biased CLARANS (“Randomized” CLARA)  CLARANS (A Clustering Algorithm based on Randomized Search) (Ng and Han’94)  CLARANS draws sample of neighbors dynamically  The clustering process can be presented as searching a graph where every node is a potential solution, that is, a set of k medoids  If the local optimum is found, CLARANS starts with new randomly selected node in search for a new local optimum  It is more efficient and scalable than both PAM and CLARA  Focusing techniques and spatial access structures may further improve its performance (Ester et al.’95)