基于大数据与集成方法的文档聚类研究  

Research on Document Clustering Based on Big Data and Integration Method

作　　者：郑芳 李志威 王巍[1] ZHENG Fang;LI Zhiwei;WANG Wei(Information Center,Wuhan No.1 Hospital,Wuhan 430022)

机构地区：[1]武汉市第一医院信息中心,武汉430022

出　　处：《计算机与数字工程》2025年第1期152-157,208,共7页Computer & Digital Engineering

基　　金：武汉市市科技局知识创新专项基础研究项目“基于动态视觉识别技术的药品调剂核对解决方案”(编号:2022010801010522)资助。

摘　　要：聚类是无监督机器学习中广泛使用的技术,但是由于通常数据标签的不可用,因此对于其结果的质量评估是一个棘手的问题。尽管之前有许多方法可以验证聚类质量,但单一的度量指标无法做到全面的评估。此外,大数据中通常包含相当比例的噪声,此时聚类时需要进行额外的改进。基于这些痛点,论文在文档聚类时,首先进行包括词元化、停用词消除、词干提取与向量空间转换等数据集预处理,然后使用基于模糊逻辑的改进K均值和K中心方法对文档大数据进行了聚类,最后基于集成方法使用七个不同的有效性度量指标并基于绝对共识和多数共识对结果进行评估,以确定模糊聚类的最优质量。此外,还使用了不同的簇数对标准文档数据集进行聚类,并证明了论文方法可以确定最佳的聚类数量。Clustering is a widely used technique in unsupervised machine learning,but assessing the quality of its results is a challenging task due to the unavailability of data labels.Although various methods have been proposed to validate cluster quality,re⁃lying on a single metric is not sufficient for comprehensive evaluation.Moreover,big data often contain a significant amount of noise,requiring additional enhancements during the clustering process.In this work,these challenges are addressed by first prepro⁃cessing the document dataset,including tokenization,stop word elimination,stemming,and vector space transformation.Then fuzzy logic-based enhancements are applied to traditional K-means and K-medoids algorithms for clustering the large document da⁃taset.Finally,an ensemble method is proposed that utilizes seven different validity measures and evaluates the results based on ab⁃solute consensus and majority consensus,aiming to determine the optimal quality of fuzzy clusters.Additionally,the paper performs clustering on the Reuters standard document dataset using different numbers of clusters and demonstrates the ability of our method to determine the optimal number of clusters.

关 键 词：集成方法 数据挖掘 文档聚类 模糊逻辑 

分 类 号：TP391[自动化与计算机技术—计算机应用技术]