泛Kennard-Stone算法的数据集代表性度量与分块采样策略  

A Dataset Representativeness Metric and A Slicing Sampling Strategy for the Kennard-Stone Algorithm

作　　者：武晴滢 祝震予 吴剑鸣[1] 徐昕[1] WU Qingying;ZHU Zhenyu;WU Jianming;XU Xin(Department of Chemistry,Fudan University,Shanghai 200438,China)

机构地区：[1]复旦大学化学系,上海200438

出　　处：《高等学校化学学报》2022年第10期290-297,共8页Chemical Journal of Chinese Universities

基　　金：国家自然科学基金(批准号:21373053,21688102)资助.

摘　　要：在大数据机器学习时代,选择更具代表性的数据集对于模型的训练和验证尤为重要.Kennard-Stone(KS)算法及其各种变种(泛KS算法)是一大类优异的数据集分割方法,但其采样比例或采样数的选择仅能依靠经验或根据建模结果事后评判.KS算法依据原始文献的计算复杂度为O(K^(3)),难以用于超大数据样本量的计算.本文基于数据集完备性的讨论,提出泛KS算法的数据集代表性度量,以简正振动采样的甲烷分子中碳氢键数据特征分布为例展示采样集代表性效果.简化KS采样过程的筛选算法,提高算法效率至O'(K^(2)).提出将数据集切分成多个子集分别实施KS采样的分块采样策略,可进一步提高算法效率至O″(K).偏最小二乘回归测试结果表明,该方法在提高采样效率的同时仍可保障采样集的代表性.In machine learning with big data,it is essential to prepare a representative dataset for training a model.The Kennard-Stone(KS)algorithm and its derivatives are a large class of excellent dataset splitting methods.But it rely heavily on empirical selection or modeling results to determine the sampling ratio and sampling number.In addition,its computational complexity is O(K^(3))according to the original literature,making it difficult to apply to massive data.In this paper,we design a metric based on dataset completeness to quantify the representativeness degree of an extracted subset to the whole dataset.An amendment using dynamic programming algorithm is put to reduce the algorithm complexity to O'(K^(2)).And a slicing sampling strategy is proposed to divide the whole dataset into several subset and implement KS sampling respectively,which can further improve the algorithm efficiency to O''(K).The partial least squares regression test results show that the method can improve the sampling efficiency while still ensuring the representativeness of the finally extracted dataset.

关 键 词：Kennard-Stone算法 数据完备性 采样集代表性 线性标度 

分 类 号：O641[理学—物理化学]