一种改进的随机检验法用于主成分选择以避免光谱分析校正模型的过拟合或欠拟合  

A Principal Components Selection Method Based on the Modified Randomization Test for Avoiding Over-Fit and Under-Fit in Spectra Calibration

作　　者：李丽娜[1] 李庆波[1] 阎侯赖[1] 张广军[1] 

机构地区：[1]北京航空航天大学仪器科学与光电工程学院,精密光机电一体化技术教育部重点实验室,北京100191

出　　处：《光谱学与光谱分析》2010年第11期3041-3046,共6页Spectroscopy and Spectral Analysis

基　　金：国家自然科学基金项目(60708026);长江学者和创新团队发展计划项目(IRT0205)资助

摘　　要：为了避免主成分个数选择不当引起的校正模型过拟合或欠拟合,提出了一种改进的随机检验法,应用该方法对样品复杂程度递增的三组近红外光谱数据进行了实验研究,并与交互验证法进行了比较,分析了模型复杂程度对光谱定量校正模型预测能力的影响,讨论了该方法对复杂样品的适用性问题。结果显示,该方法可避免交互验证法剔除样本的过程,考虑了全部训练样本的信息,可客观地选择主成分,有助于避免过拟合或欠拟合,提高校正模型的预测精度;该方法不同于一般随机检验法的统计检验过程,简化了判据,易实现,选择过程可视化、可交互;在三组实验中,分别选择4,5和8个主成分建模,其外部独立预测集的预测结果最优;该方法适用于小样本复杂样品建模。More or less principal components often give an over-fit or under-fit quantitative calibration model.In order to avoid over-fit or under-fit in spectra calibration,aprincipal components selection method based on a modified randomization test is proposed.Three near infrared spectra experiments（the complexity of the sample components in each experiment is increasing by degrees）are introduced in this paper for evaluating the proposed method.The method is compared with the cross-validation method.And the spectra model complexity of how to affect the prediction performance of calibration is discussed.Then the adaptability of this modified randomization test to the uncertainty complex spectra model is also discussed.The results indicate that the proposed method has no process of leaving some samples out like cross-validation does,and all the training samples are considered when selecting principal components,so the problem of over-fit or under-fit can be avoided,which is benefit to improve prediction performance of calibration in spectral analysis.And the modified randomization test method is different with the commonly used randomization test that a simplified criterion is introduced here and it is easy to implement.With the proposed method,the authors can have a visualized and interactive process when selecting principal components.For these three experiments,4,5 and 8 selected principal components are employed in calibration respectively and the prediction result is the best for the independent external prediction sets.It is also implied that the proposed method is adaptable to the complex samples with more variables and little samples.

关 键 词：光谱分析 定量校正 随机检验 偏最小二乘 主成分 

分 类 号：O657.3[理学—分析化学]