一种采用声学指纹去重的海量MP3文件存储架构  

Storage architecture of mass MP3 files using acoustic fingerprint deduplication

作　　者：赵晓永[1] 杨扬[1] 孙莉莉[1] 

机构地区：[1]北京科技大学计算机与通信工程学院,北京100083

出　　处：《计算机应用研究》2013年第2期472-475,共4页Application Research of Computers

基　　金：国家自然科学基金资助项目(61070182;60873192)

摘　　要：由于Hadoop自身并不适合海量小文件处理,目前的重复数据删除方法主要基于文件的二进制特征,无法识别经过信号处理后的同一首歌曲,也不能满足海量数据在线处理的要求。提出一种采用声学指纹去重的海量MP3文件存储架构,结合音乐文件自身的声学特性和MP3文件包含的元信息,通过索引、在线归并和NAF去重,很好地解决了小文件过多时内存瓶颈问题,同时提供了更好的去重效果;离线归并和副本调整模块根据系统的运行状况不断优化存储。实验结果表明,该架构在性能、去重率、可管理性和可扩展性方面达到了良好的平衡,极大地提高了去重率,与可变分块CDC相比,去重率提高了100%,具有良好的实用价值。Due to the Hadoop itself is not suitable for processing of the mass of small files. And current data de-duplication methods are mainly based on the binary characteristics of the file, so it cannot recognize the same song after the signal process- ing and also cannot meet the requirements of the online processing of massive data. This paper presented a de-duplication stor- age architecture of the mass of the MP3 file based on the acoustic fingerprint. It combined with music files on the acoustic char- acteristics and the recta-information of MP3 files, de-duplication by index, merge online and NAF, solved the memory bottle- neck problem effectively in the face of too many small files. At the same time it provided a better de-duplication effect. Offline merge and the replication place module optimized storage continually according to the operating conditions of the system. The experimental results show that the architecture can achieve a good balance on performance, the rate of de-duplication, manage- ability and scalability.

关 键 词：HADOOP MP3 声学指纹 数据消重 存储架构 

分 类 号：TP311[自动化与计算机技术—计算机软件与理论]