A Scheduling Optimization Technique Based on Reuse in Spark to Defend Against APT Attack  

作　　者：Jianchao Tang Ming Xu Shaojing Fu Kai Huang 

机构地区：[1]College of Computer, National University of Defense Technology, Changsha 410073, China [2]Sate Key Laboratory of Cryptology, Beijing 100878, China

出　　处：《Tsinghua Science and Technology》2018年第5期550-560,共11页清华大学学报（自然科学版（英文版）

基　　金：supported by the National Natural Science Foundation of China (Nos. 61379144, 61572026, 61672195, and 61501482);the Open Foundation of State Key Laboratory of Cryptology

摘　　要：Advanced Persistent Threat （APT） attack, an attack option in recent years, poses serious threats to the security of governments and enterprises data due to its advanced and persistent attacking characteristics. To address this issue, a security policy of big data analysis has been proposed based on the analysis of log data of servers and terminals in Spark. However, in practical applications, Spark cannot suitably analyze very huge amounts of log data. To address this problem, we propose a scheduling optimization technique based on the reuse of datasets to improve Spark performance. In this technique, we define and formulate the reuse degree of Directed Acyclic Graphs （DAGs） in Spark based on Resilient Distributed Datasets （RDDs）. Then, we define a global optimization function to obtain the optimal DAG sequence, that is, the sequence with the least execution time. To implement the global optimization function, we further propose a novel cost optimization algorithm based on the traditional Genetic Algorithm （GA）. Our experiments demonstrate that this scheduling optimization technique in Spark can greatly decrease the time overhead of analyzing log data for detecting APT attacks.Advanced Persistent Threat （APT） attack, an attack option in recent years, poses serious threats to the security of governments and enterprises data due to its advanced and persistent attacking characteristics. To address this issue, a security policy of big data analysis has been proposed based on the analysis of log data of servers and terminals in Spark. However, in practical applications, Spark cannot suitably analyze very huge amounts of log data. To address this problem, we propose a scheduling optimization technique based on the reuse of datasets to improve Spark performance. In this technique, we define and formulate the reuse degree of Directed Acyclic Graphs （DAGs） in Spark based on Resilient Distributed Datasets （RDDs）. Then, we define a global optimization function to obtain the optimal DAG sequence, that is, the sequence with the least execution time. To implement the global optimization function, we further propose a novel cost optimization algorithm based on the traditional Genetic Algorithm （GA）. Our experiments demonstrate that this scheduling optimization technique in Spark can greatly decrease the time overhead of analyzing log data for detecting APT attacks.

关 键 词：SPARK Advanced Persistent Threat （APT） SCHEDULE REUSE Resilient Distributed Dataset （RDD） Directed Acyclic Graph （DAG） Genetic Algorithm （GA） 

分 类 号：TP309[自动化与计算机技术—计算机系统结构]