GAAF:Searching Activation Functions for Binary Neural Networks Through Genetic Algorithm  

作　　者：Yanfei Li Tong Geng Samuel Stein Ang Li Huimin Yu 

机构地区：[1]Department of Information Science and Electronic Engineering,Zhejiang University,Hangzhou 310027,China [2]h Pacific Northwest National Laboratory,Richland,WA 99354,USA

出　　处：《Tsinghua Science and Technology》2023年第1期207-220,共14页清华大学学报（自然科学版（英文版）

摘　　要：Binary neural networks(BNNs)show promising utilization in cost and power-restricted domains such as edge devices and mobile systems.This is due to its significantly less computation and storage demand,but at the cost of degraded performance.To close the accuracy gap,in this paper we propose to add a complementary activation function(AF)ahead of the sign based binarization,and rely on the genetic algorithm(GA)to automatically search for the ideal AFs.These AFs can help extract extra information from the input data in the forward pass,while allowing improved gradient approximation in the backward pass.Fifteen novel AFs are identified through our GA-based search,while most of them show improved performance(up to 2.54%on ImageNet)when testing on different datasets and network models.Interestingly,periodic functions are identified as a key component for most of the discovered AFs,which rarely exist in human designed AFs.Our method offers a novel approach for designing general and application-specific BNN architecture.GAAF will be released on GitHub.

关 键 词：binary neural networks(BNNs) genetic algorithm activation function 

分 类 号：TN9[电子电信—信息与通信工程]