Joint Structure and Parameter Optimization of Multiobjective Sparse Neural Network

This work addresses the problem of network pruning and proposes a novel joint training method based on a multiobjective optimization model. Most of the state-of-the-art pruning methods rely on user experience for selecting the sparsity ratio of the weight matrices or tensors, and thus suffer from severe performance reduction with inappropriate user-defined parameters. Moreover, networks might be inferior due to the inefficient connecting architecture search, especially when it is highly sparse. It is revealed in this work that the network model might maintain sparse characteristic in the early stage of the backpropagation (BP) training process, and evolutionary computation-based algorithms can accurately discover the connecting architecture with satisfying network performance. In particular, we establish a multiobjective sparse model for network pruning and propose an efficient approach that combines BP training and two modified multiobjective evolutionary algorithms (MOEAs). The BP algorithm converges quickly, and the two MOEAs can search for the optimal sparse structure and refine the weights, respectively. Experiments are also included to prove the benefits of the proposed algorithm. We show that the proposed method can obtain a desired Pareto front (PF), leading to a better pruning result comparing to the state-of-the-art methods, especially when the network structure is highly sparse.