Network with intermediate feature visualization: (a) 1-bit CNN without shortcut, and (b) proposed Bi-Real net with shortcut propagating the real-valued features.
Deep convolutional neural networks (CNNs) are powerful but cumbersome. For implementing CNN on mobile devices, compressing a CNN is highly needed. We study compressing CNN via binarization. The so-called 1-bit convolutional neural networks have binary weights and binary activations. While efficient, the lack of representational capability and the training difficulty impede 1-bit CNNs from performing as well as real-valued networks.
We propose Bi-Real net with a novel training algorithm to tackle these two challenges. To enhance the representational capability, we propagate the real-valued activations generated by each 1-bit convolution via a parameter-free shortcut. To address the training difficulty, we propose improved training algorithms targeting the optimization difficulty in 1-bit convolution networks.
Our Bi-Real Net achieved 56.4% top-1 classification accuracy, which is 10% higher than the state-of-the-arts (e.g. , XNOR-Net) and proved to be effective on depth estimation task in real scenarios. The FPGA implementation of our Bi-Real net will also be available soon.
Zechun Liu, Baoyuan Wu, Wenhan Luo, Xin Yang, Wei Liu, and Kwang-Ting Cheng, "Bi-real net: Enhancing the performance of 1-bit cnns with improved representational capability and advanced training algorithm," In Proceedings of the European Conference on Computer Vision (ECCV), 2018.
Zechun Liu, Baoyuan Wu, Wenhan Luo, Xin Yang, Wei Liu, and Kwang-Ting Cheng, "Bi-real net: Binarizing deep network towards real-network performance," submitted for publication.