Tensor Perspective of Deep Neural Networks

Prof. Dmitry Vetrov (speaker)
National Research University Higher School of Economics, Yandex School of Data Analysis
Alexander Novikov, Dmitry Podoprikhin
Skoltech
Anton Osokin
INRIA - SIERRA project-team
Deep neural networks currently demonstrate state-of-the-art performance in several domains. At the same time, models of this class are very demanding in terms of computational resources. In particular, a large amount of memory is required by fully-connected layers which are used in a lot of different architectures. This makes it hard to use these models on low-end devices and stops the further increase of the model size. We propose a compact parametrization for fully-connected layers. Our approach consists in representing the dense weight matrix of the layer in the Tensor Train format [16]. This format allows to reduce the number of parameters by a huge factor while keeping the expressive power of the layer. In particular, for the Very Deep VGG networks [20] we report the compression of the dense weight matrix of a fully-connected layer up to 200000 times leading to the compression of the whole network up to 7 times.