Latency-Aware Inference on Convolutional Neural Network Over Homomorphic Encryption

Homomorphic encryption enables privacy-preserving computation in convolutional neural networks (CNNs), keeping their input and output secret from the server; however, it faces long latency because of large overhead of the encryption scheme. This paper tackles shortening the inference latency on homomorphic encryption-enabled CNNs. Since the highest inference accuracy is not always needed depending on real-world applications, finding best-fit combinations of latency and accuracy is also indispensable. We propose a combination of channel-wise packing and a structured pruning technique besides changing the active functions to shorten the inference latency while allowing accuracy degradation. Our experimental evaluation shows that we successfully tune the latency from 8.1 s to 12.9 s depending on the accuracy of 66.52% to 80.96% on the CIFAR-10 dataset.