Digital holographic imaging systems are promising as they provide 3-D information of the object. However, the acquisition of holograms during experiments can be adversely affected by the speckle noise in coherent digital holographic systems. Several different denoising algorithms have been proposed. Traditional denoising algorithms average several holograms under different experimental conditions or use conventional filters to remove the speckle noise. However, these traditional methods require complex holographic experimental conditions. Besides time-consuming, the use of traditional neural networks has been difficult to extract speckle noise characteristics from holograms and the resulting holographic reconstructions have not been ideal. To address tradeoff between speckle noise reduction and efficiency, we analyze holograms in the spectrum domain for fast speckle noise reduction, which can remove multiple-levels speckle noise based on convolutional neural networks using only a single hologram. In order to effectively reduce the speckle noise associated with the hologram, the data set of the neural network training cannot use the current popular image data set. To achieve powerful noise reduction performance, neural networks use multiple-level speckle noise data sets for training. In contrast to existing traditional denoising algorithms, we use convolutional neural networks in spectral denoising for digital hologram. The proposed technique enjoys several desirable properties, including (i) the use of only a single hologram to efficiently handle various speckle noise levels, and (ii) faster speed than traditional approaches without sacrificing denoising performance. Experimental results and holographic reconstruction demonstrate the efficiency of our proposed neural network.