In this paper, we propose an acceleration method of structural similarity (SSIM) and its multi-scaled version, called MS-SSIM. The calculation process of SSIM and MS-SSIM includes multiple Gaussian filters, and the cost of the filter is dominant for the entire process; thus, to accelerate SSIM/MS-SSIM, we replace Gaussian filtering using convolution with sliding DCT. Gaussian filter based on sliding DCT is faster than the usual convolution method. Besides, its computational complexity does not depend on the filter window length. Also, naive implementations of SSIM and MS-SSIM scan image many times for the pixel-wise operation; however, these operations can be incorporated into Gaussian filtering. Thus, we optimize the processing pipeline to achieve high cache-efficiency. As a result, the proposed SSIM computation was accelerated by 6.36 times and MS-SSIM by 8.11 times faster than the conventional approach.