The installation of photovoltaic system (PV) is increasing rapidly across the world. However, the fluctuation of PV output causes serious challenges in the power grid operation. Among the fluctuation, a quick part of fluctuation is mainly caused by the change of cloud coverage. A total-sky imager (TSI), measuring device to take sky and cloud, could be useful for short-term solar irradiance forecasting. In this paper, a convolutional neural network (CNN) is applied to forecasting model (called CNN model) to forecast 5-20 min ahead of global horizontal irradiance (GHI) using total-sky images and lagged GHI. To verify the effectiveness of CNN, three forecasting models are compared. They are the persistence model, the CNN model using only total-sky images, and the CNN model using both total-sky images and lagged GHI. From the computation, the proposed CNN model using both total-sky images and lagged GHI performs root-mean-square error (RMSE) of 49-177W/m2, 93-146W/m2, 71-118W/m2 in sunny day, partly cloudy day and overcast day, respectively. From these results, the proposed method is shown to be suitable for short-term solar irradiance forecasting.