We investigated the surface-enhanced Raman scattering activity and spatio-temporal characteristics of plasmons excited on a rough gold film using ultrafast near-field optical microscopy. Near-field nonlinear excitation images showed a number of optically amplified spots on the sample surface. Raman scattering from molecules dispersed on the sample was enhanced at the amplified fields in a space-selective manner. Time-resolved near-field measurements revealed that the dephasing time varies depending on the excited plasmon. We evaluated the resonant mode volume from the Raman signal enhancement and the dephasing time of the plasmon, and found that the volume is on the same order as the one estimated from the topography of the sample.