Bayon temple, Angkor, built in the late 12th century to the early 13th century, is a masonry temple made of sandstone which has deteriorated possibly due to rain, plants, settlements, and so on. Strong winds blowing more than 20m/sec were observed in the rainy season, which may cause vibration of the structure either directly or via ground motion, we may add wind induced vibration to the list of deterioration causes. Proceeding to a study on the response of the masonry structure to the strong wind, we carried out a micro tremor measurement to evaluate the vibration characteristics of the main tower and surrounding sub-towers in the temple. We identified translational and torsion modes of towers with their predominant frequencies, damping factors, and base-fixed frequencies. We examined all of the sub-towers to find statistically the effects of height and corridor connectivity on their vibration characteristics. Then, we attempted to construct an FEM model for the main tower by simulating the base-fixed natural frequencies of the horizontal translational modes. The model with an elastic modulus practically used for sandstone shows much higher frequencies, which let us decrease the modulus to less than 1/10. We also find that this reduced stiffness model shows about a half the observed vertical translational frequency so as to suggest a lack of consistency with continuum modelling of basically discontinuous masonry structure.