The building plane is 7.75 m x 8.35 m and the height is 47.8 m. As the result, it became difficult to control the building response for not only seismic loading but also wind loading. So, we planed to change the vibration mode from the rocking-mode to the sway-mode by softening the translation stiffness at the first story, and to make the first story into the energy-absorbing story by concentrating the building deformation. We call this system the soft first story structure. The system consists of the following three items; (a) to stiffen the structure above the second story, (b) to make columns of the first story flexible laterally and rigid axially, (c) to develop the hardening-type oil-damper. In this paper, it is reported about details of these structural elements and the dynamic performance of the proposed response control system against wind and seismic loadings throughout the time history response analysis.