With the rapid growth of Web services, recommending suitable services to users has become a big challenge. The existing service recommendation works by Quality of Service (QoS) prediction fail to fully consider the influence of situation information, such as time, location, and user relations thoroughly. Two issues must be resolved to consider situation information: issue one, rating scarcity, is that there are less data to learn when importing more situations; issue two is that an effective approach is needed to adapt many situational factors. Our solution is a two-phase method: first, to overcome rating scarcity, data is augmented with estimations of unknown QoS values by learning from observable factors. The augmented data is then used to learn the important latent factors associated with the situational factors for QoS prediction. Experiments on data of real service invocations in different situations show improvement of our method in terms of QoS prediction accuracy over several existing methods, especially in the severe rating scarcity condition. In addition, analysis on parameter selection of proposed method can further assist in obtaining better QoS prediction in practical use.