SIFT is one of the most robust feature extraction algorithms and widely used in object tracking field. While invariant to scale, rotation and other image transforms, the traditional SIFT algorithm is rather time-consuming in creating description of the large number of keypoints. Aiming at reducing tracking time and complexity to achieve real time compliance, this paper proposes Homography Matrix based SIFT for fast and robust object tracking. By introducing Homography Matrix and Concise Keypoint Selecting (CKS) scheme, search region area can be greatly decreased with less redundant keypoints but still high tracking precision especially for fast-moving object. Experiment results by the proposed methods show that the time spent on tracking a VGA video sequence can be reduced for up to 92.8% and the frame rate can achieve 5 fps while the spatial deviation of tracking could be kept within 1.9 pixels in average.