Face recognition has attracted much attention from researchers for past decades. Recently, with the development of deep learning, a deep neural network is adopted by face recognition system and better performance is obtained. Many works on metric learning have been done in the deep neural network. Meanwhile, there are several variation problems existing in face recognition, such as profile face image, low-resolution face image, different age of face image, face image wearing eyeglass, etc. In this paper, targeting at different kinds of variation problems, we proposed a novel network structure, called Extended Siamese Network. Another contribution is that a new loss function is proposed, to further take inter-class information into account based on the center loss function. The experiments show that recognition accuracy is improved in comparison with the other state-of-Art methods.