In this work, two novel local feature patterns-Modified Local Binary patterns (MLBP) and local Ternary patterns (LIP), are proposed for extract features in the facial image, which use some distinct rule to code the values in a label, respectively. These patterns are more invariant to illuminance and face expression compared to traditional one. After getting the local feature patterns, in order to take alignment of face into account, a novel matching method called Histogram Spatially constrained Earth Mover's Distance(HSEMD) is proposed. In this step, the source image is partitioned into non-overlapping local regions while the destination image is represented as a set of overlapping local regions at different positions. Meanwhile, multi-scale cascade mechanism is studied for extracting more feature patterns and obtaining global information of the face.The performance of the proposed method is assessed in the face recognition problem under different challenges. The experimental results show that the proposed method has higher accuracy than some other classic methods.