High Efficiency Video Coding (HEVC), a successor to H.264, is the next generation video compression standard. To enhance the coding efficiency of video frames, 35 intra prediction modes adopted in Prediction Unit (PU) from 44 to 6464 of HEVC. However the improvement is based on the cost of rapid increased complexity. This paper proposes a fast mode and depth decision algorithm based on edge detection and reconfiguration to alleviate the large computation complexity in intra prediction with the trivial degradation in the accuracy. In mode decision, pixel gradient statistics (PGS) and mode refinement (MR) is proposed. PGS use pixel gradient information to assist prediction mode selection after Rough Mode Decision (RMD). MR utilizes the neighboring mode information to select most probable mode (MPM). In depth decision, unit partitioning reconfiguration (UPR) is proposed. It exchanged the original partitioning order in a more reasonable structure, utilizes detecting the smoothness of coding unit to decide the prediction depth. Smoothness detecting is based on PGS result. Experiment result shows that the proposed method performs about 42.84% time saving with little degradation (BD-rate increase 0.61% and BDPSNR reduce 0.041db) in the coding gain. Comparing with previous work, it achieves more time saving and lesser efficiency loss.