Large coding unit which is also known as super macroblock, has already been adopted in the test model of next generation coding standard called high efficiency video coding. The coding unit which is larger than 16x16 and less than or equal to 64x64 provides great bit rate saving while the coding complexity increases dramatically. In this paper, we propose a fast coding unit decision algorithm in either frame level or coding unit level to accelerate encoding procedure. In frame level, by analyzing the utilization rate of coding unit in all depth, we skip several rarely used coding units in specified depth. In coding unit level, the neighbor and co-located coding unit information are referred for further skipping coding unit in unnecessary depth. Results show our proposed algorithm provides averagely 45% total encoding time reduction and negligible drop of rate distortion performance.