With the increasing demand of high video quality and large image size, adaptive interpolation filter (AIF) addresses these issues and conquers the time varying effects resulting in increased coding efficiency, comparing with recent H.264 standard. However, currently most AIF algorithms are based on either frame level or macroblock (MB) level, which are not flexible enough for different video contents in a real codec system. And most of them are facing a severe time consuming problem. This paper proposes a content based coarse to fine AIF algorithm, which can adapt to video contents by adding different filters and conditions from coarse to fine. The overall algorithm has been mainly made up by 3 schemes: frequency analysis based frame level skip interpolation, motion vector modeling based region level interpolation, and edge detection based macroblock level interpolation. The experimental results show that the proposed algorithm is able to reduce total encoding time about 41% for 720p and 25% for 1080p sequences averagely, comparing with key technology areas (KTA) Enhanced AIF algorithm, while obtains a BD-PSNR gain up to 0.004 and 3.122 BDBR reduction.