Variable block size motion estimation (VBSME) is the most computation consuming part in the newest H.264/AVC video coding standard. To reduce computation, an ultra low-complexity fast VBSME algorithm is proposed in this paper. Different to previous fast algorithms which separately pressed the 7 block modes, in the presented algorithm, all the block modes are simultaneously calculated for most of the search points, and thus the SADs can be reused by all blocks. Moreover, a low-pass filter based pixel decimation is also introduced to decrease the matching cost. Experiments show that the proposed fast VBSME algorithm can reduce the computation to about 0.2% of fast full search (FFS) with robust image quality.