The latest video compression standard Versatile Video Coding (VVC/H.266) adopts many advanced encoding tools which achieve larger BD-rate reduction but bring much time consumption. Among all new tools, an inter-prediction mode called InterIMV mode is most time consuming for occupying 12.7% of entire encoding time, with 96.8% of its check redundant in depth level and coding unit (CU) level. Current depth level mode skip methods do early termination based on original top-down structure and skip mode check in lower depth, which brings large quality decrease. For CU level, the common features are not effective to correctly skip InterIMV check. Thus, fast InterIMV algorithm targeted to remove InterIMV mode redundancy is needed. This paper proposes bottom-up InterIMV check method on depth level and temporal rate-distortion relation based check skip method on CU level. Firstly, proposed method executes InterIMV mode in a bottom-up order after all depths are traversed. This method checks InterIMV mode from the lowest depth towards upper depths and finds the suitable depth for current area. Then, InterIMV check in depths above suitable depth is removed without bringing much quality damage, since only the suitable depth is used to encode current area. Secondly, for each CU, the proportion of rate in rate-distortion cost (RD-cost) of temporal best mode is used to judge whether to skip InterIMV check, since this relationship is strongly associated with InterIMV selection. While testing on VVC benchmark VTM5 under RandomAccess configuration, the proposed methods achieve 79.78% speed up in InterIMV process on average. The BD-rate increase of proposed methods is only 0.37% which brings no significant coding efficiency decrease.