This study examines whether mass or potential disorder is more effective at reducing the thermal conductivity and phonon properties in group IV alloys using molecular dynamics (MD) simulation. In this investigation, virtual atoms were used so that the atomic mass and the interatomic potential could be varied independently. The effects of potential disorder on the thermal conductivity and phonon properties are smaller than those associated with mass disorder. The suppression of thermal conductivity by alloying is attributed only to the effects of mass disorder. In the phonon lifetime calculation, the lifetime of the phonon determined by the scattering due to potential disorder is ten times longer than that due to mass disorder. This difference is so large that the contribution of potential disorder to phonon lifetime is effectively negligible. Thus, a binary interatomic potential is unnecessary to accurately estimate the thermal conductivity of the group IV alloys.