This paper studies the patterns of cultural differences observed in pictogram interpretation. We conducted a 14-month online survey in the U.S. and Japan to ask the meaning of 120 pictograms used in a pictogram communication system. A total of 935 respondents in the U.S. and 543 respondents in Japan participated in the survey to submit pictogram interpretations which added up to compose an average of 147 English interpretations and 97 Japanese interpretations per pictogram. Three human judges independently analyzed the English-Japanese pictogram interpretation words, and as a result, 19 pictograms were found to have culturally different interpretations by two or more judges. The following patterns of cultural differences in pictogram interpretation were observed: (1) two cultures share the same underlying concept, but have different perspectives on the concept, (2) two cultures only partially share the same underlying concept, and (3) two cultures do not share any common underlying concept.