College students studying foreign languages have the capability to learn vocabulary and grammar and can practice listening, reading, and writing skills by themselves, but they often lack opportunities for speaking and conversation. Robot-assisted language learning (RALL) provides a solution. In this paper, we present an exploratory study on using humanoid robots as tutors to automatically conduct dialogue-based tests with freshman. In the test, students examine pictures, listen to questions asked by the robot, and then say an answer. The test has three parts: answer freely, choose the correct answer by speaking the number, and read the sentence aloud, listen to some options, and choose the best sentence. After each answer, through speech to text (STT) technology, the students’ responses will display on a screen immediately so that the students can ascertain whether their pronunciation or speech output is correct. In this paper, we describe the test design and setup and conduct a preliminary experiment. Finally, we provide an analysis from observations of video data and from students’ comments and discussion. We also identify some necessary improvements for the robot, such as improving its sensitivity to the foreign language learner’s voice.