In this report, a new framework is proposed for inferring the user's personality traits based on their habitual behaviors during face-to-face human-robot interactions, aiming to improve the quality of human-robot interactions. The proposed framework enables the robot to extract the person's visual features such as gaze, head and body motion, and vocal features such as pitch, energy, and Mel-Frequency Cepstral Coefficient (MFCC) during the conversation that is lead by Robot posing a series of questions to each participant. The participants are expected to answer each of the questions with their habitual behaviors. Each participant's personality traits can be assessed with a questionnaire. Then, all data will be used to train the regression or classification model for inferring the user's personality traits.