In this study, we proposed a method which could be used for mapping people's emotion state on the two-dimensional arousal-valence model of effect. The final target of our research is to apply this kind of emotional recognition system to robots or some assistant apparatus which service activities of daily living (ADL). Since in our previous studies, we have finished the work of recognizing people's emotion state on the dimension of arousal by evaluating subjects' heartbeat and LF/HF, which is calculated from the frequency domain analysis of HRV, as the second step's work, we focused on how to recognize people's emotion state on valence dimension. To be specific, we used some kinds of normative affective stimuluses to elicit subjects' emotional change, then collected multiple physiological data during this emotional stimulation process. Finally, as for data analyzing, we didn't use the supervised learning method, like SVM, but made a new attempt to apply the unsupervised clustering method to sample data, dividing the data set into several natural clusters by analyzing the physiological features we abstracted. The calculated results of our experiment have verified the feasibility of mapping human's emotion state on the two-dimensional arousal-valence model of effect at a quadrant level.