Nowadays, it has become convenient to record data related to an individual using a wearable device. However, it is difficult to utilize the data according to the individual, especially to anomaly detection. Anomaly detection is very important for healthcare, e.g., early detecting of illness. In our previous study, we proposed an approach to specifying latent factors using Structural Equation Modeling (SEM). In this paper, we propose an improved approach for anomaly detection taking account of personal status based on latent factors. To estimate the states, we adopt Hidden Markov Model (HMM). Moreover, we use Hotelling's theory to detect abnormal data statistically. By using our approach, even if states can not be explicitly obtained from a device, hidden states can be estimated to perform anomaly detection in more details.