The strategic use of indirect language is crucial in business negotiations, doctor-patient interactions, instructional discourse, and multiple other contextss. Being indirect allows interlocutors to diminish the potential threats to their interlocutors’ desired self-image—or, face threat—that may arise by being overtly direct. Handling indirectness well is important for spoken dialogue systems, as being either too indirect or too direct at the wrong time could harm the agent-user relationship. We take the first step towards handling users’ indirection by exploring different supervised machine learning approaches for the task of automatically detecting indirectness in conversations. Accurate automated detection of indirectness may help conversational agents better understand their users’ intents, gauge the current relationship with the user in order to appropriately plan a response, and inform the strategic use of indirectness to manage the task goals and social goals of the interaction. To our knowledge we are the first to use a multi-modal approach to detecting indirect language: we rely on both verbal and nonverbal features of the interaction. Our best model acheives a 62% F1 score on our dataset, outperforming non-neural baselines including approaches used by past work for related tasks such as uncertainty detection.