Authorship Attribution (AA) is a fundamental branch of text classification, aiming at identifying the authors of given texts. However, authorship attribution of short texts faces many challenges like short text, feature sparsity and non-standardization of casual words. Recent studies have shown that deep learning methods can greatly improve the accuracy of AA tasks, however they still represent user posts using a set of predefined features (e.g., word n-grams and character n-grams) and adopt text classification methods to solve this task. In this paper, we propose a hybrid model to solve author attribution of short texts. The first part is a pretrained language model based on RoBERTa to produce post representations that are aware of tweet-related stylistic features and their contextualities. The second part is a CNN model built on a number of feature embeddings to represent users' writing styles. Finally, we assemble these representations for final AA classification. Our experimental results show that our model on tweets shows the state-of-the-art result on a known tweet AA dataset.