With more and more people suffer from lung cancer, computer-aided diagnosis plays a more and more important role in lung cancer diagnosis. CNN has achieved state-of-the-art performance in image processing, and Mask R-CNN outperforms most other methods on instance segmentation. However, the target is extraordinarily small, and the background is very large in images, which results in a large number of negative examples and most of them are easy negatives. They will contribute a large part of the loss value in smooth loss function. The class imbalance problem leads to inefficient training, which makes model degenerated. In this paper, we propose a method based on Mask R-CNN to segment lung nodules. Due to the non-uniformity of CT values, we use the Laplacian operator to do feature dimensionality reduction for filtering out part of the noise. In our model, the novel function Focal Loss is used to suppress well-classified examples. The model is tested on LIDC-IDRI dataset and the results showed that the average precision of lung nodules reaches 78%. Compared with the smooth loss function in Mask R-CNN it improves by 7%.