Clustering a given set of data is crucial in many fields including image processing. It plays important roles in image segmentation and object detection for example. This paper proposes a framework of building a similarity matrix for a given dataset, which is then used for clustering the dataset. The similarity between two points are defined based on how other points distribute around the line connecting the two points. It can capture the degree of how the two points are placed on the same line. The similarity matrix is considered as a kernel matrix of the given dataset, and based on it, the spectral clustering is performed. Clustering with the proposed similarity matrix is shown to perform well through experiments using an artificially designed problem and a real-world problem of detecting lines from a distorted image.