Color clustering is widely used for color image segmentation, transmission and visualization. However, traditional clustering algorithms have difficulty in adaptive determination of the proper cluster number. In this paper, a variable structure self-organizing map neural network (VSSOM) is proposed to achieve adaptive color clustering. It can provide the suitable cluster number and the palette for generating the clustered color image. The characteristic of the algorithm is that the output neuron of VSSOM is changed from 2 to N based on the defined learning rules during training. When the network converges, its N is the number of color clusters and the palette of clustered image is corresponding to the weights of the network. Experimental results show that the proposed algorithm has the desired ability for color clustering.