Elliptical object detection in color images is an important step of many image recognition systems. In this paper, we present a novel approach to accurately detect elliptical objects in color images. The proposed method has an important feature of integrating color image edge detection result with a novel ellipse parameter estimation algorithm efficiently. First, we utilize an elaborate edge detection algorithm to extract image edge and obtain precise gradient direction of each edge pixel. Second we carefully select valid edge pixel pairs and an individual edge pixel to accumulate ellipses from the edge map by estimating five parameters of an arbitrary ellipse. The gradient direction information and geometric analysis are intelligently integrated into our algorithm to reduce computation complexity and avoid detecting false ellipses. Experimental results demonstrate that our algorithm is robust and efficient in locating multiple elliptical objects simultaneously with respect to different size, orientation and color.