Development of a scalable search engine using open source softwares

Web search APIs such as Google API and Yahoo API are often used for developing Web applications. However, the number of Web pages obtained by using a Web API is limited and the full texts of these Web pages cannot be directly accessed by the APIs. These are serious problems for the developers. In order to solve the problems, we have developed an original search engine which are composed of open source softwares such as Heritrix, Apache Lucene and MySQL. Partitioning database and index, we have developed the search engine which has scalability to the sizes of database and index. We have confirmed that the speed performance of the developed search engine is not decreased by partitioning database and index.