Development of a scalable search engine using open source softwares

Ryota Hayasaka*, Takahiro Hayashi, Rikio Onai

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review


Web search APIs such as Google API and Yahoo API are often used for developing Web applications. However, the number of Web pages obtained by using a Web API is limited and the full texts of these Web pages cannot be directly accessed by the APIs. These are serious problems for the developers. In order to solve the problems, we have developed an original search engine which are composed of open source softwares such as Heritrix, Apache Lucene and MySQL. Partitioning database and index, we have developed the search engine which has scalability to the sizes of database and index. We have confirmed that the speed performance of the developed search engine is not decreased by partitioning database and index.

Original languageEnglish
Pages (from-to)138-156
Number of pages19
JournalComputer Software
Issue number4
Publication statusPublished - 2009
Externally publishedYes

ASJC Scopus subject areas

  • Software


Dive into the research topics of 'Development of a scalable search engine using open source softwares'. Together they form a unique fingerprint.

Cite this