Development of a scalable search engine using open source softwares

Ryota Hayasaka, Takahiro Hayashi, Rikio Onai

Research output: Contribution to journalArticle

Abstract

Web search APIs such as Google API and Yahoo API are often used for developing Web applications. However, the number of Web pages obtained by using a Web API is limited and the full texts of these Web pages cannot be directly accessed by the APIs. These are serious problems for the developers. In order to solve the problems, we have developed an original search engine which are composed of open source softwares such as Heritrix, Apache Lucene and MySQL. Partitioning database and index, we have developed the search engine which has scalability to the sizes of database and index. We have confirmed that the speed performance of the developed search engine is not decreased by partitioning database and index.

Original languageEnglish
Pages (from-to)138-156
Number of pages19
JournalComputer Software
Volume26
Issue number4
Publication statusPublished - 2009
Externally publishedYes

    Fingerprint

ASJC Scopus subject areas

  • Software

Cite this

Hayasaka, R., Hayashi, T., & Onai, R. (2009). Development of a scalable search engine using open source softwares. Computer Software, 26(4), 138-156.