Development of a large-scale web crawler and search engine infrastructure

Susumu Akamine; Yoshikiyo Kato; Daisuke Kawahara; Keiji Shinzato; Kentaro Inui; Sadao Kurohashi; Yutaka Kidawara

doi:10.1145/1667780.1667805

Development of a large-scale web crawler and search engine infrastructure

Susumu Akamine^*, Yoshikiyo Kato, Daisuke Kawahara, Keiji Shinzato, Kentaro Inui, Sadao Kurohashi, Yutaka Kidawara

^*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

7 Citations (Scopus)

Abstract

This paper reports the ongoing development of a large-scale Web crawler and search engine infrastructure at National Institute of Information and Communications Technology. This infrastructure has the following characteristics: (1) It collects one billion Japanese Web pages while keeping them up-to-date. (2) It selects 100 million pages from among the collected pages and converts them into a standard data format to store the results of morphological analysis, dependency parsing, and synonym augmentation. (3) The selected set of pages is searchable and accessible to the users. (4) The scalability of the system is achieved by using a large-scale cluster machine for distributed data processing.

Original language	English
Title of host publication	Proceedings of the 3rd International Universal Communication Symposium, IUCS 2009
Pages	126-131
Number of pages	6
DOIs	https://doi.org/10.1145/1667780.1667805
Publication status	Published - 2009
Externally published	Yes
Event	3rd International Universal Communication Symposium, IUCS 2009 - Tokyo, Japan Duration: 2009 Dec 3 → 2009 Dec 4

Publication series

Name	ACM International Conference Proceeding Series

Conference

Conference	3rd International Universal Communication Symposium, IUCS 2009
Country/Territory	Japan
City	Tokyo
Period	09/12/3 → 09/12/4

Keywords

Crawler
Search engine
Web information analysis

ASJC Scopus subject areas

Software
Human-Computer Interaction
Computer Vision and Pattern Recognition
Computer Networks and Communications

Access to Document

10.1145/1667780.1667805

Cite this

Development of a large-scale web crawler and search engine infrastructure. / Akamine, Susumu; Kato, Yoshikiyo; Kawahara, Daisuke et al.

Proceedings of the 3rd International Universal Communication Symposium, IUCS 2009. 2009. p. 126-131 (ACM International Conference Proceeding Series).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Akamine, S, Kato, Y, Kawahara, D, Shinzato, K, Inui, K, Kurohashi, S & Kidawara, Y 2009, Development of a large-scale web crawler and search engine infrastructure. in Proceedings of the 3rd International Universal Communication Symposium, IUCS 2009. ACM International Conference Proceeding Series, pp. 126-131, 3rd International Universal Communication Symposium, IUCS 2009, Tokyo, Japan, 09/12/3. https://doi.org/10.1145/1667780.1667805

@inproceedings{b1603328b05449d3a1c27731a313980d,

title = "Development of a large-scale web crawler and search engine infrastructure",

abstract = "This paper reports the ongoing development of a large-scale Web crawler and search engine infrastructure at National Institute of Information and Communications Technology. This infrastructure has the following characteristics: (1) It collects one billion Japanese Web pages while keeping them up-to-date. (2) It selects 100 million pages from among the collected pages and converts them into a standard data format to store the results of morphological analysis, dependency parsing, and synonym augmentation. (3) The selected set of pages is searchable and accessible to the users. (4) The scalability of the system is achieved by using a large-scale cluster machine for distributed data processing.",

keywords = "Crawler, Search engine, Web information analysis",

author = "Susumu Akamine and Yoshikiyo Kato and Daisuke Kawahara and Keiji Shinzato and Kentaro Inui and Sadao Kurohashi and Yutaka Kidawara",

year = "2009",

doi = "10.1145/1667780.1667805",

language = "English",

isbn = "9781605586410",

series = "ACM International Conference Proceeding Series",

pages = "126--131",

booktitle = "Proceedings of the 3rd International Universal Communication Symposium, IUCS 2009",

note = "3rd International Universal Communication Symposium, IUCS 2009 ; Conference date: 03-12-2009 Through 04-12-2009",

}

TY - GEN

T1 - Development of a large-scale web crawler and search engine infrastructure

AU - Akamine, Susumu

AU - Kato, Yoshikiyo

AU - Kawahara, Daisuke

AU - Shinzato, Keiji

AU - Inui, Kentaro

AU - Kurohashi, Sadao

AU - Kidawara, Yutaka

PY - 2009

Y1 - 2009

N2 - This paper reports the ongoing development of a large-scale Web crawler and search engine infrastructure at National Institute of Information and Communications Technology. This infrastructure has the following characteristics: (1) It collects one billion Japanese Web pages while keeping them up-to-date. (2) It selects 100 million pages from among the collected pages and converts them into a standard data format to store the results of morphological analysis, dependency parsing, and synonym augmentation. (3) The selected set of pages is searchable and accessible to the users. (4) The scalability of the system is achieved by using a large-scale cluster machine for distributed data processing.

AB - This paper reports the ongoing development of a large-scale Web crawler and search engine infrastructure at National Institute of Information and Communications Technology. This infrastructure has the following characteristics: (1) It collects one billion Japanese Web pages while keeping them up-to-date. (2) It selects 100 million pages from among the collected pages and converts them into a standard data format to store the results of morphological analysis, dependency parsing, and synonym augmentation. (3) The selected set of pages is searchable and accessible to the users. (4) The scalability of the system is achieved by using a large-scale cluster machine for distributed data processing.

KW - Crawler

KW - Search engine

KW - Web information analysis

UR - http://www.scopus.com/inward/record.url?scp=76749136736&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=76749136736&partnerID=8YFLogxK

U2 - 10.1145/1667780.1667805

DO - 10.1145/1667780.1667805

M3 - Conference contribution

AN - SCOPUS:76749136736

SN - 9781605586410

T3 - ACM International Conference Proceeding Series

SP - 126

EP - 131

BT - Proceedings of the 3rd International Universal Communication Symposium, IUCS 2009

T2 - 3rd International Universal Communication Symposium, IUCS 2009

Y2 - 3 December 2009 through 4 December 2009

ER -

Development of a large-scale web crawler and search engine infrastructure

Abstract

Publication series

Conference

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this