Adaptive Focused Website Segment Crawler

Tanaphol Suebchua, Arnon Rungsawang, Hayato Yamana

    Research output: Chapter in Book/Report/Conference proceedingConference contribution

    5 Citations (Scopus)


    Focused web crawler has become indispensable for vertical search engines that provide a search service for specialized datasets. These vertical search engines have to collect specific web pages in the web space, whereas search engines such as Google and Bing gather web pages from all over the world. The problem in focused crawling research is how to collect specific web pages with minimal computing resources. We previously addressed this problem by proposing a focused crawling strategy, which utilizes an ensemble machine learning classifier to find the group of relevant web pages, referred to as relevant website segment. In this paper, we enhance the proposed crawler as follows: 1) We increase the accuracy of predicting website segments, by preparing two predictors: a predictor learned by features extracted from relevant source website segments and another predictor learned by features from irrelevant ones. The idea is that there may exist different characteristics between these two types of source website segments. 2) We also propose a noisy data elimination method when updating the predictor incrementally during the crawling process. A preliminary experiment shows that our enhanced crawler outperforms a crawler that equips neither of these approaches by around 12%, at most.

    Original languageEnglish
    Title of host publicationNBiS 2016 - 19th International Conference on Network-Based Information Systems
    PublisherInstitute of Electrical and Electronics Engineers Inc.
    Number of pages7
    ISBN (Electronic)9781509009794
    Publication statusPublished - 2016 Dec 16
    Event19th International Conference on Network-Based Information Systems, NBiS 2016 - Ostrava, Czech Republic
    Duration: 2016 Sep 72016 Sep 9


    Other19th International Conference on Network-Based Information Systems, NBiS 2016
    Country/TerritoryCzech Republic


    • classifier ensemble
    • focused crawler
    • machine learning
    • noise reduction
    • topic specific web crawler
    • website segment crawler

    ASJC Scopus subject areas

    • Information Systems
    • Computer Networks and Communications


    Dive into the research topics of 'Adaptive Focused Website Segment Crawler'. Together they form a unique fingerprint.

    Cite this