Search engines and Web dynamics

被引:39
|
作者
Risvik, KM [1 ]
Michelsen, R [1 ]
机构
[1] Fast Search & Transfer ASA, NO-0120 Oslo, Norway
关键词
dynamic information retrieval; indexing; document crawling; scalable architecture; algorithms; scheduling;
D O I
10.1016/S1389-1286(02)00213-X
中图分类号
TP3 [计算技术、计算机技术];
学科分类号
0812 ;
摘要
In this paper we study several dimensions of Web dynamics in the context of large-scale Internet search engines. Both growth and update dynamics clearly represent big challenges for search engines. We show how the problems arise in all components of a reference search engine model. Furthermore, we use the FAST Search Engine architecture as a case study for showing some possible solutions for Web dynamics and search engines. The focus is to demonstrate solutions that work in practice for real systems. The service is running live at www.alltheweb.com and major portals worldwide with more than 30 million queries a day, about 700 million full-text documents, a crawl base of 1.8 billion documents, updated every I I days, at a rate of 400 documents/second. We discuss future evolution of the Web, and some important issues for search engines will be scheduling and query execution as well as increasingly heterogeneous architectures to handle the dynamic Web. (C) 2002 Elsevier Science Ltd. All rights reserved.
引用
收藏
页码:289 / 302
页数:14
相关论文
共 50 条
  • [41] Why People Search for Images using Web Search Engines
    Xie, Xiaohui
    Liu, Yiqun
    de Rijke, Maarten
    He, Jiyin
    Zhang, Min
    Ma, Shaoping
    WSDM'18: PROCEEDINGS OF THE ELEVENTH ACM INTERNATIONAL CONFERENCE ON WEB SEARCH AND DATA MINING, 2018, : 655 - 663
  • [42] Overlap among major web search engines
    Spink, Amanda
    Jansen, Bernard J.
    Kathuria, Vinish
    Koshman, Sherry
    INTERNET RESEARCH, 2006, 16 (04) : 419 - 426
  • [43] Fast and Flexible Compression for Web Search Engines
    Farina, Antonio
    Brisaboa, Nieves R.
    Paris, Cristina
    Parama, Jose R.
    ELECTRONIC NOTES IN THEORETICAL COMPUTER SCIENCE, 2006, 142 : 129 - 141
  • [44] Overlap among major Web search engines
    Spink, Amanda
    Jansen, Bernard J.
    Blakely, Chris
    Koshman, Sherry
    THIRD INTERNATIONAL CONFERENCE ON INFORMATION TECHNOLOGY: NEW GENERATIONS, PROCEEDINGS, 2006, : 370 - +
  • [45] A categorization scheme for semantic web search engines
    Esmaili, Kyumars Sheykh
    Abolhassani, Hassan
    2006 IEEE INTERNATIONAL CONFERENCE ON COMPUTER SYSTEMS AND APPLICATIONS, VOLS 1-3, 2006, : 171 - +
  • [46] Optimizing the Number of Robots for Web Search Engines
    J. Talim
    Z. Liu
    P. Nain
    E.G. Coffman
    Telecommunication Systems, 2001, 17 : 243 - 264
  • [47] Project Whistlestop: An evaluation of search engines on the Web
    Kochtanek, T
    Laffey, J
    Ervin, J
    Tunender, H
    Borwick, J
    19TH ANNUAL NATIONAL ONLINE MEETING, PROCEEDINGS, 1998, : 211 - 221
  • [48] Project Whistlestop: An evaluation of search engines on the web
    Kochtanek, T
    Laffey, J
    Ervin, J
    Tunender, H
    Borwick, J
    19TH ANNUAL NATIONAL ONLINE MEETING, PROCEEDINGS-1998, 1998, : 211 - 221
  • [49] Multimedia Chinese web search engines: A survey
    Chang, Yun-Ke
    Arroyo, Miguel Angel Morales
    Spink, Amanda
    INTERNATIONAL CONFERENCE ON INFORMATION TECHNOLOGY, PROCEEDINGS, 2007, : 481 - +
  • [50] Snippet Generation for Semantic Web Search Engines
    Penin, Thomas
    Wang, Haofen
    Tran, Thanh
    Yu, Yong
    SEMANTIC WEB, PROCEEDINGS, 2008, 5367 : 493 - +