Rate of change on the Web
[Cho00] 720K pages from 270 popular sites sampled daily from Feb 17 -
Jun 14, 1999
-
Any changes: 40% weekly, 23% daily
[Fett02(04)] Massive study 151M pages checked over few months
-
Significant changed - 7% weekly
-
Small changes - 25% weekly
[Ntul04] 154 large sites re-crawled from scratch weekly
-
8% new pages/week
-
8% die
-
5% new content
-
25% new links/week
Cho J, Garcia-Molina H. 2000. The evolution of the Web and
implications for an incremental crawler.
Proc. 26th
Int. Conf. on Very Large Databases. Morgan Kaufmann, 200-209.
Ntoulas, A., Cho, J., and Olston, C. 2004. What's new on the web?: the
evolution of the web from a search engine perspective.
Proc. 13th Int. Conf. on World Wide Web, ACM, New York, 1-12.