Rate of change on the Web

[Cho00] 720K pages from 270 popular sites sampled daily from Feb 17 - Jun 14, 1999

• Any changes: 40% weekly, 23% daily

[Fett02(04)] Massive study 151M pages checked over few months

• Significant changed - 7% weekly
• Small changes - 25% weekly

[Ntul04] 154 large sites re-crawled from scratch weekly

• 8% new pages/week
• 8% die
• 5% new content
• 25% new links/week

Cho J, Garcia-Molina H. 2000. The evolution of the Web and implications for an incremental crawler.
    Proc. 26th Int. Conf. on Very Large Databases. Morgan Kaufmann, 200-209.
Ntoulas, A., Cho, J., and Olston, C. 2004. What's new on the web?: the evolution of the web from a search engine perspective.
    Proc. 13th Int. Conf. on World Wide Web, ACM, New York, 1-12.