User-agent: * Crawl-delay: 10 # Custom Disallow: /competition-winners/photo/2013 Disallow: /competition-winners/photo/2012 Disallow: /competition-winners/photo/2011 Disallow: /competition-winners/video/2013 Disallow: /competition-winners/video/2012 Disallow: /competition-winners/video/2011 # Don't index PDFs: Disallow: /*.pdf$ Sitemap: https://www.volunteerhq.org/sitemap.xml