Google Crawls RSS/Atom Feeds to Discover New URLs Faster!
October 30, 2009
Google has launched a new feature that uses RSS and Atom feeds to discover new web pages quicker. These features will allow Google to process and index web pages much faster than traditional methods and allow new content to be displayed on search results as soon as it goes live.
Google may use several potential sources to access feed updates, including direct crawls of feeds, notification services, or Reader.
In order for Google to use RSS/Atom feeds to discover & index your content, it is important that your robots.txt files allow the feeds to be crawled. You can test if the Googlebot can crawl your feeds by testing your feed URLs with robots.txt tester in Google Webmaster Tools.
Source: http://blog.searchenginewatch.com/091030-021320
