Google has a realtime feed of Twitter

#1 2009-07-08 by Tim Mahoney

Very cool. Considering that the URLs people tweet are usually bit.ly or tr.im urls, I wonder if Google follows those URLs through to their end result...

#2 2009-07-08 by Johann

Tim,

I tested it and the only link that was crawled by Google was the one shortened through ow.ly so they probably have tinyurl, bit.ly, cli.gs and is.gd on their blacklist.

#3 2009-07-08 by Johann

Correction: 5 minutes later, the cli.gs link was also crawled.

#4 2009-07-09 by York

Johann,

Google crawls everything, even banned sites. Sometimes tinyurl links have been spotted in the old version of the Google Webmastertools, so Google must know the target of those links. Currently, the tools seem to list only links that do not go through a redirect. tinyurl does not have a robots.txt file to disallow crawling. I don't know about the other URL shortening services.

The nofollow attribute has been introduced not to prevent crawling, the name is misleading. I think nofollow links will result in a crawl when disovered by Googlebot, just like ordinary links.

#5 2009-07-10 by Johann

York,

of course nofollow does not necessarily prevent discovery, but I found it strange that Google has apparently blacklisted some URL shortening services.