Yahoo! bots list – all user agents

Posted 2008-01-12 in WWW by Johann.

Yahoo! has a huge number of bots. In this bot list, I’ll try to list all and explain shortly what they do.

  • Yahoo-MMCrawler/3.x (mms dash mmcrawler dash support at yahoo dash inc dot com)
  • Mozilla/5.0 (Yahoo-MMCrawler/4.0; mailto:vertical-crawl-support@yahoo-inc.com)

Image or multimedia crawlers.

YahooFeedSeeker/2.0 (compatible; Mozilla 4.0; MSIE 5.5; http://publisher.yahoo.com/rssguide; users …; views …)

News feed crawler.

Mozilla/5.0 (compatible; Yahoo! Slurp China; http://misc.yahoo.com.cn/help.html)

Crawler for the Chinese Yahoo.

Mozilla/5.0 (compatible; Yahoo! DE Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Crawler for the German Yahoo.

  • Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; http://help.yahoo.com/help/us/ysearch/slurp)
  • Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

The well known Slurp crawler, probably the most active legit crawler. I don’t see the first one a lot.

  • Nokia6682/2.0 (3.01.1) SymbianOS/8.0 Series60/2.6 Profile/MIDP-2.0 configuration/CLDC-1.1 UP.Link/6.3.0.0.0 (compatible;YahooSeeker/M1A1-R2D2; http://help.yahoo.com/help/us/ysearch/crawling/crawling-01.html)
  • LG-C1500 UP.Browser/6.2.3 (GUI) MMP/1.0 (compatible;YahooSeeker/M1A1-R2D2; http://help.yahoo.com/help/us/ysearch/crawling/crawling-01.html)
  • MOT-V975/81.33.02I MIB/2.2.1 Profile/MIDP-2.0 Configuration/CLDC-1.1 (compatible;YahooSeeker/M1A1-R2D2; http://help.yahoo.com/help/us/ysearch/crawling/crawling-01.html)
  • Nokia6682/2.0 (3.01.1) SymbianOS/8.0 Series60/2.6 Profile/MIDP-2.0 configuration/CLDC-1.1 UP.Link/6.3.0.0.0 (compatible;YahooSeeker/M1A1-R2D2; http://help.yahoo.com/help/us/ysearch/crawling/crawling-01.html)
  • SGH-Z130 SHP/VPP/R5 SMB3.1 SMM-MMS/1.1.0 profile/MIDP-2.0 configuration/CLDC-1.0 (compatible;YahooSeeker/M1A1-R2D2; http://help.yahoo.com/help/us/ysearch/crawling/crawling-01.html)
  • Nokia6610/1.0 (3.09) Profile/MIDP-1.0 Configuration/CLDC-1.0 (compatible;YahooSeeker/M1A1-R2D2; http://help.yahoo.com/help/us/ysearch/crawling/crawling-01.html)
  • YahooSeeker/M1A1-R2D2

These user agent strings belong to Yahoo!’s mobile web index crawler.

  • Vodafone/1.0/V705SH (compatible; Y!J-SRD/1.0; http://help.yahoo.co.jp/help/jp/search/indexing/indexing-27.html)
  • DoCoMo/2.0 SH902i (compatible; Y!J-SRD/1.0; http://help.yahoo.co.jp/help/jp/search/indexing/indexing-27.html)
  • KDDI-CA33 UP.Browser/6.2.0.10.4 (compatible; Y!J-SRD/1.0; http://help.yahoo.co.jp/help/jp/search/indexing/indexing-27.html)

Some more mobile web crawlers, probably specific to the Japanese Yahoo.

Y!J-BSC/1.0 (http://help.yahoo.co.jp/help/jp/blog-search/)

A blog spider.

Yahoo! Slurp/Site Explorer

Bot that verifies site authentication through Yahoo! Site Explorer.

4 comments

IFrame support on mobile devices

Posted 2007-12-28 in Mobile Web by Johann.

How viable are IFrame for use on mobile devices? IFrames could come in handy to add specific content to more capable browsers and hide it from down-level browsers.

In this blog entry, I’m looking at 6 mobile browsers and their support of the IFrame element.

Motorola

IFrame support in Motorola V600

The Motorola mobile browser doesn’t support IFrames (tested here in the Motorola V600 emulator).

OpenWave 6

IFrame support in OpenWave 6

OpenWave 6 doesn’t support IFrames.

OpenWave 7

IFrame support in OpenWave 7

OpenWave 7 also doesn’t.

BlackBerry 8700

IFrame support in BlackBerry 8700

BlackBerry 8700 running a 4.2.1 firmware does not support IFrames.

BlackBerry 8830

IFrame support in BlackBerry 8830

BlackBerry 8830 running a 4.2.2 firmware also does not support IFrames.

IEMobile 6.8

IFrame support in IEMobile 6.8

IEMobile on Windows Mobile (formerly Pocket Internet Explorer) supports IFrames.

Opera Mobile 8.5

IFrame support in Opera Mobile 8.5

Opera Mobile 8.5 does not support IFrames.

Summary

Can you use IFrames in a mobile site? Yes, but the content will only be loaded by more capable browsers so plan accordingly.

Automate JavaScript compression with YUI Compressor and /packer/

Posted 2007-12-17 in JavaScript by Johann.

/packer/ output

JavaScript compression should be one of the finishing touches to your website, like optimizing your images with OptiPNG and other automated optimizations. Shrinking the size of your JavaScript code will make your site load faster and your visitors happier.

In this blog entry, I will show you how to automatically compress your various JavaScript source files into one single compressed file. The tools I will use is the YUI Compressor and Dean Edward’s /packer/.

Prerequisites

  1. Create the following folders:
    • js for the compressed JavaScript.
    • js-in for the input files.
    • packer for /packer/.
    • yui for YUI Compressor.
  2. Download YUI Compressor.
  3. Extract yuicompressor-2.2.5.jar to the yui directory.
  4. Download Rhino.
  5. Extract js.jar to packer.
  6. Download packer.js (originally written by David McNab) to packer.

Create the batch file

Copy the following code to a file called pack.bat and save it to your js folder.

type ..\js-in\* > jb.js
java -jar ..\yui\yuicompressor-2.2.5.jar -o jb-yui.js jb.js
cd ..\packer
java -jar js.jar packer.js ..\js\jb-yui.js ..\js\jb.js
cd ..\js
del jb-yui.js

(Replace jb with the name of your site or something similar.)

Tips

  • If you use multiple input files, you should name them like this:
    • 10-jquery-1.2.1.js
    • 20-jquery.inc-5.js
    • 30-menu.js
    This ensures that your files are compressed in the right order.

2 comments

Mobile browser statistics: WebKit on the rise, Opera losing share

Posted 2007-12-09 in Mobile Web by Johann.

Two months ago, I presented mobile browser statistics for my websites. The clear winner at that time was NetFront. Let’s do this again.

Mobile browser list

The top 15 mobile browsers, taken from the last four weeks of log files.

1. Mozilla/4.0 (PSP (PlayStation Portable); 2.00)
4004 requests. NetFront-based.
2. Mozilla/5.0 (iPhone; U; CPU like Mac OS X; en) AppleWebKit/420.1 (KHTML, like Gecko) Version/3.0 Mobile/3A109a Safari/419.3
338 requests.
3. Opera/9.50 (J2ME/MIDP; Opera Mini/4.0.9800/209; U; en)
270 requests.
4. Mozilla/5.0 (iPhone; U; CPU like Mac OS X; en) AppleWebKit/420.1 (KHTML, like Gecko) Version/3.0 Mobile/3B48b Safari/419.3
232 requests.
5. Mozilla/5.0 (SymbianOS/9.2; U; Series60/3.1 NokiaN95/11.0.026; Profile/MIDP-2.0 Configuration/CLDC-1.1 ) AppleWebKit/413 (KHTML, like Gecko) Safari/413
231 requests.
6. Mozilla/5.0 (SymbianOS/9.2; U; Series60/3.1 NokiaE90-1/07.24.0.3; Profile/MIDP-2.0 Configuration/CLDC-1.1 ) AppleWebKit/413 (KHTML, like Gecko) Safari/413
211 requests.
7. Opera/8.01 (J2ME/MIDP; Opera Mini/2.0.6530/1724; en; U; ssr)
168 requests.
8. Opera/8.01 (J2ME/MIDP; Opera Mini/1.1.4948/hifi/tmobile/uk; SonyEricsson K510i; en; U; ssr)
167 requests
9. Opera/9.10 (Nintendo Wii; U; ; 1621; en)
134 requests
10. Mozilla/5.0 (iPod; U; CPU like Mac OS X; en) AppleWebKit/420.1 (KHTML, like Gecko) Version/3.0 Mobile/3A110a Safari/419.3
124 requests.
11. Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0) BlackBerry8800/4.2.1 Profile/MIDP-2.0 Configuration/CLDC-1.1 VendorID/125
124 requests.
12. Mozilla/4.0 (compatible; MSIE 6.0; Symbian OS; Nokia N80/; 9712) Opera 8.65 [en]
123 requests.
13. Mozilla/5.0 (SymbianOS/9.2; U; Series60/3.1 NokiaE90-1/07.40.1.2; Profile/MIDP-2.0 Configuration/CLDC-1.1 ) AppleWebKit/413 (KHTML, like Gecko) Safari/413
107 requests.
14. Nokia6300/2.0 (05.50) Profile/MIDP-2.0 Configuration/CLDC-1.1
96 requests.
15. HPiPAQhw6900/1.0/Mozilla/4.0 (compatible; MSIE 4.01; Windows CE; PPC; 240x240)
77 requests.

Raw numbers

Two months ago, I looked at 2242 requests.

  1. NetFront: 1066 requests, 48 %
  2. Opera Mini: 626 requests, 28 %
  3. Opera: 390 requests, 17 %
  4. AvantGo: 148 requests, 15 %
  5. WebKit: 12 requests, < 1 %

This time, I looked at 6282 requests.

  1. NetFront: 4004 requests, 63 %, up 15 %
  2. WebKit: 1243 requests, 19 %, up 18 %
  3. Opera Mini: 605 requests, 9 %, down 19 %
  4. Opera: 257 requests, 4 %, down 13 %
  5. Nokia: 96 requests, 1 %
  6. IEMobile/Pocket Internet Explorer: 77 requests, 1 %

The verdict

  • Whereas I could see a slow shift from Internet Explorer to Mozilla-based browsers on johannburkard.de, the mobile web changes a lot faster and more deeply.
  • Microsoft’s mobile browsers are not popular at all, as are Gecko-based browsers. The mobile web looks like an anti-web from the browser perspective.
  • The mobile web usage has almost tripled in two months on my sites.

Pages

Page 9 · Page 10 · Page 11 · Page 12 · Page 13 · Page 14 · Page 15 · Next Page »

Subscribe

RSS 2.0, Atom or subscribe by Email.

Top Posts

  1. DynaCloud - a dynamic JavaScript tag/keyword cloud with jQuery
  2. 6 fast jQuery Tips: More basic Snippets
  3. xslt.js version 3.2 released
  4. xslt.js version 3.0 released XML XSLT now with jQuery plugin
  5. Forum Scanners - prevent forum abuse
  6. Automate JavaScript compression with YUI Compressor and /packer/

Categories

Navigation