During the last few months I’ve been noticing serious changes in search engine behaviors especially on my large websites having more than 100k pages. Some of them have been sitting on my HostGator account who’s well know as being one of the largest shared hosting overseller offering unlimited domains hosting. When I moved those sites to new dedicated server that I’ve purchased for my own needs, I’ve noticed sudden surge of search engine bot activities. The site got crawled by tons of search engines and less than a week later there was more than 600k indexed pages in Google while previously there were around 40k to 60k pages.

My findings confirm that search engine crawlers count the number of domains per IP address and divide the approximate time spent on each one therefore if there are a lot of domains hosted on the same ip address your index time allocation is divided by the number of domains hosted. This immediately results in poor/slow indexing especially on large websites.

The best way to find how many other pages is to run a http://whois.domaintools.com/yourdomain.com and find out your particular name server domains usage. NOTE: This does not work for some hosting companies that use centralized DNS management. If you see more than 2k websites then most definitely that’s not accurate and you must find some other way of detecting the domains usage on your shared server. This tool works fine when the name servers are independent for each hosting server. If you see that you account’s name servers are something like ns1.hostserv3.com and ns2.hostserv3.com then its most definitely OK and the number should be accurate…

I’m not really sure but I suspect that if you use your own Dedicated IP (even on shared hosting) you should not have any crawling issues, but then this depends on the average server load of the server your account is hosted on. This makes shared hosting really bad for websites that made the money to afford themselves powerful VPS or even dedicated server packages. If you don’t have the money to spend on dedicated server you should check Maiahost’s Semi-Dedicated hosting plans which are pretty much the cheapest solution to having high speed low domains hosting.