Keeping the Search Engines Happy (part 2)

Submitting to Crawler-based Search Engines

Submitting to crawler-based search engines (Altavista, Excite, Lycos, Infoseek) is quite easy. All they need is your home page URL and your email address. Once their spider gets around to visiting your site, it will automatically index all your pages, so there's no need to submit more than the home page URL. Theoretically, search engines will list your site with no questions asked, unless you violate their rules (most of them will not list porno sites or sites that promote illegal activity, and they won't list URLs that trigger redirects to other URLs). In fact, however, a few sites fall through the cracks, and simply don't get listed, for unknown reasons. Also, many search engines will drop a site from their listings after a time, also for unknown reasons. Therefore, it's a good idea to resubmit your site to all the crawler-based engines at least every six months. This is perfectly allowable, and won't ruffle any feathers. Of course, submitting the same site over and over again (say, more than a half-dozen times in a week) may very well land you in the doghouse, and is furthermore totally pointless. Once your site is included in a crawler's database, it's in there. There's no such thing as having more than one listing for the same URL.

Submitting different domain names that point to the same page is a bit of a gray area. Officially, crawlers frown on the practice, but if you don't overdo it, it's unlikely to annoy them, and may cause your pages to come up a little more frequently in search results. For example, The Web Developer's Journal has 3 different addresses: http://WebDevelopersJournal.com, http://WebDevelopersJournal.co.uk, and http://wdj.co.uk, all of which are indexed in the major search engines (and, of course, only the co.uk URLs are listed in the UK-specific search engines).

What about auto-submitters? The consensus seems to be that none of the crawlers object to the use of auto-submitters, and I recommend them to save time. In fact, the best of them, Selfpromotion.com , (see the WebDevelopersJournal article), not only saves you time, but can help you do a better job of your submissions, with its built-in error-correction features and scads of valuable advice.

Submitting to Directories

Directories work a little differently. They do not automatically spider sites, but list only the specific pages whose submissions have been accepted, and real human beings look at each submission to decide whether it should be included. Therefore, it's important to choose the right categories in which to submit. Different directories have different policies concerning how many categories you may be listed under. Some limit you to one or two categories per page, but most allow you to submit a page to as many categories as are appropriate. Of course, most of them make you go through the whole submission process for each category, so submitting a page to lots of marginally-related categories wouldn't be a wise use of your time.

Yahoo is by far the most popular search site, and thus the most valuable one to get listed in. Unfortunately, it's also the hardest to get into. Whether it's because they're very selective or simply overworked (more likely), it's very rare to get listed in Yahoo on the first or even second attempt, and many a worthy site may never get listed at all. Increase your chances by taking special care with your Yahoo submission, and follow their rules to the letter. Some good tips are to be found on the rather obscure page called "How to Suggest Your Site" and selfpromotion.com also has some good Yahoo tips.

Yahoo will list a particular page in no more than two categories, so choose them wisely. Individual pages from the same site (that is, under the same domain name), however, can be listed in separate categories if they really have different topics.