Traffic-building Ideas from your Log Files (part 1)

There are as many different reasons for having a Web site as there are businesses or individuals, and every site has a different set of goals. There's one thing we all have in common, however: We want more traffic! If you're selling products online, then more traffic means more potential customers. If you're advertising a product or service, more traffic means more people get your message. If you're running an ad-supported site, then more traffic means more page impressions, and more money in your pocket.

If I had some sure-fire way to build Web site traffic quickly, I'd be fabulously well-to-do, just as I would if I had a sure-fire way to predict stock prices. There are some tried-and-true methods, however, and while they can't create instant success, if applied diligently over time, they'll help you build your Web site traffic slowly but surely.

• Submit your site to the search engines and directories, re-submitting periodically.
• Study how the search engines work, and tweak your site to try to maximize your search rankings.
• Solicit links from related sites.
• Run ads on appropriate Web sites and mailing lists. These may be paid ads, or banner swaps with related sites. Automated banner exchange programs like LinkExchange and HyperBanner are also useful.
• Build a database of press contacts, and send out press releases about newsworthy events concerning your company or your site.
• Participate in discussion groups and mailing lists that are relevant to your business, and discreetly plug your site and yourself.
• Constantly develop good new content. This is the only sure way to grow traffic in the long run, but it also happens to be a lot of work.

The ambitious site owner will use all of these tactics on an ongoing basis, but this article is not about any one traffic-building technique. It's about using your Web server log files to direct your efforts and measure your success. A good marketer can always think of lots of things to do to promote a site, far more than could ever be accomplished with the time and money available. You can't do everything, so you need to figure out exactly what works, and direct your efforts to the most effective tactics. Careful analysis of the information in your log files can give you lots of promising traffic-building ideas, and also help you measure which ones live up to their promise.

How Log Files Work

Every time a file is retrieved from a Web site, the server software keeps a record of it (assuming that logging is turned on). The server stores this information in text files, (usually with a .txt or .log extension), called the Access Log, Error Log and Referrer Log. The log files contain not only a record of which pages were requested at which times, but a good bit of information about the people (or other entities) that requested them.

As you can imagine, log files can get huge very quickly, and take up an enormous amount of expensive hard drive space at your hosting service. Therefore, most Web servers are set up to "rotate" or "cycle" the log files in some way, to make sure that all the files get saved, but that they don't hang around on the server. A simple way to do this is to have the server automatically email a copy of the log files to somebody periodically. This lucky individual transfers them to some permanent storage location, and the server automatically purges the original log files after a certain amount of time.

If you want to have decent stats for your site, be careful about keeping your log files organized. It's a pain in the neck, but worth it - any gap in your data can screw up your reports, and once it's lost it's lost.

The wealth of data in the log files is not readily mined with the naked eye. A raw log file entry looks something like this:

206.135.203.174 - - [19/Jul/1999:00:00:04 -0100%] "GET /studio/drives.html HTTP/1.1" 200 20607 "http://www.webdevelopersjournal.com/studio/hard.html" "Mozilla/4.0 (compatible; MSIE 5.0; Windows 98; DigExt)"

As you can see, this entry shows what page was requested, when it was requested, where the visitor came from, and even what browser and OS they were running. As I'm sure you can also see, you won't learn much of interest just by looking at the raw log files. There's page after page of this stuff.

To get the most out of the data, you need to be able to see totals for the whole site, and compare the figures over time. That's where a log analysis software package comes in. These handy tools range from Getstats (a free Unix program that can run on your Web server) to various cheap shareware options, to industrial-strength packages like Marketwave Hit List Pro 4.0 ($395 list) or WebTrends Log Analyzer 4.52 ($399 list).