There are more than 2 billion documents already on the Web, with about a million more pages added each day. Since studies have shown a whopping 85 percent of all Internet users go directly to search engines to find what they are looking for online, a website owners task is clear: get listed in the search engines top five results of a keyword search.

If you’ve ever endeavored to get listed in the search engines, you probably know there are ways to go about it that will get you a bit closer to the top position. You could call upon search engine optimization (SEO) companies, whose job is to make a Web page relevant with the keywords that are in it, or you can educate yourself with tips that will get you the clicks.

What do you need to do to make your site indexable? Does automated software submission work anymore? Are sponsored links effective? So what are the big secrets?

One respected expert has been studying the search engine industry since the mid-‘90’s. Danny Sullivan is editor of U.K.-based SearchEngineWatch.com, considered by many to be the destination site to get the final word on search engine issues.

We asked Danny Sullivan to unlock some of the mysteries of the search engine submission game.

Q: Why did you start SearchEngineWatch.com?

Sullivan: It goes back to ’95 when I was doing web development. Of course, as soon as you built the site, you’d get asked, “How come this isn’t showing up in search engines”. In 1995, nobody really knew anything about search engines. So I started looking into how they crawl and list websites, discovered lots of interesting things and published it in SearchEngineWatch.com. We’re getting both people that are trying to promote websites and people who are searching the web and find information more easily.

Q: It seems getting listed in search engines is still a big mystery for most people. Why is that?

Sullivan: It’s ironic. It’s probably easier to get listed in search engines today that it ever has been. Yet it is also more complicated. As the search engines evolved, we’ve had all sorts of new programs emerge. For $20 per month, you can get your listings going with Overture and that can actually get you present on a whole bunch of different search engines. You’re site will come up for the keywords you want - at least for a short period of time, until you run out of money. But it is totally easy. You can literally be up and running in a couple of days. But if you wanted to be listed in Yahoo!, it would probably take you some time to figure out the connection between it and Overture and work out the details. That can be a mystery for those without a clue - yet that is one of the easiest ways!

Then you start talking about how Google is going to interact with your frame-based site that’s served out of a database which is all built out of Flash - it can become a real nightmare! So they are definitely complicated. But the upside is there are actually more opportunities to get listed.

Q: Certainly you can pay to get listed as a sponsored link, but getting a free listing is still just as challenging as it ever has been. Are there changes to help people get the free listing?

Sullivan: There have been changes. The free listings have historically come from people that crawl the web, like Google, Inktomi, new players like Teoma or Fast/AllTheWeb.com. They all have programs, except for Google, where you can pay to speed up the process of getting listed.

There’s a much more greater reliance these days of listing pages based on people linking to you. If people link to your website, the odds of those pages getting picked up for free and naturally appear are much greater than if you had created a web page and just went to the “Add URL” pages that they all have. In the past, those “Add URL” pages used to be very responsive, but they just don’t depend on them anymore because they get so much bad stuff coming to them. People submit millions of URL’s that are nothing but search engine spam and low-quality content.

Alta Vista once said a number that made everybody gasp when they said at one of our conferences that “90% of what they receive is spam” through their “Add URL” service. What they meant was not that 90% of people were spammers, but that most of the people who use the page are submitting good things, but they’re getting drowned out by a few people who are using it to excess.

That’s why they can’t depend on those pages any longer.

~ WTG

(Dana Greenlee is producer and co-host of the WebTalkGuys Radio Show. WebTalkGuys, a Seattle-based talk show featuring technology news and interviews. It is broadcast on CNET Radio in San Francisco and Boston, on the web at CNET Radio, WebTalkGuys Radio, Sonic Box and via the XM satellite network and the telephone via the Mobil Broadcast Network.  Past show and interviews are also webcast via the Internet at http://www.webtalkguys.com).

PC World magazine names WebTalkGuys "Best of Today's Web Hidden Gems" in their August 2002 issue.