What Makes a Good Web Directory, and Why Google Penalized Dozens of Bad Ones

Two weeks ago, Google took some severe action against a number of general topic web directories. There was the usual uproar on various forums, lots of blog posts from web directory owners and even a post on Sphinn that blamed me (which is hilariously awesome):

All of these directories no longer rank for their trade mark names, Rand Fishkin (who condones link buying) does not like directories so he has been complaining to his buddy Matt Cutts and Matt has gone out and manually penalized a large number of the leading directories.

However, today I wanted to seriously cover the topic of how and why Google might take this move, along with some advice for anyone building a directory in the future. First off, though, I'd like to examine the directories that have been penalized. I'm sure this isn't an exhaustive list (and if you know of more, feel free to list them in the comments), but it does represent a good sampling of the affected domains:

• AvivaDirectory.com
• AliveDirectory.com
• Haabaa.com
• DirectoryDump.com
• BigWebLinks.com
• ElegantDirectory.com
• eWebPages.org
• LinkBook.org
• Trincas.org
• CBravo.com
• CDHNow.com
• FreeWebIndex.com
• Mingleon.com
• PremiumDir.com
• Submission4u.com
• Aerospect.com
• Bakie.com
• LinkForever.net
• LinksArena.com
• LinksFactory.net
• LinksHolder.com
• WebVerve.com
• Wezp.com
• BestInternetResource.info
• DirSpace.com
• Eonte.com
• Frogengine.com
• LinkLister.co.uk
• LinkVerve.com
• LivelyDirectory.com
• Submitdotcom.com

How do we know these sites are penalized? Search results are usually pretty good evidence, and here's the type of pattern you see for a site like Alive Web Directory:

• Doesn't rank for "alive directory"
• Doesn't rank for "alivedirectory"
• Doesn't rank for "alive web directory - internet site resources powerful human edited web directory" (title tag plus a few words from the meta description)
• Ranks #1 for "alivedirectory.com" (so we know it's still in the index; and it's even got the sitelinks - keep that in mind next time someone tells you sitelinks are a sign of trust)

I ran through a lot of searches - avivadirectory, aviva directory, haabaa, haabaa directory, directorydump, bigweblinks, elegantdirectory, elegant directory, ewebpages org, ewebpages web directory & catalog, and on and on, through every one of the directories listed above. The patterns were always the same - the sites couldn't rank for their own name, even when combined with obvious phrases from their homepage. They only appeared when the "domain.com" format was used. I would bet money (and lots of it) that these domains are receiving very little, if any, traffic from Google and that links that appear on them aren't currently worth squat.

Why?

Because of these Attributes of Obviously Manipulative Directories (I almost gave that name to this post, but I figured it might be a little much). These aren't "hard and fast" rules - they're just common traits that many of the low quality directories seem to share. I'm NOT saying that you can't run a directory and do any of these - there's always going to be gray areas and matters of intent. These signals, however, are ones that, particularly when combined, make me shy away from a directory:

1. General in subject matter - This isn't a bad thing on its own, but it's certainly a signal that you may be getting a manipulative directory . While there are a few good general subject directories that Google probably does want to count (Lii, Yahoo!, DMOZ), there are far more who simply build general subject because it maximizes potential revenue (as anyone can apply).
2. Anyone can get in - If you don't filter out low quality, spammy websites from being listed in your directory, even a pretty badly built algorithm can easily spot and remove you. Besides which, Google has been on a tear for years about bad links and bad neighborhoods and how they use the sites you link to as a signal for spam identification.
3. Marketing to Webmasters - If your forum signature at Digitalpoint (sorry to stereotype, but it's just so true) contains links to three directories you own, you're probably in possession of three obviously manipulative directories. I'm sure there are a couple exceptions, but if I were Matt Cutts, I'd just tell one of my quality control guys to go spend a few days trawling DP for directory domains.
4. Promoting Search Engine Link Value, not Traffic - The great majority of the domains I listed use phrases like "search engine optimized" or "high PageRank" or "highly ranked" to describe their directory. Once again, this should be a clear signal that you're not selling listings in a directory, you're selling links that are supposed to manipulate the search engine rankings.
5. Use of Manipulative Link Building - Since the general directory industry seems to pride itself on toolbar PageRank, there's a lot of very shady link building tactics being employed by many directory owners. Sponsoring blog template themes, buying links at crappy directories (I know, the delicious irony of it all is hilarious), putting out junk press releases, releasing link-passing affiliate programs, joining webmaster forums that allow signature links, etc.
6. Stuffing Links & Content to "Look Natural" - It's rough to see the effort that many directory owners put into trying to "appear" natural, by adding links to government and education resource websites, major media sites, etc. A lot of the time, it's really easy to spot this "looking natural" business over an actual, naturally built directory. It's usually by category - the section on social sciences is filled with a few great sites, while the page on Minnesota DUI Lawyers looks a little funny.
7. Setting up "Premium" Sponsorships - When directories have a higher price you can pay for "extra links" or a higher placement on the page or assurance that you'll be linked to in every category, that's a decent sign that Google's spam team is going to come calling one of these days.
8. Interlinking with Other Directories - If I can buy entry in your directory, along with three other directories for "one low price,"  I'd probably be better off burning those twenties for warmth (or, you know, trading them in for $19 Canadian).
9. Common Popular Links - When I look through a directory's "most recent additions" and see a cosmetic surgeon, an Internet casino games site, a UK mortgage property, and a Pennsylvania health insurance provider, I can be relatively assured that any decent, self-respecting search engine probably wants to yank the link value pretty quickly.
10. Bid for Links - This has to be the most obvious link manipulation ploy I've seen in a while. How could you honestly think that search engines would want to count those links? It's like the eBay of spam, only without negative feedback.
11. Multiple Links with Your Choice of Anchor Text - I shouldn't have to explain this one - if you can choose your anchor text and point to several pages on your domain from your listing, it's pretty clear that the directory isn't targeting humans.
12. Banner Ads from Your Directory on SEO Sites - It's like waving a flag with a voice-activated, wind-powered speaker that yells "Ban me! Ban me!" Sure, you might get clicks and money and submissions, but you've gotta know that search quality team members read SEO blogs, too - so if you do this, make sure your directory is ready to be manually reviewed by search engineers.
13. Demanding Reciprocal Links - If a directory requires that you link back to them in order to be included, or that you can link to other sites they promote in exchange for reduced payment or free inclusion, it's almost certainly trying to manipulate search rankings through linkage.
14. Choose Your Own Anchor Text - Not nearly as fun as Choose Your Own Adventure, this screams "manipulative and built for rankings, not humans." DMOZ & Yahoo! and lots of the more legit directories will only use the company name or a site description, rather than allowing the user to decide on their own anchor text. This is particularly egregious when the directory lets you link to 4 or 5 pages and pick the anchor text for each link.

Here's what gnaws at me a bit, though - why these fifty sites? Why did Google penalize a few dozen (or even a few hundred, since I probably don't know about all of them) directories, yet leave the great bulk of low quality, obviously manipulative ones alone?

Maybe they didn't - maybe Google penalized many more directories as well by removing their ability to pass link juice. It's possible, but it doesn't look that way right now. A few friends on the shadier side of link building told me that they can still get top rankings for moderately competitive phrases (usually with local modifiers) just by buying a few hundred directory links. It's a bit expensive, but it still works - and that's a fundamental problem.

I'm going to say this for the record - so long as Google (and Yahoo! & MSN/Live) keep ranking sites and pages purely on the strength of directory links, the directory industry will never disappear. If the search engines want to get serious about paid links and manipulative directories, they're going to need to hit a few thousand general directories harshly. Only when that's been done can they claim real credibility in this arena. Until that time, it's just fear-mongering to keep link buyers on their toes and, hopefully, make the less savvy ones shy away from spending money since they won't know if a directory's been penalized. And yes, for those who are keeping track, I think Google and the other engines should absolutely penalize a directory like SOCEngine (which, as longtime readers might recall, was a directory that SEOmoz started in 2004 & left inactive for the last couple years - what can I say, even I was tempted by the easy money of directories once).

One more thing on that subject - penalizing directories like Alive while still keeping the toolbar PageRank showing as 6/10 isn't going to stop very many people from buying those links. Google may be more concerned about letting the owners of the directory know that they can't manipulate their index, but personally, I wish Google would also worry about the uninformed webmasters and businesses who think that directory link buying is a good way to conduct search marketing. When you show an endorsement in the form of green fairy dust, you can't expect buyers to run a search, see the website's not listed, and make the right assumption about the value.

Before I let you go - I think it's important to cover the other big topic of this post - what makes for a good directory? The answer isn't quite as simple as "Do the opposite of the 12 steps above," although that's certainly a good start. Here's some tips for you directory builders out there who want to reform, take a new stab, and build a truly high quality resource:

1. Start with a Niche - Find a topic you're seriously passionate about, from birds to routers to online clothing merchants.
2. Don't Just Make a Directory - Put great content about your subject on the site: blog posts, articles, tools, resource lists, charts, diagrams, investigative journalism, etc.
3. Offer to Review Sites in Your Niche - But, for goodness sake, only include them if you'd really, honestly endorse them.
4. Provide a Reason Why They're Listed - Imagine a fellow hobbyist or researcher in your topic of interest in real life - if you couldn't sit down with that person at a table and show them on your laptop why you included a particular site, DON'T include it.
5. Don't Offer Gimmicks or Link Juice - Offer listings on a site that real people who are really interested in your topic read and use and enjoy. If you start down the path of selling links for search engine value, you've lost your way. It can always be a secret side benefit, and plenty of folks who'll come to you for links will be thinking about it, but if you want to be truly immune to any future penalties or devaluations, you can't make it a focus.

Finally, I want to wrap up by addressing those folks who are seeking good, solid directory links that will add value in the long run. Granted, it's no easy task, but there are tens, if not hundreds of thousands of great directories out there to be listed in. They usually don't look much like the directories I've commented on above, and many of them require antiquated submission, payment over the phone, a personal email, or even a pitch on why you should be included. Sites like Te Puna Web Directory, the Atlantic Canada Portal Web Directory, Harvard University's Molecular and Cellular Biology Dept. Biolinks, Comic Books X (a good example of some of the points I laid out above, if not a spectacularly designed site), and the Directory of the Center for Indigenous Environmental Resources may be far more difficult to gain entry into, but that's one of the big reasons they'll provide value.

BTW - If search engineers are seriously having trouble finding manipulative web directories, here's a good place to start. Honestly, I don't mind the penalties, just the inconsistent way they're applied.

p.s. I had originally scheduled an interview with Jeff Behrendt of AvivaDirectory (one of the better directories on that list, at least in my opinion) about this topic, but Jeff's bowed out, unfortunately. Sorry about that - hopefully the post is still valuable without his input (though I would have loved to have it).