StepForth SEO News Blog

Wednesday, November 24, 2004

How to write for Slurp the spider

As the world's second most popular search tool, Yahoo moves a tremendous amount of traffic and is a very credible alternative to Google. Yahoo receives over 2.76 billion page views per day from hundreds of millions of unique users. It boasts over 157 million registered users enjoying mail, shopping and discussion groups and an increasingly personalized search and news services. For the past two years, Yahoo, Google and MSN have been embroiled in a hard-fought battle for the loyalty of search engine users forcing all three firms into the hyper-evolution we are witnessing today. Over the next three Wednesdays we are going to examine how the Big-3 spiders work, what they look for and how to best prepare your sites for multiple visits from the bots that rank them. Today, we are starting with Yahoo's bot, SLURP.

Getting found by Slurp
The first thing to know about Slurp is that like its better known cousin, Google-bot, Slurp "discovers" sites by following links from one site to another, reading and recording nearly everything it finds in its path. The majority of websites referenced by Yahoo were originally included in its database because they were accessed by Slurp following links from another site.

Yahoo suggests adding an inbound link to all pages in your site to guarantee those pages will be discovered by Slurp. They also recommend an internal sitemap linked to from the Index (or home) page of the site. To encourage Slurp to spend more time deep-crawling your content, Yahoo recommends the addition of "good authoritative links pointing into your site", from highly reputable sources such as news sites, established business partners and other sites relevant to your business or service.

Manual submission of the site is only recommended if for some reason or another Slurp does not find the site on its own. This is increasingly rare however as server-logs show Slurp is one of the most active spiders out there. In other words, if a site Slurp has already indexed links to your site, Slurp will almost certainly be visiting very soon. Webmasters should never have to pay submission fees to get into Yahoo's index as according to Yahoo's Tim Mayer, 99% of Yahoo's index is crawled by Slurp for free.

It is still important to make sure your site is ready to receive a visit from Slurp. To ensure Slurp is able to travel across your entire site, provide standard HREF text links as opposed to forms, Flash or java script navigation tools. Webmasters are encouraged to avoid tracking and communication methods that rely on using cookies across every page of the site. If you have a database driven site or a site that creates unique sessions for each user, avoid embedding session IDs in URL's. Lastly, use 404 pages to redirect users (and spiders) to the root (index) page if a page or site URL becomes invalid. Yahoo also asks webmasters of sites with shopping carts to use robot.txt exclusions in the source of the shopping carts.

Where your site has been included. Results May Vary...
Yahoo has seen enormous change over the past seven years. What started as a paid-inclusion, human edited search directory, has grown into the second largest database of indexed content. Yahoo is on the cutting edge of integrating several forms of media into their search offerings and will likely soon produce its own entertainment content like an online HBO. Yahoo is flirting with the concept of becoming an infotainment portal again but the core of its offerings remains firmly rooted in search.

Yahoo search results come in multiple formats including: Yahoo-Local, Yahoo-products, MY-Yahoo (personalized results), specific nation-based Yahoo's, and the standard Yahoo.com One of Yahoo's goals appears to be presenting individual search-users with results that best match their personal needs. For instance, Yahoo would like to present constantly updated geographic-specific references when a user searches for daily-use items such as groceries, repair-workers, real estate and other services one would normally use a telephone directory to find. Similarly, Yahoo wants to present the entire global database of references when a user searches for international news, trans-national products or vacation plans. Being certain your website gets served up for all levels of search, local, regional and global, will be important if you wish to serve a market larger than your general region or community.

Getting Rankings
Yahoo's search engine ranks sites based on a formula that is very similar to the algorithms used by rivals Google and MSN. Yahoo values many of the same elements other search engines do including keyword enriched domain names, titles, meta tags, and content. Yahoo also values keywords found in the anchor text of internal links, though the effect at Yahoo is not as powerful as it is on Google.

According to Yahoo, well optimized pages and sites will continue to get good results across all versions of their search engine. By opening your site to Yahoo Slurp and performing well-planned optimization services across every page, a good SEO can nearly always achieve Top placements on Yahoo. The trick is in offering Slurp the information it needs to read, record and rank your site. If that information is included on each page, a set of text-based links is woven through the site to provide easy passage for Slurp, and Yahoo is told what your business is, where your business is located, and who your business serves, your site should achieve strong rankings.

Due to the advent of personal, local, regional and global search results, it is highly recommended to add full contact and address information on every page of a site. This information should be as precise as possible and should include street address, unit or suite number, zip or postal code, state or province, county and country, full telephone information (including area codes), and if possible, the approximate longitude and latitude of your business location. (look up your longitude and latitude here: http://www.astro.com/atlas)

When writing for Slurp here are a few basic fundamentals:

• Have a descriptive URL
• Use keyword enriched titles on each page of the site
• Place keyword enriched description and keywords meta tags on each page of the site
• Use robot.txt files to keep Slurp out of your shopping cart or log in pages
• Place keyword enriched text in the first paragraphs of your site-copy
• Use HREF links to direct Slurp through each page of the site
• Add a sitemap page and be certain there is a link from the INDEX page to the sitemap
• Be certain that geographic specific information is mentioned on each page of the site. Always have a contact page that also lists geographic specific information.
• Write a press release and send it to as many blogs, news-wires and press release sites as possible.
• Acquire strong, relevant incoming links from sites with topics similar to yours.
• Update your site frequently.
• Enjoy and value your placements.
  

Copyright ©2006-2012, The SEO News Blog, StepForth SEO Services
View this Article Only: Yahoo's Spider Slurp

Wednesday, November 17, 2004

It has been a full year since the infamous Florida Update rewrote Google's rankings with a massive pre-Christmas purge of previously well placed sites. The update, which caught virtually everyone by surprise is assumed by most to be the introduction of semantic contextualization software added to a variation of the Hilltop Expert Document Algorithm. It took Google about six to eight weeks to re-establish stable listings and they took a savage beating in the SEO press during that period. For a short time it looked like the shift was a failure with spammy sites and "big-box" stores dominating the Top listings but after a while Google's listings began to make sense again.

The inclusion of Hilltop added great weight to certain types of links and as Mike Banks Valentine points out in today's WebProNews, radically cut the value of reciprocal links. Incoming links were the meat and potatoes of PageRank but five years of commercialization had turned them into junk food. Googlebot needed a better diet and direct one way reference links from "credible" and relevant sites is considered much healthier spider-food than the junk peddled by fast-link dealers or cooked up in kitchen-sink sites between friends.

The Florida Update effected more than site rankings in the SEO industry. New businesses sprang up finding, buying and selling links as SEOs and site owners became obsessed with PageRank values and link-building. Savvy site designers and a few SEOs began producing instant expert sites, using them as "back-door leader-pages" designed to drive spiders rather than attract them. Websites with incoming links from "authority sites" such as news-sites, major forums and other high PageRank properties started to rank better than they had before, thus increasing the value and popularity of these authority sites.

Perhaps the greatest effect of the Florida Update was the sudden rise in the popularity of BLOGS. As explained in dozens of articles over the past two months, Blogs have become big, primarily because of their extraordinary effect on link-densities at Google. When a thousand bloggers rapidly create links to a specific website or Blog entry, that site or entry rapidly rises in the rankings. Proof of this phenomena is found in the results of both Google Ranking contests held earlier this year. The winner in both cases harnessed the power of keyword enriched anchor text-links from thousands of Blogs.

It is easy to speculate on Google's commercial interest in promoting incoming links from Blogs and news-sites. Google owns the biggest Blog software developer, Blogger, and has AdWords/AdSense being displayed across many "authority sites". Google also controls the world's most popular News-Search tools, Google-News. With the pressures of becoming a publicly traded company, Google needs to harness as many revenue sources as possible. When you control the environment, forcing a change in user behaviours may be simpler in the long run than waiting for change to happen organically. There may be some merit to this view. The people working for Google wouldn't be the geniuses they are if they didn't see the financial angles but after a year has passed, it appears the shift was motivated by much more complex changes happening at the Googleplex.

Google has a number of interesting features and has been busy acquiring the means to develop others. Through direct purchase of software (and often the companies that make it), innovative in-house development and the massive infusion of energy spurred by the IPO, Google has raced ahead of its competition in the past three months and is seen to be leading in just about every facet of search. It is rumoured to be developing a proprietary browser and may even be working on a proprietary operating system. The introduction of a semantic/context driven algorithm that values links from established authoritative sites has improved Google's listings and provided the stability needed to integrate Google's various features into a branded base of products. That many of the most beneficial incoming links originate from websites already involved with Google in one way or another, (aside from their own listings), more likely stems from chasing the brand as opposed to chasing the buck. Once the brand is established, the bucks tend to follow.

Google has established one of the most popular brands that has ever existed but that brand, in most consumer's eyes, only extends to its search engine. Google's founders have always stated that information-flow was their major focus. Information, is a bigger word than search and requires a larger vision to articulate. One year after the biggest shake-up in search engine history, Google is positioning itself to surpass simple search and grow into branded information provision.

Copyright ©2006-2012, The SEO News Blog, StepForth SEO Services
View this Article Only: Florida Update One Year Later - The Year Google Grew Up

Wednesday, November 10, 2004

Commercial websites are getting larger. Driven by the rapid evolution of content management systems, shopping carts and e-biz facilitation, and by the increasing sophistication of Internet retailers, "small" business sites averaging 500+ pages have become common.

Some large sites are very well focused and present relatively few problems for SEOs. Most larger sites however list a wide array of products, services and information. The optimization of large retail sites presents multiple issues for SEOs to work through. Achieving product-specific placements for sites featuring numerous products is much more difficult than achieving placements for smaller, more focused sites. Fortunately, good SEOs are good problem solvers and almost every technical problem has a solution.

A placement campaign for a large site is a long-term venture and clients approaching SEOs should be prepared to be involved in the campaign and to exercise patience while waiting for results. It might even take a few days for the SEO to prepare a preliminary plan for the campaign. There are a number of questions and challenges clients should expect from their SEOs.

First of all, the SEO will want to define a broad range of goals and expectations for the campaign. Meeting a client's expectations is made easier when both the client and the SEO have taken the time to define what those expectations are. Most conflicts between clients and good SEOs tend to stem from a lack of communication regarding client/service expectations. Issues such as perceived time-lines, delivery dates, the staggered nature of achieving Top10 results under multiple keyword phrases and long-term communications are frequently the greatest causes of mid-campaign client concern. Most clients don't fully understand the challenges or the process. Wise SEOs help their clients to understand the process, find ways to mitigate the challenges and temper their expectations with an honest version of campaign realities.

Meeting a client's goals is infinitely less challenging when the client has realistic expectations. Initially, the client's goals are often very straight-forward. Clients tend to want Top10 placements under a number of keyword phrases describing their businesses or it's products. After hearing about the wonders of optimization, clients don't always understand that achieving these placements is always easier said then done. With a multi-focused site the idea of achieving multiple keyword targets can set some fairly high expectations. Establishing a series of goals for a large-site campaign is a smart way to create a manageable campaign plan and to manage client/SEO expectations.

Long-term campaign goal setting can be likened to mountain climbing. The climbers study the terrain and create a goal-orientated plan before climbing. When they do begin climbing, they aim to reach a series of plateaus to rest and re-group. Thinking about a long-term placement campaign in terms of attainable plateau-goals creates the conditions for better results.

The first goal should be "brand name" recognition, or placement of the Index page under a narrow range of highly descriptive phrases. These phrases should be the mainstream market keywords describing the best selling products or the types of product the company wishes to brand their site under. With the power of multiple pages, a common theme can be woven through titles and anchor text. Once placements for the Index page under easier keyword phrases are achieved, it is somewhat easier to reach other plateaus. Subsequent goals would include Top10 placements for the second-level internal pages and eventually, specific-product placements under an increasingly wider array of keyword phrases.

The third set of goals can present another series of expectations and challenges for both clients and SEOs. The Internet allows retailers to "stock" an infinite number of products. For some, the temptation to build huge sites is matched only by the incredible availability of products to sell. As long as they can ship the product to the consumer, they can list the product on their sites. How many keyword phrases does the client wish to rank under? Are there multiple keyword phrases that could be used to find individual products? How deep does this database go anyway?

After establishing initial goals and expectations, the SEO will want to know as much about the client's industry, products and services as possible. Knowing about the sector and the competition helps the SEO learn about the potential visitors the client wants to visit their site. Are the target-visitors corporate buyers or individuals purchasing products from their homes? Are there industry-specific keywords that are used by buyers in the industry? How does the competition market itself? Answering these questions helps SEOs target the “right” visitors and hone in on what should be the strongest performing keyword phrases.

Choosing keyword phrases is half the battle. The other half is using them properly when optimizing the site. The optimization process of a large site is often very complex. Just as SEOs should provide detailed information about techniques and progress in the campaign, clients should provide huge amounts of product detail at the beginning of the campaign.

In most cases, a full SEO effort requires rewriting text. The more detail SEOs receive from their clients, the easier it is for them to craft descriptive, keyword enriched text. When the site in question is a multiple product site, having as much detail as possible is indispensable for SEOs as they struggle to find the right words to build well-crafted sentences. When you are working on someone else's business site, correct is critical. SEOs should not be afraid to request or even demand more information from their clients if they feel it is necessary or if the information provided by the client is insufficient.

Another typical large-site scenario challenging SEOs is the array of Content Management Systems (CMS), shopping carts, and E-commerce facilitators such as Yahoo-Shopping. Each of these types of systems can cause unique issues for different reasons. While most CMS systems share similarities, each presents a learning curve for SEOs used to working on code or in a standard HTML Editor such as Dreamweaver. Every shopping cart, even the most SEO friendly will present challenges for SEOs, and optimizing Yahoo Shopping sites can be as frustrating as polishing a tuba with a tooth-brush. Nevertheless, there are solutions for all problems presented in a technical universe. With complicated sites however, these solutions are often best found working in collaboration with the client's staff. Clients should be prepared to help their SEO smooth over rough patches with other web-technicians working for their business. When negotiating the original agreement, the client should, if possible, supply the SEO with a technical liaison and the contact information for the ISP running their content management system or shopping cart.

Difficulties posed by differing CMS systems and shopping carts can often lead to higher costs for the client. SEOs can shave some time from their estimates by providing SEO consultancy services as opposed to direct hands on services. In other words, the SEO can prepare a document outlining all modifications recommended for the site. This can cut hundreds or even thousands of dollars off the cost the optimization and prevent the SEO from getting bogged down under a steep learning curve. A quick word of advice on this tactic... Prepare and present your consultancy document in NOTEPAD and include line-numbers. The first time I offered consultancy over hands-on work, I presented the information in a WORD document without considering that WORD-formatting code might spill over into the site if the client's IT person cut and pasted the suggested modifications to their site code.

The last major topic a client and SEO will want to cover is long-term responsibility for maintaining and updating the optimization effort on the site. There are dozens of factors to be considered from changes to search engine algorithms (rare) to changes to the client's site (very common). As time moves on, new additions to the client's site and those of the competition will effect rankings gained through the optimization process. Having a maintenance plan or a monthly review of the client's rankings and site should be a part of every optimization contract. Reputable SEOs provide exclusivity under keyword phrases to their clients based on the simple idea that even the best SEO can't make more than one site #1 under any given keyword phrase. Long-term maintenance contracts often serve several purposes. First of all it offers two levels of insurance for the client by extending keyword-exclusivity and guaranteeing if there is a change in rankings, the SEO will be on top of it as soon as possible. Long-term maintenance agreements also serve to compensate the SEO for lost business stemming from having to turn away other businesses competing for the same keyword phrases. As a large-site campaign is a long-term commitment on the part of the client and the SEO, this form of compensation is more than reasonable.

Finally, the client should be prepared to pay for submission of their large site to directories such as Yahoo, Joe Ant, WebAtlas, and other paid-inclusion directories. The costs of these submissions might end up around $500 but the effect of making the submissions will be beneficial in the long run. As this is a large site with a staggered optimization plan, the two – three months it will take to fully optimize should be enough time for the site to be reviewed and included in the directories. SEOs should also submit the site to the Open Directory Project (though they should be prepared to be patient.) as Google continues to give weight to sites found in the ODP.

When the optimization and submission processes are complete the SEO and client can relax for a couple of weeks to await results. When the results begin to show on the search engines, either more work or immediate kudos are in order. Either way, by this point, the relationship between the SEO and the client should be strong and communicative. Large sites can mean long-term business for an SEO firm and a good SEO can mean increased traffic and strong sales for the client. Keeping the optimization effort simple in the face of complexity pays off in the end with a happy and educated client, and good placements for both the client and the SEO.

Copyright ©2006-2012, The SEO News Blog, StepForth SEO Services
View this Article Only: Organic SEO - Getting Started with larger sites