Latent Semantic Indexing (LSI) and SEO

Indexing has always been considered a highly targeted science. Enter a search query into Google search and the pages that are displayed are generally optimized towards that exact word or term. However, in their continual battle to server the most relevant but most natural pages with genuinely useful information Google has injected latent semantic indexing (LSI) into its algorithms.

What Is LSI?

LSI is a unique indexing method that potentially takes Google search one step closer to becoming human in its way of thinking. If we were to manually search through web pages to find information related to a given search term we would be likely to generate our own results based on the theme of the site, rather than whether a word exists or doesn’t exist on the page.

Why Search Engines Might Adopt Latent Semantic Indexing

The extremely rigid form of “keyword indexing” also meant that black hat SEO techniques were easier to implement. Search engines could be manipulated into ranking a site highly by using set formula. Originally, cramming a page with a particular keyword or set of keywords meant a site would rank highly for that search term. The proceeding set of algorithms ensured that your link profile played more of an important part than your keyword density. Reciprocal linking soon followed once again making it possible to manipulate the search engine spiders by exchanging links with tens, hundreds, or thousands of websites.

Reciprocal linking was soon beaten as Google and to a lesser extent Yahoo and MSN gave less credence to a reciprocal link as they did to a one-way inbound link. Latent Semantic Indexing is another, particularly powerful, method to try and make their result pages appear more natural with natural pages filled with natural content.

The Effects

The introduction of LSI has seen some dramatic changes in the search engine result pages already. Sites that had previously performed well because of an impressive link profile based on a single keyword have found their pages slip in the rankings. Other pages with a more diverse portfolio of inbound links are taking the lead with search terms for which they had not previously performed.

SEO is far from dead because of LSI, in fact if anything, it has probably increased the need for professional white-hat SEO on your website. The field of SEO, though, has almost certainly changed. Website content copywriting for Google’s benefit is not merely made up of keyword density and keyword placement as it once was and link-building techniques will need to change to incorporate LSI algorithms but it can be done.

Writing Content For LSI

If optimizing solely for Google then a web page can, theoretically, be naturally written and naturally worded. When we write we instinctively include the appropriate keyword in our text. In order to avoid repetition (or keyword optimization, as it was once called) we would often alter some instances of these keywords for other words with the same or very similar meaning. We naturally include the plural or singular form of a keyword as well as different tenses and a number of different stems of that keyword. In the eyes of LSI algorithms this is all good news.

Looking At Your Link Profile

A link profile should no longer consist of thousands of links with the same anchor text (that of your primary keyword). There’s no reason to panic if you already have this kind of profile. Instead you should look at relevant and similar terms and improve your link profile by gaining links using these as your anchor text.

What It Offers Web Searchers

From the point of view of web searchers, LSI offers some distinct advantages over its earlier form of indexing. For example, LSI recognizes that the word “engine” in “search engine optimization” is not related to searches for terms like “steam engine” or “locomotive engine” and is instead related to Internet marketing topics. In theory, LSI results give a much more accurate page of results as well as providing a broader range of pages still geared towards a particular topic.

Where Google Leads, Others Generally Follow

It is widely acknowledged that Google is the search engine at the forefront of latent semantic indexing. On the whole they try to generate results pages that are literally filled with genuine, useful results and LSI certainly provides another string in their bow. Yahoo and MSN, for now, seem more than happy to go along with keyword specific indexing although Yahoo are known to look at singular and plural keyword variations as well as keyword stemming when judging keyword density.

The Effect On Your Website

How it affects the individual Webmaster is dependent on how they go about promoting their site already. If the pages are filled with natural content including keywords and keyword alternatives, and the link profile is similarly diversified for a number of related keywords then the fact is it won’t change very much. However, if all of your efforts have been concentrated, either on-page or off-page, with a single keyword then it’s time to readdress the balance.

Meta Tags Optimization

If you are familiar with Search Engine Optimization techniques, it is most likely that you must have already come across a lot of information on Meta Tags . In this article, we simply talk about what Meta Tags are, their importance, the important Meta Tags and useful tips on how to optimize your Meta Tags for a better ranking with the search engines. Improve your website ranking by availing our meta tags optimization , meta optimization , meta optimization services at affordable prices. Meta tag optimization is an important part of SEO.
What are Meta Tags ?Meta Tags are the information inserted in the <head> area of the HTML code of your web pages, where apart from the Title Tag , other information inserted is not visible to the person surfing your web page but is intended for the search engine crawlers. Meta Tags are included so that the search engines are able to list your site in their indexes more accurately. A typical Meta Tag would look as given below. Here is what this page contains; you can see this if you click on the view “Page source” from your browser menu.

<HEAD>
<TITLE>Creating Successful meta tags, list of all meta tags, title tag limit, href tag, meta tag analysis</TITLE>
<meta name=”description” content=”Successful Meta tags: What are meta tags? Meta tag analysis, list of all meta tags, title tag limit, creating meta tags, background meta tags. This article covers the most important aspects of meta tags in relation to website optimization or SEO. “>
<meta name=”keywords” content=”meta tags, meta tag refresh, meta tag optimization, meta tag analysis, meta tag keywords, what are meta tags, successful meta tags, using meta tags, html meta tags, common meta tags, author meta tag, alt tags, list of all meta tags, meta tags html, meta tag description, about meta tags”>
</HEAD>

Using Meta Tags in HTML is not necessary while making your web pages. There are many websites that don’t feel the requirement to use Meta Tags at all. In short Meta information is used to communicate information to the search engine crawlers that a human visitor may not be concerned with. Infoseek and AltaVista were the first major crawler based search engines to support Meta keywords Tag in 1996. Inktomi and Lycos too followed thereafter.

Why are Meta Tags used?

Meta Tags were originally designed to provide webmasters with a way to help search engines know what their site was about. This in turn helped the search engines decide how to rank the sites in their search results. Making Meta Tags is a simple process. As the competition increased, webmasters started manipulating this tool through spamming of keywords . In turn most search engines withdrew their support to Meta keywords Tag , which included Lycos and AltaVista. From being considered as one of the most reliable and important tool, Meta Tags are now often abused. In the present day scenario a vital feature that the Meta Tags provide to the websites is the ability to control, to a certain extent, how some search engines describe its web pages. Apart from this, Meta Tags also offer the ability to specify that a certain website page should not be indexed.

Using Meta Tags, however, provides no guarantee that your website page would rank highly in the search engine rankings. Due to the rampant abuse and manipulation of the Meta keywords Tag by webmasters, most search engines don’t support it anymore.

Types of Meta Tags

The more important Meta Tags are discussed below in detail. A list of several more Meta Tags
is given at the end of this article.

The Title Tag
The Title Tag is not a Meta Tag. However, since it’s a very important Tag, we thought it necessary to discuss it here. The Title Tag is an HTML code that shows the words that appear at the top title bar of your browser. The Title Tag is not displayed anywhere else on the page. It is these words or phrase that appear as the title of your page in the hyperlink listings on the search engine results.

The users in-turn click on this hyperlink to go to your website from Search Engine Results Page (SERP). Therefore, the significance of the Title Tag is evident as all search engines use the Title Tag to gather information about your site.

Syntax for the Title Tag is :

<HEAD>
<TITLE>Creating Successful meta tags, list of all meta tags, title tag limit, href tag, meta tag analysis</TITLE>
</HEAD>

Useful Tips for Writing the Title Tag
The Title Tag plays a vital role in determining your site’s search engine rankings. One needs to pay attention to the words that appear in your Title Tag and the order in which they appear. As the tag suggests, Title Tag announces the summary of the content appearing on the page. Think of it as a crisply worded sales phrase and include your most relevant and specific keywords or key phrases in the Title Tag describing the content of the page.

You could put the product or service you are offering in the Title Tag, instead of your firm or company’s name. For example, if you have a site offering kitchen appliances, instead of having a title like “www.cooking.com”, you could have a specific keyword rich Title Tag like “Bar Blenders: Waring Kitchenaid Hamilton beach Cuisinart bar blenders and blender”. This title would cover several keyword phrases like- bar blenders, bar blender, Waring bar blenders, Kitchenaid bar blenders, Hamilton Beach bar blenders, Cuisinart bar blenders, blender etc.

Most users search for specific products and services, not company names. If you do wish to include the site name, then you can add it at the end of the Title Tag. Having your most important keyword phrases towards the beginning of the Title Tag helps display your title in bold in the SERP (search engines results page). Title Tags are also the text that gets captured when users bookmark your page.

Google usually reads about 90 characters of your Title Tag. So, leverage this length to cover prominent keywords relevant to the page. A common mistake webmasters make is to have the same Title Tag throughout the site. You must leverage different Title Tags for each page and customizing it depending on the content theme of each page. Also read our detailed article on Title Tag Optimization.
Meta Description Tag
The Meta Description Tag is an HTML code that allows you to give a short and concise summary of your web page content. The words placed in this Meta Tag, are often used in the SERP, just below the Title Tag as a brief description of your page. In the Search Engine Results Pages, after reading the Title of the page, a user goes through the description of the page and decides whether she wants to go to your site or not. It is therefore important that your Meta Description Tag is nicely composed describing your page offering while enticing the user to click on your listing.

Syntax for Meta Description Tag is :

<HEAD>
<meta name=”description” content=”Successful Meta tags: What are meta tags? Meta tag analysis, list of all meta tags, title tag limit, creating meta tags, background meta tags. This article covers the most important aspects of meta tags in relation to website optimization or SEO.”>
</HEAD> Useful Tips for Writing Meta Description Tag In case you have not provided any Meta Description Tag to your web page, the search engines try to make one for you, often using the first few words of your web page or a text selection where the keyword phrases searched by the users appear. If the Search Engine makes up a description by picking up text from your page, it is not necessary that the generated description would do justice to your web page.

The Meta Description Tag should be written in such a way that it interests the user, thus tempting her to click on the link to your site and visit your web page.

The Meta Description Tag needs to be kept brief yet informative. About 25-30 words description should do fine. Keywords and key phrases should be included in the Meta Description Tag, though care should be taken not to repeat them too often. Like the Title Tag, the Meta Description Tag should be different and customized for each page depending on the content theme of each page. For more information on Meta Description Tag, read our other article on Meta Description Tag Optimization.
Meta Keywords Tag
Most search engines do not read the Meta Keywords Tag anymore. It is okay to ignore the Meta Keywords Tags . However, if you feel more comfortable using it, you can have about 15 important non-repetitive keywords in this Tag, separated by commas.

Syntax for Meta Keywords Tag is :

<HEAD>
<META name=”keywords” content=”Keyword phrases that describe your web page”>
</HEAD>
Meta Robots Tag
The Meta Robots Tag gives you the ability to specify whether search engines should index that page or follow the links appearing on that page. However, there is no need for using Meta Robots tag if one is already using detailed robots.txt file to block any specific indexing. Read our article on Working with the robots.txt file.

The various commands used under Meta Robots Tag are:

Index: allows the spider to index that page.

Noindex : instructs the spider not to index the page.

Follow : instructs the spider to follow the links from that page and index them.

Nofollow: instructs the spider not to follow links from that page for indexing.

<HEAD>
<meta name=”robots” content=”index,follow”>
<meta name=”robots” content=”noindex,follow”>
<meta name=”robots” content=”index,nofollow”>
<meta name=”robots” content=”noindex,nofollow”>
</HEAD>

Note: Use only one of the above given commands .

If you have not specified any Meta Robots Tag on a page, by default, the spiders understand that the page and all the links appearing on that page are open for indexing. Therefore, it makes more sense to use this Meta Tag in case you don’t want certain parts of your web page indexed.

Apart from these popular Meta Tags, there are few more Tags that are not so popular. These are :
Meta Refresh Tag
The Meta Refresh Tag is used to redirect a visitor from one page to another or refresh the same page in a specified time. Same page refresh comes in use when the page content is changing very quickly, for example if there is a running score of a game or stock prices updates happening on that page. When inserted into an HTML document, it causes the user’s browser to load the same or a new web page after a specified number of seconds. This Meta Tag has been manipulated and is sometimes abused as a tool to “bait and switch” a user from a doorway page.

To refresh the current page:

<HEAD>
<meta http-equiv=”refresh” content=”60″>
</HEAD>

Content = “60″ denotes refresh after 60 seconds

To redirect to another page:

<HEAD>
<meta http-equiv=”refresh” content=”2;url=”>
</HEAD>

Using a Meta Refresh Tag to redirect to another page is likely to get it dropped from search engines, since this method is tagged as a doorway page in search engine algorithms. Unless you have a compelling reason to use Meta Refresh Tag, do not use it.
Meta Author Tag
The Meta Author Tag identifies the author of a particular web page. Though not highly supported, its format usually includes the name, e-mail address of the Webmaster, name of the company or the Internet address.

<META name=”Author” content=”Author Information”>
Meta Distribution Tag
The Distribution Meta Tag identifies the level of distribution of your web page and how it should be further classified in context to the World Wide Web. The three forms of distribution supported by this Tag are:

‘Global’ specifies that your web page is for mass distribution.
‘Local’ lets you specify that your document is for local distribution only.
‘Internal Use or IU’ specifies that your document is not intended for public distribution.

Syntax for different Meta Distribution Tags:
(Use only one of the following)

<HEAD>
<META name=”Distribution” content=”Global”>
<META name=”Distribution” content= “Local”>
<META name=”Distribution” content= “IU”>
</HEAD>
Meta Revisit Tag
Historically, the Meta Revisit Tag was believed to specify how often a search engine should visit a site for re-indexing. Although not proven, webmasters could indicate the frequency of how often their content changes and is ready for a re-index by search engines. These days, much as one would like to, one cannot command the re-visit frequency of any search engine, unless you go the PFI route.

<HEAD>
<META name=”Revisit-After” content=”X Days”>
</HEAD>
Note: X indicates the number

List of other Meta Tags

1. Abstract Meta Tag : gives an overview of the entire web page in a sentence.

<HEAD>
<META name=”Abstract” content=”Abstract phrase”>
</HEAD>

2. Copyright Meta Tag : gives copyright information about your page, which you wish to disclose.

<HEAD>
<META name=”Copyright” content=”Copyright Statement”>
</HEAD>

3. Expires Meta Tag : tells the search engines when the content on your web site would expire.

<HEAD>
<META name=”Expires” content=”Mon, 22 Jan 2004 17:45:05 GMT”>
</HEAD>

4. Language Meta Tag : defines the language used on your web page.

<HEAD>
<META HTTP-EQUIV=”Content-Language” content=”EN”>
</HEAD>

5. Ratings Meta Tag : tells the search engines if your site is for general, restricted or adult audiences.

<HEAD>
<META name= “Rating” content= “General”>
</HEAD>

Process of website indexing by Google & other Search Engines

There is a lot of speculation about how search engines index websites. The topic is shrouded in mystery about exact working of search engine indexing process since most search engines offer limited information about how they architect the indexing process. Webmasters get some clues by checking their log reports about the crawler visits but are unaware of how the indexing happens or which pages of their website were really crawled.

While the speculation about search engine indexing process may continue, here is a theory, based on experience, research and clues, about how they may be going about indexing 8 to 10 billion web pages even so often or the reason why there is a delay in showing up newly added pages in their index. This discussion is centered around Google, but we believe that most popular search engines like Yahoo and MSN follow a similar pattern.

• Google runs from about 10 Internet Data Centers (IDCs), each having 1000 to 2000 Pentium-3 or Pentium-4 servers running Linux OS.
• Google has over 200 (some think ‘over 1000′) crawlers / bots scanning the web each day. These do not necessarily follow an exclusive pattern, which means different crawlers may visit the same site on the same day, not knowing other crawlers have been there before. This is what probably gives a ‘daily visit’ record in your traffic log reports, keeping web masters very happy about their frequent visits.
• Some crawlers’ jobs are only to grab new URLs (lets call them ‘URL Grabbers’ for convenience) – The URL grabbers grab links & URLs they detects on various websites (including links pointing to your site) and old/new URL’s it detects on your site. They also capture the ‘date stamp’ of files when they visit your website, so that they can identify ‘new content’ or ‘updated content’ pages. The URL grabbers respect your robots.txt file & Robots Meta Tags so that they can include / exclude URLs you want / do not want indexed. (Note: same URL with different session IDs are recorded as different ‘unique’ URLs. For this reason, session ID’s are best avoided, otherwise they can be misled as duplicate content. The URL grabbers spend very little time & bandwidth on your website, since their job is rather simple. However, just so you know, they need to scan 8 to 10 Billion URLs on the web each month. Not a petty job in itself, even for 1000 crawlers.
  
  
• The URL grabbers write the captured URL’s with their date stamps and other status in a ‘Master URL List’ so that these can be deep-indexed by other special crawlers.
• The master list is then processed and classified somewhat like -
  a) New URLs detected
  b) Old URLs with new date stamp
  c) 301 & 302 redirected URLs
  d) Old URLs with old date stamp
  e) 404 error URLs
  f) Other URLs
• The real indexing is done by (what we’re calling) ‘Deep Crawlers’. A deep crawler’s job is to pick up URLs from the master list and deep crawl each URL and capture all the content – text, HTML, images, flash etc.
• Priority is given to ‘Old URLs with new date stamp’ as they relate to already indexed but updated content. ‘301 & 302 redirected URLs’ come next in priority followed by ‘New URLs detected’. High priority is given to URLs whose links appear on several other sites. These are classified as ‘important’ URLs. Sites and URL’s whose date stamp and content changes on a daily or hourly basis are ’stamped’ as ‘News’ sites which are indexed hourly or even on minute-by-minute basis.
• Indexing of ‘Old URLs with old date stamp’ and ‘404 error URLs’ are altogether ignored. There is no point wasting resources indexing ‘Old URLs with old date stamp’, since the search engine already has the content indexed, which is not yet updated. ‘404 error URLs’ are URLs collected from various sites but are broken links or error pages. These URLs do not show any content on them.
• The ‘Other URLs’ may contain URLs which are dynamic URLs, have session IDs, PDF documents, Word documents, PowerPoint presentations, Multimedia files etc. Google needs to further process these and assess which ones are worth indexing and to what depth. It perhaps allocates indexing task of these to ‘Special Crawlers’.
• When Google ’schedules’ the ‘Deep Crawlers’ to index ‘New URLs’ and ‘301 & 302 redirected URLs’, just the URLs (not the descriptions) start appearing in search engines result pages when you run the search “site:www.domain.com” in Google.
  
  
• Since Deep Crawlers need to crawl ‘Billions’ of web pages each month, they take as many as 4 to 8 weeks to index even updated content. New URL’s may take longer to index.
• Once the Deep Crawlers index the content, it goes into their originating IDCs. Content is then processed, sorted and replicated (synchronized) to the rest of the IDCs. A few years back, when the data size was manageable, this data synchronization used to happen once a month, lasting for 5 days, called ‘Google Dance’. Nowadays, the data synchronization happens constantly, which some people call ‘Everflux’
• When you hit www.google.com from your browser, you can land at any of their 10 IDCs depending upon their speed and availability. Since the data at any given time is slightly different at each IDC, you may get different results at different times or on repeated searches of the same term (Google Dance).
• Bottom line is that one needs to wait for as long as 8 to 12 weeks, to see full indexing in Google. One should consider this as ‘cooking time’ in ‘Google’s kitchen’. Unless you can increase the ‘importance’ of your web pages by getting several incoming links from good sites, there is no way to speed up the indexing process, unless you personally know Sergey Brin & Larry Page, and have a significant influence over them.
• Dynamic URLs may take longer to index (sometimes they do not get indexed at all) since even a small data can create unlimited URLs, which can clutter Google index with duplicate content.

Summary & Advise:

1. Ensure that you have cleared all roadblocks for crawlers and they can freely visit your site and capture all URLs. Help crawlers by creating good interlinking and sitemaps on your website.
2. Get lots of good incoming links to your pages from other websites to improve the ‘importance’ of your web pages. There is no special need to submit your website to search engines. Links to your website on other websites are sufficient.
3. Patiently wait for 4 to 12 weeks for the indexing to happen.

Disclaimer: The actual functioning and exact architecture of the search engines may vary but in essence, this is what we believe they do.

Hello world!

Welcome to WordPress.com. This is your first post. Edit or delete it and start blogging!