Break

Sega’s recent announcement that it will be incorporating mind-reading technology from NeuroSky into some games prompts the question: Is the age of brain control upon us? Not quite — the tech needs work, but many companies plan to incorporate it into games and gadgets.

Powered by ScribeFire.

Written by Matt Cutts and Maile Ohye

Our goal is to provide users the best search experience by presenting equitable and accurate results. We enjoy working with webmasters, and an added benefit of our working together is that when you make better and more accessible content, the internet, as well as our index, improves. This in turn allows us to deliver more relevant search results to users.

If, however, a webmaster chooses to buy or sell links for the purpose of manipulating search engine rankings, we reserve the right to protect the quality of our index. Buying or selling links that pass PageRank violates our webmaster guidelines. Such links can hurt relevance by causing:

- Inaccuracies: False popularity and links that are not fundamentally based on merit, relevance, or authority
- Inequities: Unfair advantage in our organic search results to websites with the biggest pocketbooks

In order to stay within Google's quality guidelines, paid links should be disclosed through a "rel=nofollow" or other techniques such as doing a redirect through a page which is robots.txt'ed out. Here's more information explaining our stance on buying and selling links that pass PageRank:

February 2003: Google's official quality guidelines have advised "Don't participate in link schemes designed to increase your site's ranking or PageRank" for several years.

September 2005: I posted on my blog about text links and PageRank.

December 2005: Another post on my blog discussed this issue, and said

Many people who work on ranking at search engines think that selling links can lower the quality of links on the web. If you want to buy or sell a link purely for visitors or traffic and not for search engines, a simple method exists to do so (the nofollow attribute). Google’s stance on selling links is pretty clear and we’re pretty accurate at spotting them, both algorithmically and manually. Sites that sell links can lose their trust in search engines.

September 2006: In an interview with John Battelle, I noted that "Google does consider it a violation of our quality guidelines to sell links that affect search engines."

January 2007: I posted on my blog to remind people that "links in those paid-for posts should be made in a way that doesn’t affect search engines."

April 2007: We provided a mechanism for people to report paid links to Google.

June 2007: I addressed paid links in my keynote discussion during the Search Marketing Expo (SMX) conference in Seattle. Here's a video excerpt from the keynote discussion. It's less than a minute long, but highlights that Google is willing to use both algorithmic and manual detection of paid links that violate our quality guidelines, and that we are willing to take stronger action on such links in the future.

June 2007: A post on the official Google Webmaster Blog noted that "Buying or selling links to manipulate results and deceive search engines violates our guidelines." The post also introduced a new official form in Google's webmaster console so that people could report buying or selling of links.

June 2007: Google added more specific guidance to our official webmaster documentation about how to report buying or selling links and what sort of link schemes violate our quality guidelines.

August 2007: I described Google's official position on buying and selling links in a panel dedicated to paid links at the Search Engine Strategies (SES) conference in San Jose.

September 2007: In a post on my blog recapping the SES San Jose conference, I also made my presentation available to the general public (PowerPoint link).

October 2007: Google provided comments for a Forbes article titled "Google Purges the Payola".

October 2007: Google officially confirmed to Search Engine Land that we were taking stronger action on this issue, including decreasing the toolbar PageRank of sites selling links that pass PageRank.

October 2007: An email that I sent to Search Engine Journal also made it clear that Google was taking stronger action on buying/selling links that pass PageRank.

We appreciate the feedback that we've received on this issue. A few of the more prevalent questions:

Q: Is buying or selling links that pass PageRank a violation of Google's guidelines? Why?
A: Yes, it is, for the reasons we mentioned above. I also recently did a post on my personal blog that walks through an example of why search engines wouldn't want to count such links. On a serious medical subject (brain tumors), we highlighted people being paid to write about a brain tumor treatment when they hadn't been aware of the treatment before, and we saw several cases where people didn't do basic research (or even spellchecking!) before writing paid posts.

Q: Is this a Google-only issue?
A: No. All the major search engines have opposed buying and selling links that affect search engines. For the Forbes article Google Purges The Payola, Andy Greenberg asked other search engines about their policies, and the results were unanimous. From the story:

Search engines hate this kind of paid-for popularity. Google's Webmaster guidelines ban buying links just to pump search rankings. Other search engines including Ask, MSN, and Yahoo!, which mimic Google's link-based search rankings, also discourage buying and selling links.

Other engines have also commented about this individually, e.g. a search engine representative from Microsoft commented in a recent interview and said

The reality is that most paid links are a.) obviously not objective and b.) very often irrelevant. If you are asking about those then the answer is absolutely there is a risk. We will not tolerate bogus links that add little value to the user experience and are effectively trying to game the system.

Q: Is that why we've seen some sites that sell links receive lower PageRank in the Google toolbar?
A: Yes. If a site is selling links, that can affect our opinion about the value of that site or cause us to lose trust in that site.

Q: What recourse does a site owner have if their site was selling links that pass PageRank, and the site's PageRank in the Google toolbar was lowered?
A: The site owner can address the violations of the webmaster guidelines and submit a reconsideration request in Google's Webmaster Central console. Before doing a reconsideration request, please make sure that all sold links either do not pass PageRank or are removed.

Q: Is Google trying to tell webmasters how to run their own site?
A: No. We're giving advice to webmasters who want to do well in Google. As I said in this video from my keynote discussion in June 2007, webmasters are welcome to make their sites however they like, but Google in turn reserves the right to protect the quality and relevance of our index. To the best of our knowledge, all the major search engines have adopted similar positions.

Q: Is Google trying to crack down on other forms of advertisements used to drive traffic?
A: No, not at all. Our webmaster guidelines clearly state that you can use links as means to get targeted traffic. In fact, in the presentation I did in August 2007, I specifically called out several examples of non-Google advertising that are completely within our guidelines. We just want disclosure to search engines of paid links so that the paid links won't affect search engines.

Q: I'm aware of a site that appears to be buying/selling links. How can I get that information to Google?
A: Read our official blog post about how to report paid links from earlier in 2007. We've received thousands and thousands of reports in just a few months, but we welcome more reports. We appreciate the feedback, because it helps us take direct action as well as improve our existing algorithmic detection. We also use that data to train new algorithms for paid links that violate our quality guidelines.

Q: Can I get more information?
A: Sure. I wrote more answers about paid links earlier this year if you'd like to read them. And if you still have questions, you can join the discussion in our Webmaster Help Group.

Written by Patrick Chapman, Search Quality Team

Our goal for the Webmaster Help Group is to be an authoritative source for accurate, friendly information and discussion. There are many terrific members of the Webmaster Groups community, and we're glad to know them all. In our English discussion group, a big Webmaster Central (WMC) thank-you to our comrades and fellow webmasters for their helpful knowledge and insight: webado, Phil Payne, JLH, cass-hacks, cristina, Sebastian, and dockarl, just to name a few.

Webado and cass-hacks both speak several languages -- thankfully, some of us do as well. We now have Googlers posting to the Google Webmaster Help Group in 12 languages! Here's a brief introduction of the Googlers, most of whom work together at our European headquarters in Dublin, in the Non-English groups (several have been posting for months, but we'd still like to give them an intro). :)


French Webmaster Help Group

Salut, I come from the French city Bordeaux where I spent most of my time, before I moved to Paris and then Dublin where I work now in Google Search Quality. When not in front of my computer, I like to go to the cinema, play chess and organize dinners
with my friends.
- Guide Google

Italian Webmaster Help Group

Ciao, my name is Stefano and I’m responsible for the Italian Webmaster Help Group. I work on search quality issues in Italian. I’m from Italy and have been living in Ireland for more than 2 years. I do love the multicultural environment you can find in Dublin and all the people from everywhere you get to know here, but sometimes it’s difficult to be so far away from my favorite football team, so now and then I really have to fly back home to get a bit of Serie A.
- Guida Google

German Webmaster Help Group

Grüss Gott! My name is Uli, and I post in the German Webmaster Help Group. I am originally from Germany but live in Ireland now. Unfortunately, I don't have my own website to show off. The German Help Group has grown into a big, vibrant community of very helpful and savvy webmasters, so if you speak German, go and check it out!
- Google Webmeister Guide

Spanish Webmaster Help Group

Hola! My name is Alvar and I'll be monitoring the Spanish Webmaster Help Group. Please join us if you speak a word or two in Spanish :-) More on the personal side, I don't own a portal or something like that but rather a tiny blog with nearly no visibility on the Internet, and I'm happy with that. I studied telecommunication engineering and my hobbies include soccer, foosball, table tennis, basically almost any other sport, traveling, photography, cinema, and technology, so I admit sitting in front of a computer can be counted as a hobby :-) Another important fact about me is that I'm from Barcelona, a city everyone should visit at least once in their life. What are you waiting for?
- Guía de Google para webmasters

Hola, I'm Rebecca. I studied to be a librarian but somehow along the way ended up being drawn into the digital side of information. So while I still snuggle up to books at night, computers take up most of my day. As for things I like to do (but wouldn't go so far as to call them hobbies…) I'm still pretty new to Dublin so I rather enjoy walking around until I'm lost and then trying to figure out how to get back home, and then when I get back home I like to play with my cat, best known for her fantastic Gollum impersonation when she gets riled up.
- Guía de Google

Dutch Webmaster Help Group

Hallo, I'm Andre. I'm very fond of Dutch music. But since living in Dublin for almost 2 years now, my taste for music has fused with the Irish sound. I like listening to live music in pubs, hanging out with the locals, have a pint or two and talking about upcoming gigs, artists, and all other topics that pass the day.
- André

Swedish Webmaster Help Group

Hejsan! My name is Hessam and I'm responsible for the Swedish Webmaster Help Group. I've been with Google for the last 2 years, working on search quality issues in Sweden. I'm originally from Sweden but moved to Dublin two years ago. My main interest is traveling and living in Dublin makes it easy to visit to all corners of Europe without blowing the budget. Thanks to cheap airlines, it takes merely a few hours from my door to the beer gardens of Munich, wine bars of Paris, ski slopes of Italy or beaches of Spain, depending on the mood. Looking forward to talking to you all!
- Google Webbansvarig Guide

Finnish Webmaster Help Group

Hei, I'm Anu and I work in the Search Quality team. I'm originally from Finland but these days I hold my umbrella high in Dublin. When I'm not online, you can catch me cycling (be it one or two wheels), playing virtual tennis or at the airport. I've been bitten by the travel bug, and try to see as many places near and far as possible. Besides all things webmaster related, I also have an interest in foreign languages, books and films. I look forward to meeting you in the Finnish Webmaster Help Group!
- Googlen Web-ylläpidon Ryhmän Opas

Polish Webmaster Help Group

Cześć, I'm Guglarz (it stands for Googler in Polish), the Googler on the Polish Webmaster Help Group. I was lucky to grow up in the city of Kraków, Poland's most beautiful city and the place where Google recently opened a research center. I've been with Google for two years now and I still love this job as much as I did the very first day. It's my favorite hobby activity in fact. If I don't work, I like to keep myself busy with general aviation, running or bowling, a sport I recently found out I was talented in. ;-)

I discovered my passion for the Internet early in school and after graduating in information science studies I was looking for a challenging position in the industry, although after the year 2000 crash there was little hope for that. It took me a couple of jobs in the established industries and some traveling around the globe before I found my dream job here at Google.

Ever since I started helping on the Polish Webmaster Help Group, it has been growing rapidly, both in terms of user numbers as much as in terms of the activity. It's really exciting to see how Polish webmasters help each other and make the web a more interesting place. Three group members, Cezary Lech, Umik and krzys in particular made an effort to vitalize the community in its early days. I'd like to say dziękuję (thank you in Polish) and please keep up the great spirit - thumbs up!
- Guglarz

Portuguese Webmaster Help Group

Olá, my name is Pedro. I'm Portuguese and I'm part of the Search Quality team. I've been working at Google since March 2006 mostly focused on the Portuguese language markets. I grew up in Tavira, a small town in the Algarve region – South of Portugal – and I always had a nerdy side, playing with computers since my very early days when memory meant 128KB. Most of my interests fall on my origins, I enjoy sailing and scuba diving, music is also on my top list. I'm based in the European Headquarters – Dublin office, and I'll be looking to strengthening contact with Portuguese webmasters (non Portuguese are also welcome).
- Ajuda a Webmasters do Google

Russian Webmaster Help Group

Привет! My name is Oxana and I come from Moldova, a teeny tiny country in Eastern Europe. My background is in mathematics and computer sciences and I have worked as a web developer for more than 7 years now. Of course I have a web site, but it features only an, unfortunately, eternal "under construction" message and a hope for a better future. :) I love to read and to travel, and at the moment I am a helpless wannabe photographer. Also, I'm a passionate WoW player and soon I'll become the best Warlock Orc on this side of Kalimdor! When I'm a grown-up person I work at Google on the Search Quality team and I primarily support the Russian market.
- Оксана

Danish Webmaster Help Group

Hej, my name is Jonas, and I am from Copenhagen, the wonderful capital of beautiful Denmark. I've been a webmaster of a blog since 2001, where I still drop a few lines every now and then. I am a jack of many trades, with a background in human geography and communication, design, and media. I've done some authoring for the web, but mostly administrative backends in PHP/MySQL, so they are not that interesting. I've been active on Usenet for awhile as well, and spent many hours there, getting smarter with the help of others.

I've been with Google for a couple of years now, working exclusively with search quality and I am now helping out in the Danish Webmaster Help Group. Looking forward to seeing you there (:
- GoogleGuide

Written by David Sha, Webmaster Tools Team

Welcome Go Daddy webmasters to the Google Webmaster Tools family! Today, we're announcing that Go Daddy, the world's largest hostname provider in the web hosting space, is working with us as a pilot partner so that their customers can more easily access Google Webmaster Tools. Go Daddy is a great partner, and we hope to educate more webmasters on how to make their site more search engine-friendly.

Go Daddy users will now see our link right in their hosting control center, and can launch Google Webmaster Tools directly from their hosting account. And Go Daddy makes the Google Webmaster Tools account creation process faster by adding the site, verifying the site, and submitting Sitemaps on behalf of hosting customers. Our tools show users how Google views their site, give useful stats like queries and links, diagnose problems, and share information with us in order to improve their site's visibility in search results.

As a continuation of these efforts, we look forward to working with other web hosting companies to add Google Webmaster Tools to their products soon.

And in case you're wondering, Webmaster Tools will stay 100% the same for current users. If you have questions or suggestions about our partnership with Go Daddy, let us know in our Webmaster community discussion groups.

Written by Michael Wyszomierski and Greg Grothaus, Search Quality

Many webmasters have discovered the advantages of using Ajax to improve the user experience on their sites, creating dynamic pages that act as powerful web applications. But, like Flash, Ajax can make a site difficult for search engines to index if the technology is not implemented carefully. As promised in our post answering questions about Server location, cross-linking, and Web 2.0 technology, we've compiled some tips for creating Ajax-enhanced websites that are also understood by search engines.

How will Google see my site?

One of the main issues with Ajax sites is that while Googlebot is great at following and understanding the structure of HTML links, it can have a difficult time finding its way around sites which use JavaScript for navigation. While we are working to better understand JavaScript, your best bet for creating a site that's crawlable by Google and other search engines is to provide HTML links to your content.

Design for accessibility

We encourage webmasters to create pages for users, not just search engines. When you're designing your Ajax site, think about the needs of your users, including those who may not be using a JavaScript-capable browser. There are plenty of such users on the web, including those using screen readers or mobile devices.

One of the easiest ways to test your site's accessibility to this type of user is to explore the site in your browser with JavaScript turned off, or by viewing it in a text-only browser such as Lynx. Viewing a site as text-only can also help you identify other content which may be hard for Googlebot to see, including images and Flash.

Develop with progressive enhancement

If you're starting from scratch, one good approach is to build your site's structure and navigation using only HTML. Then, once you have the site's pages, links, and content in place, you can spice up the appearance and interface with Ajax. Googlebot will be happy looking at the HTML, while users with modern browsers can enjoy your Ajax bonuses.

Of course you will likely have links requiring JavaScript for Ajax functionality, so here's a way to help Ajax and static links coexist:
When creating your links, format them so they'll offer a static link as well as calling a JavaScript function. That way you'll have the Ajax functionality for JavaScript users, while non-JavaScript users can ignore the script and follow the link. For example:


Note that the static link's URL has a parameter (?foo=32) instead of a fragment (#foo=32), which is used by the Ajax code. This is important, as search engines understand URL parameters but often ignore fragments. Web developer Jeremy Keith labeled this technique as Hijax. Since you now offer static links, users and search engines can link to the exact content they want to share or reference.

While we're constantly improving our crawling capability, using HTML links remains a strong way to help us (as well as other search engines, mobile devices and users) better understand your site's structure.

Follow the guidelines

In addition to the tips described here, we encourage you to also check out our Webmaster Guidelines for more information about what can make a site good for Google and your users. The guidelines also point out some practices to avoid, including sneaky JavaScript redirects. A general rule to follow is that while you can provide users different experiences based on their capabilities, the content should remain the same. For example, imagine we've created a page for Wysz's Hamster Farm. The top of the page has a heading of "Wysz's Hamster Farm," and below it is an Ajax-powered slideshow of the latest hamster arrivals. Turning JavaScript off on the same page shouldn't surprise a user with additional text reading:

Wysz's Hamster Farm -- hamsters, best hamsters, cheap hamsters, free hamsters, pets, farms, hamster farmers, dancing hamsters, rodents, hampsters, hamsers, best hamster resource, pet toys, dancing lessons, cute, hamster tricks, pet food, hamster habitat, hamster hotels, hamster birthday gift ideas and more!

A more ideal implementation would display the same text whether JavaScript was enabled or not, and in the best scenario, offer an HTML version of the slideshow to non-JavaScript users.

This is a pretty advanced topic, so please continue the discussion by asking questions and sharing ideas over in the Webmaster Help Group. See you there!

Written by Ríona MacNamara, Webmaster Tools Team

Common feedback we hear from webmasters is that you want us to improve the freshness of the data in Webmaster Tools. Understood. :) We've increased the update frequency for your verified sites' data, such as crawl, index, and search query stats. Much of this data depends on the content of your site. If your content doesn't change very often, or if you're not getting new links to your site, you may not see updates to your data every time you sign in to Webmaster Tools.

Please continue to post your Suggestions & feature requests in the Webmaster Help Group. It's one of our most important sources of feedback from the webmaster community. We seriously take it seriously.

Wordpress has the ability to embed images into postings. You should give your site a little pizazz by adding images to your posts. However, the default behavior of these images is to just sit out there on their own. Not particularly appealing.

By adding a simple CSS attribute to your tag you can improve the placement of the images by having your blog text wrap around your images.

Here are two examples. The first example places an image in line with your post, between paragraphs.

Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Quisque in metus. Mauris odio ante, lacinia in, feugiat ut, fringilla ac, sem. Quisque consequat. Mauris ultrices pede ac nunc. Nunc ut nisi. Curabitur urna odio, iaculis at, tincidunt at, eleifend nec, est. Praesent feugiat eros non ante. Nunc a libero. Curabitur sagittis, velit ac sodales mollis, lacus mi bibendum neque, id sagittis risus ligula nec metus. Duis nec purus vitae magna scelerisque mattis.

Donec eget leo. In tincidunt interdum arcu. Maecenas viverra, metus ut iaculis tempus, erat risus rhoncus est, ultrices porttitor elit est a magna. Cras interdum pede suscipit urna. Ut tortor nulla, tempus a, malesuada eu, tempus non, augue. Aenean sollicitudin dignissim urna. Aenean in nisi. Morbi mollis. Nullam sagittis, libero vel venenatis condimentum, nisi eros facilisis arcu, id vehicula tellus orci ac diam. Mauris condimentum, ligula in tempus scelerisque, sem erat luctus lectus, nec posuere lectus libero id massa. Morbi a orci pharetra nulla congue blandit. Nam eget leo. Etiam aliquam, odio malesuada laoreet accumsan, nisl erat ornare eros, id blandit dui tortor sed justo. Donec nibh. In non lorem vitae sapien convallis vulputate.

This second example, places the image in the same location but I add an attribute to the image anchor.

Integer a erat. Sed sed ante. Nam eros justo, consectetuer id, fringilla vel, mollis in, lacus. Pellentesque habitant morbi tristique senectus et netus et malesuada fames ac turpis egestas. Donec vel lorem. Pellentesque vitae quam. Phasellus tellus. Vestibulum ut nunc. Phasellus quam. Curabitur felis velit, suscipit lacinia, posuere ut, nonummy vitae, odio. Ut rhoncus lacus sed eros. Fusce lacinia velit eu nulla semper aliquet. Proin molestie quam eget justo. Ut egestas volutpat nisl. Suspendisse eu diam non libero varius tempus. Nulla condimentum enim et eros. Vivamus ac tortor ac ipsum gravida fermentum. Suspendisse potenti. Quisque diam.

Vestibulum ante ipsum primis in faucibus orci luctus et ultrices posuere cubilia Curae; Morbi a massa ut sem dignissim imperdiet. Vivamus viverra tempus eros. Aliquam mattis porttitor justo. Aliquam ligula neque, sagittis a, pretium at, dictum ac, libero. Quisque tempor mauris at risus facilisis lobortis. Etiam tempus, leo eget vulputate convallis, mauris nisl condimentum nibh, in facilisis nisl risus scelerisque quam. Nunc nec augue et risus tristique feugiat. Vestibulum ut metus. Sed nibh.



In the second example, my image tag looks like this.

< img src="sampleimage.jpg"
alt="20070704-IMG_2038"
width="160"
height="240"
border="0"
class="alignleft" />

Note that I have class="alignleft" The alignleft class is defined in the default Wordpress template. When you apply this to an image, it will push the image to the left side and run the text around it to the right. The Wordpress style sheet also defines an alignright and aligncenter class which does the obvious.

This should give you enough information to add the finishing touches on your blog entry. Don’t let your images stand alone anymore.

Technorati Tags: Wordpress, Tutorial, CSS

Written by Ben D'Angelo, Search Quality Team

First, a brief recap: In late 2005, we started emailing webmasters to let them know that their site is violating our Webmaster Guidelines and that we have temporarily removed some of their pages from our index. A few months ago we put these emails on hold due to a number of spoofed messages being sent from outside Google, primarily to German webmasters. Then, in mid-July, we launched Message Center in our webmaster console, which allows us to send messages to verified site owners.

While Message Center is great for verified site owners, it doesn't allow us to notify webmasters who aren't registered in Google's Webmaster Tools. For this reason, we plan to resume sending emails in addition to the Message Center notifications. Please note that, as before, our emails will not include attachments. Currently, the Message Center won't keep messages waiting if you haven't previously registered, but we hope to add that feature in the next few months. We'll keep you posted as things change.

Written by Trevor Foucher, Webmaster Tools Team

I'm happy to announce that Webmaster Tools is expanding support for webmasters outside of the English-speaking world, by supporting Internationalizing Domain Names in Applications (IDNA). IDNA provides a way for site owners to have domains that go beyond the domain name system's limitations of English letters and numbers. Prior to IDNA, Internet host names could only be in the 26 letters of the English alphabet, the numbers 0-9, and the hyphen character. With IDNA support, you'll now be able to add your sites that use other character sets, and organize them easily on your Webmaster Tools Dashboard.

Let's say you wanted to add http://北京大学.cn/ (Peking University) to your Webmaster Tools account before we launched IDNA support. If you typed that in to the "Add Site" box, you'd get back an error message that looks like this:



Some webmasters discovered a workaround. Internally, IDNA converts nicely encoded http://北京大学.cn/ to a format called Punycode, which looks like http://xn--1lq90ic7fzpc.cn/. This allowed them to diagnose and view information about their site, but it looked pretty ugly. Also, if they had more than one IDNA site, you can imagine it would be pretty hard to tell them apart.



Since we now support IDNA throughout Webmaster Tools, all you need to do is type in the name of your site, and we will add it correctly. Here is what it looks like if you attempt to add http://北京大学.cn/ to your account:



If you are one of the webmasters who discovered the workaround previously (i.e., you have had sites listed in your account that look like http://xn--1lq90ic7fzpc.cn/), those sites will now automatically display correctly.

We'd love to hear your questions and feedback on this new feature; you can write a comment below or post in the Google Webmaster Tools section of our Webmaster Help Group. We'd also appreciate suggestions for other ways we can improve our international support.

Posted by John Blackburn, Webmaster Tools and Matt Dougherty, Search Quality

Reporting is better, too. You'll now be told of multiple problems per line if they exist, unlike earlier versions which only reported the first problem encountered. And we've made other general improvements to analysis and validation.

Imagine that you're responsible for the domain www.example.com and you want search engines to index everything on your site, except for your /images folder. You also want to make sure your sitemap gets noticed, so you save the following as your robots.txt file:

You visit Webmaster Central to test your site against the robots.txtanalysis tool using these two test URLs:

Earlier versions of the tool would have reported this:



The improved version tells you more about that robots.txt file:





We also want to make sure you've heard about the new unavailable_after META
tag announced by Dan Crow on the Official Google Blog a few weeks ago. This allows for a more dynamic relationship between your site and Googlebot. Just think, with www.example.com, any time you have a temporarily available news story or limited offer sale or promotion page, you can specify the exact date and time you want specific
pages to stop being crawled and indexed.

Let's assume you are running a promotion that expires at the end of 2007. In the headers of page www.example.com/2007promotion.html, you would use the following:

<META NAME="GOOGLEBOT"
CONTENT="unavailable_after: 31-Dec-2007 23:59:59 EST">

The second exciting news: the new X-Robots-Tag directive, which adds Robots Exclusion Protocol (REP) META tag support for non-HTML pages! Finally, you can have the same control over your videos, spreadsheets, and other indexed file types. Using the example above, let's say your promotion page is in PDF format. For www.example.com/2007promotion.pdf, you would use the following:

X-Robots-Tag: unavailable_after: 31 Dec
2007 23:59:59 EST

Remember, REP META tags can be useful for implementing noarchive, nosnippet, and now unavailable_after tags for page-level instruction, as opposed to robots.txt, which is controlled at the domain root. We get requests from bloggers and webmasters for these features, so enjoy. If you have other suggestions, keep them coming. Any questions? Please ask them in the Webmaster Help Group.