Can We Resubmit / Force Robots.txt To Be Updated ?

If you are new to ‘webmaster’ing, one of the first things you should be aware of – is of robots.txt. For a starter who is figuring what robots.txt is, these two links will give you fair idea about it: link 1 and link 2.

When you discuss the¬†advantages of robots.txt, different people have different views: some think it’s a very good idea to have robots.txt on their site, while others opine it’s not all that¬†necessary. As they say, it’s to each his own. If you believe that having a robots.txt file on your domain will give you an edge, then go ahead and direct the Search Engine Bots what they should index and what they should exclude. If you are not concerned about everything on your server being indexed, then don’t bother using one. If you ask my personal opinion, I’d say, just for the heck of it have one, even if you want everything on your server to be indexed.

=> This link will give you some insight to advantages of robots.txt.

If you are wondering what possibly can go wrong with one ‘robots.txt’ file, then continue reading:

I had never put up a good robots.txt and one fine day I decided to put up one – Mistake !

Instead of carefully handcrafting a good robots.txt file, I decided to just randomly pick some code which was freely floating around on web, saved it in a text file named robots.txt and uploaded it onto the server- Big Mistake !!

So here’s what exactly happened:

Techbuzz is built on php, and the code which I found was blocking all the ‘.php’ files, as it was custom built for ‘.html’ sites. I overlooked this factor and went ahead. And the price I had to pay for it was: I lost more than 50% of my traffic in less than 24 hours !!

Luckily, Chibcha and Webado pitched in and their suggestions helped me regain a significant portion of the lost traffic.

Lesson’s I learnt from this mistake:

  • Don’t mess with robots.txt without complete knowledge ! Read the links given above, they’ll give significant idea about robots.txt, and then create a tailor made robots.txt to suit your website requirements.
  • There is no way to Re-Submit your robots.txt file.
  • You cannot force update your robots.txt file once they have been crawled.
  • Most importantly- you do not SUBMIT robots.txt file, instead the bots read them during every crawl-cycle!
  • If you end up screwing your robots.txt file, then the only best thing you can do is- create a new robots.txt file which is good to save your site, upload it to replace the older robots.txt and wait till the spiders crawl your site and update their database with the latest file. There is no fixed time when they crawl your site (it was around 24hrs in my websites case and another 24 hours before changes were noticeable), but there is no set time scale as such ! You may be lucky to get the traffic back, or just a bit unlucky to loose the traffic (I was lucky to regain the positions), but you really have to research what’s the wait period before you get back all your traffic.
  • The robots.txt file behave very differently than I expected them to: You’ll be surprised to see that most of your URL’s will still be indexed even if you ban the Robots.txt from crawling your website. It only prevents the content found on those URLs from being indexed (Ref: Webado) !

I guess by now you have enough information that’ll protect your website from any ‘robots.txt’ malfunction.

I’m no expert, but just an experienced webmaster. If there are any doubts regarding ‘robots.txt‘ please do drop a comment. I’ll answer if I have any idea about it, or point you to a solution at the least!

My Recent WordPress Blog Posts Are Not Getting Indexed In Google

Well if you are a regular blogger, then Google Bot’s crawl rate on your blog will be pretty high. Which inturn means that your blog posts get indexed very soon.

Any new blog post in my blog, from my observation, usually gets indexed in less than 5 mins. However, since 20th October 2010, none of my blog posts were getting indexed. At first, I was a bit worried, only to realize that it was not some random problem with my blog alone, in fact a lot of other popular blogs are having similar problems.

I spoke to Amit Bhawani and he too confirmed about struggling with the same problem.

Read this post on his blog, which gives more insight into the problem: Google Indexing Problem