Introduction to Robots.txt for Non-Technical People

You've heard the name before, "robots.txt," but what is it? And why is it important?

This 15-minute mini-lesson is intended as a very basic introduction to "robots.txt" and will explain

  • What robots, spiders, and bots are, and how Google and Bing rely on them to make their search engines work
  • The role of a search engine "index"
  • Why your website must be Search Engine Friendly before you can work on your SEO
  • The purpose of a robots.txt and how it can govern the behavior of search engine spiders/robots
  • How you can see whether you might have a problem with the robots.txt on your website
  • How to use the special "site" operator in search
  • The two methods for checking out your robots.txt
  • How to use Google's robots.txt checker
  • Basic robots.txt syntax (nothing too complicated)

Your Instructor

Ross Barefoot
Ross Barefoot

Ross Barefoot has 19 years of experience in website programming and development, and 13 years of experience in Search Engine Optimization and Marketing (SEO/SEM). Ross is a SEA Certified Master Level Instructor with the Search Engine Academy and serves on its board of directors and educational committee. Ross has been with Horizon Web Marketing since 2012 as chief SEO strategist and trainer, and in 2014 took on the role of CTO. Ross uses his industry expertise to set client strategies, troubleshoot difficult SEO issues, and keep our staff trained and up to date on cutting-edge SEO techniques so we deliver the best possible results for our clients.

For upcoming live workshop dates, see

Course Curriculum

  Instructional Video
Available in days
days after you enroll

Frequently Asked Questions

When does the course start and finish?
The course starts now and never ends! It is a completely self-paced online course - you decide when you start and when you finish.
How long do I have access to the course?
How does lifetime access sound? After enrolling, you have unlimited access to this course for as long as you like - across any and all devices you own.
What if I am unhappy with the course?
We would never want you to be unhappy! If you are unsatisfied with your purchase, contact us in the first 30 days and we will give you a full refund.

Get started now!