What is the robots.txt file?
The robots.txt file is a standard text file that tells search engine crawlers which pages they can access, scrape, and ultimately list in their search engine results.
The file is listed on the root URL of your site, domain.com/robots.txt, and can easily be added to your Django project.
Be sure to use the correct casing and spelling for anything added to the robots.txt file or it will not work correctly.
A simple robots.txt file
User-agent: *
Disallow: /
This file will disallow all robots.
Create a robots.txt file
You have two options about where to create the robots.txt file in your Open edX project:
In your base templates of LMS at the following path,
edx-platform/lms/templates/robots.txtIn your theme directory (if you have enabled comprehensive theming),
edx-platform/themes/my-theme/lms/templates/robots.txt
Either way is correct but it is preferable to enable comprehensive theming and follow the second option.
Add a URL path to robots.txt
Go to the edx-platform/lms/urls.py file and add the following code:
from django.views.generic.base import TemplateView
# For koa or earlier releases
urlpatterns = [
url(r'^robots.txt', TemplateView.as_view(template_name="robots.txt", content_type="text/plain")),
]
# For lilac or later releases
urlpatterns = [
path("robots.txt", TemplateView.as_view(template_name="robots.txt", content_type="text/plain")),
]
Restart your LMS service and you can see the robots.txt file at: domain.com/robots.txt