django serving robots.txt efficiently

Question

Here is my current method of serving robots.txt

url(r'^robots\.txt/$', TemplateView.as_view(template_name='robots.txt',
                                            content_type='text/plain')),

I don't think that this is the best way. I think it would be better if it were just a pure static resource and served statically. But the way my django app is structured is that the static root and all subsequent static files are located in

http://my.domain.com/static/stuff-here

Any thoughts? I'm amateur at django but

 TemplateView.as_view(template_name='robots.txt',
                                  content_type='text/plain')

looks a lot more resource consuming than just a static call to my static directory which is served on nginx.

Answer 1

Yes, robots.txt should not be served by Django if the file is static. Try something like this in your Nginx config file:

location  /robots.txt {
    alias  /path/to/static/robots.txt;
}

See here for more info: http://wiki.nginx.org/HttpCoreModule#alias

Same thing applies to the favicon.ico file if you have one.

The equivalent code for Apache config is:

Alias /robots.txt /path/to/static/robots.txt