Merge pull request #39 from mutantmonkey/robots.txt

add a file blacklist and add robots.txt
This commit is contained in:
Andrei Marcu 2015-10-04 21:21:06 -04:00
commit 46d6b7b98a
3 changed files with 16 additions and 0 deletions

3
static/robots.txt Normal file
View file

@ -0,0 +1,3 @@
User-agent: *
Allow: /$
Disallow: *