添加AhrefsBot和MJ12bot的爬虫禁止访问网站

robots.txt

@@ -32,6 +32,10 @@ User-agent: googlebot-mobile
 Disallow: /
 User-agent: yahoo-mmcrawler
 Disallow: /
+User-agent: AhrefsBot
+Disallow: /
+User-agent: MJ12bot
+Disallow: /
 User-agent: yahoo-blogs/v3.9
 Disallow: /
 User-agent: psbot