Theme: 
  Linux4U › System › /home › Block over zealous web crawlers Bottom
‹ No Topic | Next Topic ›
Page Index Toggle Pages: 1
Topic Tools
Normal Topic Block over zealous web crawlers (Read 6 times)
 
Root
Forum Administrator
*
Online


I love YaBB 2.6.12!

Posts: 4
Location: The Land of YaBB
Joined: 14.06.2026
Gender: Male
Block over zealous web crawlers
Today at 19:37:13
   
Report Post  
some crawlers just can't read, when they ignore robots.txt & keep crawling the same files, add this to .htaccess
those 3 are the worst but add more in the same format in that line


RewriteEngine On

RewriteCond %{HTTP_USER_AGENT} (Reflectionbot|PetalBot|DotBot) [NC]
RewriteRule .* - [F,L]
  
Back to top
WWW  
IP Logged
 
Page Index Toggle Pages: 1
Topic Tools
‹ No Topic | Next Topic ›
Bookmarks: Digg Facebook Linked in reddit X
Link to This Topic
 
  « Board Index ‹ Board Top  

Linux4U » Powered by YaBBForum 3.4
YaBB Forum Software © 2000-2026. All Rights Reserved.

Valid RSS Valid XHTML Valid CSS Powered by Perl

Page completed in 0.9876 seconds.