- Code: Select all
192.168.xxx.xxx - - [15/Sep/2009:10:34:58 -0400] "GET /taxonomy/term/324/feed/ HTTP/1.0" 301 325 "-" "Yahoo-Newscrawler/3.9 (news-search-crawler at yahoo-inc dot com)"
Would something like the following prevent search engine bots from crawling a certain Drupal directory structure between a time frame (i.e. 6am until 11pm), but allowing everyone else to still access the feeds?
- Code: Select all
RewriteCond %{HTTP_USER_AGENT} ^.*(msnbot|googlebot|yahoo|newsbrain|rome) [NC]
RewriteCond %{QUERY_STRING} ^taxonomy/term/[^/]+/feed$
RewriteCond %{TIME_HOUR}%{TIME_MIN} >0600
RewriteCond %{TIME_HOUR}%{TIME_MIN} <2300
RewriteRule .* - [F]
-- M