Blocking Bots in Apache Using htaccess

TL;DR: Two lines of .htaccess block every bad bot you care to name on Apache: one RewriteCond on %{HTTP_USER_AGENT} with the names joined by |, one RewriteRule that returns 403. That is the whole technique, and it has not changed since 2014. What has changed is what belongs in the list: half of my original entries were search engines, SEO crawlers, archives, and HTTP libraries I would not block today, and the "SQL injection protection" block that used to sit under it in my boilerplate was theater. Both are covered below, along with where AI crawlers fit.

This started when an application of mine became the victim of bot spam. Imperva’s annual Bad Bot Report has put automated traffic at roughly half of everything on the web for years; most of it is inconsequential and can be refused, or pointed at a cache, without touching your rankings. I chose to refuse it, and to key on the user agent rather than the address, because the bots that matter rotate IPs and the user agent is the one thing they tend to keep. The IIS version of this post makes the same argument with URL Rewrite.

How do you block a user agent in .htaccess?

Most examples you find do it one bot per line:

RewriteCond %{HTTP_USER_AGENT} ^Snoopy [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^VB\ Project [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^WWW::Mechanize [NC,OR]
RewriteCond %{HTTP_USER_AGENT} RPT-HTTPClient [NC]
RewriteRule .* - [R=403,L]

That grows the file by a line per bot and, worse, invites the next person to add a fifth condition and forget the OR. The same thing in two lines, with the names joined by the regex alternation character:

RewriteEngine on
RewriteCond %{HTTP_USER_AGENT} (^$|HTTrack|WebZIP) [NC]
RewriteRule .* - [F,L]

Three details. ^$ inside the group is the regex for an empty string: I do not serve pages to anything that will not identify itself, and in practice the only things hitting these applications with no user agent were security scanners someone had left running. [NC] makes the match case-insensitive, and because nothing anchors the other alternatives it is a substring match, so be specific: fire matches Firefox. [F] is a 403 without a redirect, which is what you want; the R=403 form in the first example works but reads as a redirect that isn’t one. My original boilerplate wrapped the group in ^.*(...).*$. That does nothing a bare group doesn’t, and it is slower.

The original two-hundred-name version is at the bottom of this post. Adding a name is one more alternation, which is the entire reason I moved to this form: it was two lines doing the work of 232.

Does this work on Apache 2.4?

Yes, unchanged; mod_rewrite syntax survived the 2.2 to 2.4 transition intact. Two things to know. .htaccess only works if the enclosing <Directory> has AllowOverride FileInfo or All, and Apache reads the file on every request, walking every parent directory. If you control the server, put the same two lines in the virtual host and set AllowOverride None; it is the same rule with the per-request file read removed. Second, on 2.4 you can express the block without mod_rewrite at all:

<If "%{HTTP_USER_AGENT} =~ /(^$|HTTrack|WebZIP)/i">
  Require all denied
</If>

Either works. I still use the rewrite form because it is the one the next person will recognise.

The part of my old boilerplate I would delete

The .htaccess file this post used to embed had a second section under the bot rule: thirty RewriteCond %{QUERY_STRING} lines labelled “SQL Injection Protection”, matching union select, base64_encode, ../, and, in one line, any query string containing a bracket, parenthesis, angle bracket, or apostrophe. I copied it from somewhere in 2014 and it looked like defense.

It isn’t. A rule that 403s every query string with a ( in it breaks search boxes, and a rule that matches the literal text select after a quote stops nobody who has read a tutorial on encoding. Query-string regexes are not a WAF, and the WAF is not a substitute for parameterised queries either. If you want the request-inspection layer, run ModSecurity with the OWASP Core Rule Set, which is maintained by people who track the bypasses; if you want the application to be safe, that is a code review, not an .htaccess. I have taken the section out of the boilerplate, and I would take it out of yours.

Which entries would I not block in 2026?

The 2014 list blocks Yahoo’s Slurp, msnbot, Baiduspider, Yandex, Naver, and Seznam, which are search engines; AhrefsBot, MJ12bot, DotBot, and rogerbot, which are the SEO tools you or your agency pay for; ia_archiver and commoncrawl, which are the Wayback Machine and the corpus most language models train on; and curl, Wget, urllib, libwww-perl, PHP/, and Java/, which are the default user agents of every script that never set one, including your own health checks. The IIS post carries the full table with what each group costs you. The rest of the list is site-rippers, download managers, email harvesters, and dead crawlers, and I would still block every one of them. Read the list before you paste it, which is more than I did.

What about AI crawlers?

None of them existed when this was written, and on a content site they are now the bulk of the new automated traffic. GPTBot, ClaudeBot, PerplexityBot, CCBot, and Applebot-Extended identify themselves and honor robots.txt, so robots.txt is where that decision belongs; the rewrite rule is for the ones reported to ignore it, with Bytespider the usual name. I block none of them here. This site publishes an llms.txt and wants to be cited, and a rule that refuses ClaudeBot would be working against the reason the writing exists. If you sell the content itself, decide the other way, but decide it.

Questions people search for

How do I block bots by user agent in .htaccess?

One RewriteCond on %{HTTP_USER_AGENT} with the bot names joined by | inside a group, flagged [NC], followed by RewriteRule .* - [F,L]. Adding a bot is one more alternation, not another line.

Does blocking bots in .htaccess hurt SEO?

Only if the list includes search engines, and the popular copy-paste lists do: Slurp, msnbot, Baiduspider, Yandex, and Naver are all in mine from 2014. Blocking scrapers, download managers, and email harvesters has no effect on rankings.

Should I block bots with .htaccess or robots.txt?

Both, for different bots. robots.txt is a request that well-behaved crawlers honor and bad bots ignore. A rewrite rule refuses the request before your application runs, so the scrapers on this list never reach it.

The full 2014 list

Every name the original rule matched, joined by | into one group at the time. Duplicates removed; nothing else edited. A starting point, not a rule.

^$
EasouSpider
Add Catalog
PaperLiBot
Spiceworks
ZumBot
RU_Bot
Wget
Java/1.7.0_25
Slurp
FunWebProducts
80legs
Aboundex
AcoiRobot
Acoon Robot
AhrefsBot
aihit
AlkalineBOT
AnzwersCrawl
Arachnoidea
ArchitextSpider
archive
Autonomy Spider
Baiduspider
BecomeBot
benderthewebrobot
BlackWidow
Bork-edition
Bot mailto:craftbot@yahoo.com
botje
catchbot
changedetection
Charlotte
ChinaClaw
commoncrawl
ConveraCrawler
Covario
crawler
curl
Custo
data mining development project
DigExt
DISCo
discobot
discoveryengine
DOC
DoCoMo
DotBot
Download Demon
Download Ninja
eCatch
EirGrabber
EmailSiphon
EmailWolf
eurobot
Exabot
Express WebPictures
ExtractorPro
EyeNetIE
Ezooms
Fetch
Fetch API
filterdb
findfiles
findlinks
FlashGet
flightdeckreports
FollowSite Bot
Gaisbot
genieBot
GetRight
GetWeb!
gigablast
Gigabot
Go-Ahead-Got-It
Go!Zilla
GrabNet
Grafula
GT::WWW
hailoo
heritrix
HMView
houxou
HTTP::Lite
HTTrack
ia_archiver
IBM EVV
id-search
IDBot
Image Stripper
Image Sucker
Indy Library
InterGET
Internet Ninja
internetmemory
ISC Systems iRc Search 2.1
JetCar
JOC Web Spider
k2spider
larbin
LeechFTP
libghttp
libwww
libwww-perl
linko
LinkWalker
lwp-trivial
Mass Downloader
metadatalabs
MFC_Tear_Sample
Microsoft URL Control
MIDown tool
Missigua
Missigua Locator
Mister PiX
MJ12bot
MOREnet
MSIECrawler
msnbot
naver
Navroad
NearSite
Net Vampire
NetAnts
NetSpider
NetZIP
NextGenSearchBot
NPBot
Nutch
Octopus
Offline Explorer
Offline Navigator
omni-explorer
PageGrabber
panscient
panscient.com
Papa Foto
pavuk
pcBrowser
PECL::HTTP
PHP/
PHPCrawl
picsearch
pipl
pmoz
PredictYourBabySearchToolbar
RealDownload
Referrer Karma
ReGet
reverseget
rogerbot
ScoutJet
SearchBot
seexie
seoprofiler
Servage Robot
SeznamBot
shopwiki
sindice
sistrix
SiteSnagger
smart.apnoti.com
SmartDownload
Snoopy
Sosospider
spbot
suggybot
SuperBot
SuperHTTP
SuperPagesUrlVerifyBot
Surfbot
SurveyBot
swebot
Synapse
Tagoobot
tAkeOut
Teleport
Teleport Pro
TeleportPro
TweetmemeBot
TwengaBot
twiceler
UbiCrawler
uptimerobot
URI::Fetch
urllib
User-Agent
VoidEYE
VoilaBot
WBSearchBot
Web Image Collector
Web Sucker
WebAuto
WebCopier
WebFetch
WebGo IS
WebLeacher
WebReaper
WebSauger
Website eXtractor
Website Quester
WebStripper
WebWhacker
WebZIP
Wells Search II
WEP Search
Widow
winHTTP
WWWOFFLE
Xaldon WebSpider
Xenu
yacybot
yandex
YandexBot
YandexImages
yBot
YesupBot
YodaoBot
yolinkBot
youdao
Zao
Zealbot
Zeus
ZyBORG