Blocking Bots in IIS
One URL Rewrite rule blocks bad bots in IIS by user agent. Refreshed for 2026: the web.config, which entries in the 2014 list you should not block now, and where AI crawlers fit.
TL;DR: Block bad bots in IIS with one URL Rewrite rule keyed on
{HTTP_USER_AGENT}: match the bot strings you actually mean, abort the request before it reaches your application, and put the rule first so nothing rewrites around it. This post is from 2014 and the technique has not changed — URL Rewrite 2.1 on IIS 10 takes the same web.config. What has changed is the list. Half of it I would not block today, and none of it covers the AI crawlers that make up most of the new traffic. Both of those are below.
This started when an Apache application of mine became the victim of bot spam. I wrote up the .htaccess version, then set out to put the same protection on a Windows Server running IIS. Imperva’s annual Bad Bot Report has put automated traffic at roughly half of everything on the web for years now; most of it is inconsequential and can be refused without touching your search rankings, as long as you are careful about which half.
Why block by user agent instead of IP?
Because the bots that matter rotate addresses and the ones that don’t are not worth a rule. A scraper running from a cloud provider has a range of IPs it can burn through; its user agent string is the one thing it tends to keep, because changing it means admitting it is a scraper. Blocking on the header catches the whole fleet with one line. The trade is that anything can lie about its user agent, so this is a filter for the lazy and the honest, not a security control. Treat it as noise reduction: fewer log lines, fewer wasted worker threads, fewer entries in the forms you did not want crawled.
How do you add the rule in IIS?
You need the URL Rewrite module, which does not ship with IIS. The current build is URL Rewrite 2.1, and it installs on everything from IIS 7 through IIS 10 on Windows Server 2016, 2019, 2022, and 2025. I originally wrote this against 2.0 on IIS 7.5; the rule below is unchanged.
The GUI path: open IIS Manager, select the server node, open URL Rewrite, click Add Rule(s)… in the Actions pane, choose Request blocking, set Block access based on to User-agent Header, set Using to Regular Expressions, and paste the pattern.
The GUI writes web.config. I would rather you write it yourself, because then it lives in source control and survives the next person who clicks through the wizard:
<system.webServer>
<rewrite>
<rules>
<rule name="Block bad bots" stopProcessing="true">
<match url=".*" />
<conditions>
<add input="{HTTP_USER_AGENT}"
pattern="^$|HTTrack|WebZIP|EmailWolf" />
</conditions>
<action type="AbortRequest" />
</rule>
</rules>
</rewrite>
</system.webServer>
Three things in there are deliberate. ^$ is the regex for an empty string: I do not serve pages to anything that will not identify itself, and in practice the only things hitting these applications with no user agent were security scanners someone had left running. stopProcessing="true" plus first position in the rule list is what makes match order work in your favour; a redirect rule above it would send the bot to the canonical URL instead of refusing it. And AbortRequest drops the connection without a response body, which costs you nothing. If you want the bot to see a status code, swap it for <action type="CustomResponse" statusCode="403" statusReason="Forbidden" statusDescription="Forbidden" />.
The condition is case-insensitive by default, and it is a substring match unless you anchor it. Be specific. fire matches Firefox. DOC, which is in my original list, matches any user agent with those three letters in it. The more specific string is also the more informative one for the next person who has to touch that setting and wonder why a customer is getting 403s.
One more caution from the original post that still holds: I had a rule for Java/1.7.0_25 because a bot on that exact runtime was hammering the servers. Blocking a language’s default user agent is a blunt instrument. ColdFusion runs on the JVM and makes requests to localhost with the Java user agent to assemble PDFs; JRuby, Groovy, and Scala services can do the same. Pin the version if you must block one at all.
Which of these would I not block in 2026?
The full list from 2014 is at the bottom of this post, kept as the reference it has been for twelve years. Read it before you paste it. Several entries were questionable then and are wrong now:
| Entries | What they are | What blocking them costs |
|---|---|---|
| Slurp, msnbot, Baiduspider, yandex, YandexBot, naver, SeznamBot, Sosospider, youdao, Exabot, gigablast | Search engines: Yahoo, legacy Bing, Baidu, Yandex, Naver, Seznam, Sogou, Youdao | Your pages leave those indexes. Fine for a US-only internal app, not for anything that sells outside the US. |
| AhrefsBot, MJ12bot, DotBot, rogerbot, spbot, sistrix, seoprofiler | SEO crawlers: Ahrefs, Majestic, Moz, OpenLinkProfiler, Sistrix | Your own SEO tools go blind, and so do the competitors you might want to study you. Google does not care either way. |
| ia_archiver, archive, heritrix, commoncrawl | Internet Archive and Common Crawl | No Wayback Machine copy of your site, and no presence in the corpus most language models are trained on. Decide that on purpose. |
| curl, Wget, urllib, libwww-perl, PHP/, Fetch, winHTTP, Java/ | Default user agents of HTTP libraries and tools | Every script that does not set a user agent, including your own health checks, deploy hooks, and the one your customer wrote against your API. |
| uptimerobot, changedetection | Monitoring services | Someone is paying to watch your uptime and you just told them the site is down. |
The rest of the list is site-rippers, download managers, email harvesters, and dead crawlers from 2009, and I would still block every one of them. What I would not do is paste two hundred alternations into one condition on a production server without reading them, which is exactly what I did in 2014.
What about AI crawlers?
None of them existed when this list was written, and they are the bulk of the new automated traffic on any content site. The well-known ones identify themselves: GPTBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot, Bytespider (ByteDance), CCBot (Common Crawl), Applebot-Extended. Google’s Google-Extended is a robots.txt token rather than a user agent; the crawler is still Googlebot, so a rewrite rule cannot separate the two without also blocking search.
I do not block any of them. This site publishes an llms.txt and wants to be cited; a rule that refuses ClaudeBot would be working against the reason the writing exists. If you sell the content itself, the order of operations is robots.txt first, because GPTBot, ClaudeBot, CCBot, and Applebot-Extended honor it, and a rewrite rule second for the ones that have been reported not to. Bytespider is the usual name in that second group. Whatever you decide, decide it: the default of blocking nothing and the default of blocking everything are both positions, and only one of them is the one you meant.
Questions people search for
How do you block a bot by user agent in IIS?
Install the URL Rewrite module, then add a rule to web.config with a condition on {HTTP_USER_AGENT} that matches the bot string and an action of AbortRequest or a 403 CustomResponse. The list above is the pattern; match order decides which rule fires first, so put the block rule ahead of any rewrite that would otherwise let the request through.
Does this still work on IIS 10 and Windows Server 2022?
Yes. URL Rewrite 2.1 is still the module for IIS 10, and the web.config rule syntax has not changed. Rules written for IIS 7.5 or 8.5 carry over as they are.
Should you block bots with a rewrite rule or robots.txt?
Both, for different bots. robots.txt is a request that polite crawlers honor and bad bots ignore. A URL Rewrite rule refuses the request before your application code runs, so the scrapers and download tools on this list never reach it.
The full 2014 list
Below is every string the original rule matched, joined by | into one pattern at the time. Duplicates removed; nothing else edited. Use it as a starting point, not as a rule.
^$
EasouSpider
Add Catalog
PaperLiBot
Spiceworks
ZumBot
RU_Bot
Wget
Java/1.7.0_25
Slurp
FunWebProducts
80legs
Aboundex
AcoiRobot
Acoon Robot
AhrefsBot
aihit
AlkalineBOT
AnzwersCrawl
Arachnoidea
ArchitextSpider
archive
Autonomy Spider
Baiduspider
BecomeBot
benderthewebrobot
BlackWidow
Bork-edition
Bot mailto:craftbot@yahoo.com
botje
catchbot
changedetection
Charlotte
ChinaClaw
commoncrawl
ConveraCrawler
Covario
crawler
curl
Custo
data mining development project
DigExt
DISCo
discobot
discoveryengine
DOC
DoCoMo
DotBot
Download Demon
Download Ninja
eCatch
EirGrabber
EmailSiphon
EmailWolf
eurobot
Exabot
Express WebPictures
ExtractorPro
EyeNetIE
Ezooms
Fetch
Fetch API
filterdb
findfiles
findlinks
FlashGet
flightdeckreports
FollowSite Bot
Gaisbot
genieBot
GetRight
GetWeb!
gigablast
Gigabot
Go-Ahead-Got-It
Go!Zilla
GrabNet
Grafula
GT::WWW
hailoo
heritrix
HMView
houxou
HTTP::Lite
HTTrack
ia_archiver
IBM EVV
id-search
IDBot
Image Stripper
Image Sucker
Indy Library
InterGET
Internet Ninja
internetmemory
ISC Systems iRc Search 2.1
JetCar
JOC Web Spider
k2spider
larbin
LeechFTP
libghttp
libwww
libwww-perl
linko
LinkWalker
lwp-trivial
Mass Downloader
metadatalabs
MFC_Tear_Sample
Microsoft URL Control
MIDown tool
Missigua
Missigua Locator
Mister PiX
MJ12bot
MOREnet
MSIECrawler
msnbot
naver
Navroad
NearSite
Net Vampire
NetAnts
NetSpider
NetZIP
NextGenSearchBot
NPBot
Nutch
Octopus
Offline Explorer
Offline Navigator
omni-explorer
PageGrabber
panscient
panscient.com
Papa Foto
pavuk
pcBrowser
PECL::HTTP
PHP/
PHPCrawl
picsearch
pipl
pmoz
PredictYourBabySearchToolbar
RealDownload
Referrer Karma
ReGet
reverseget
rogerbot
ScoutJet
SearchBot
seexie
seoprofiler
Servage Robot
SeznamBot
shopwiki
sindice
sistrix
SiteSnagger
smart.apnoti.com
SmartDownload
Snoopy
Sosospider
spbot
suggybot
SuperBot
SuperHTTP
SuperPagesUrlVerifyBot
Surfbot
SurveyBot
swebot
Synapse
Tagoobot
tAkeOut
Teleport
Teleport Pro
TeleportPro
TweetmemeBot
TwengaBot
twiceler
UbiCrawler
uptimerobot
URI::Fetch
urllib
User-Agent
VoidEYE
VoilaBot
WBSearchBot
Web Image Collector
Web Sucker
WebAuto
WebCopier
WebFetch
WebGo IS
WebLeacher
WebReaper
WebSauger
Website eXtractor
Website Quester
WebStripper
WebWhacker
WebZIP
Wells Search II
WEP Search
Widow
winHTTP
WWWOFFLE
Xaldon WebSpider
Xenu
yacybot
yandex
YandexBot
YandexImages
yBot
YesupBot
YodaoBot
yolinkBot
youdao
Zao
Zealbot
Zeus
ZyBORG