Apache / IIS 8.5 User-Agent Block

iis Oct 30, 2017

There are a lot of articles about this and most of them reflects the same words and actions to do on IIS, however, this will be used as a KB to usual configurations on our day-to-day work.

Before we continue, please ensure that URL Rewrite Module is installed on your IIS and it's visible on Website configuration. If yes, let's procceed, if not..you will need to install and restart web service.

So, on Apache there are several ways to do this block, using a Rewrite Condition or define a Env to Deny, for example:

<IfModule mod_rewrite.c>
  RewriteEngine On
  RewriteCond %{HTTP_USER_AGENT} ^(AgentToBlock) [NC]
  RewriteRule .* - [F,L]
</IfModule>
<IfModule mod_setenvif.c>
  SetEnvIfNoCase User-Agent "^AgentToBlock" block_bot
  Order Allow,Deny
  Allow from all
  Deny from env=block_bot
</IfModule>

In both scenarios, we are blocking only one User-Agent, but if you pretend to block more than one..

<IfModule mod_rewrite.c>
  RewriteEngine On
  RewriteCond %{HTTP_USER_AGENT} ^(AgentToBlock|AnotherToBlock) [NC]
  RewriteRule .* - [F,L]
</IfModule>
<IfModule mod_setenvif.c>
  SetEnvIfNoCase User-Agent (AgentToBlock|AnotherToBlock) block_bots
  
  Order Allow,Deny
  Allow from all
  Deny from env=block_bots
</IfModule>

This will work and you don't have to worrie about that User-Agent. Let's see on IIS 7 / 8.5.

Let's open IIS Manager > Domain > URL Rewrite

Next, we will add the rule as "Request blocking"

iis_8_rewrite_request_block

So, after selecting this step, we will follow the same reasoning as the Apache for block Agents.

What we want to block? User-Agent Header

Based on what? On a Pattern (word / name) on the Header

What we will use to define that Pattern? Regular Expression

What we want to do? Send to 403 error (Forbidden)

That's right, so, you will need something like this:

iis_8_rewrite_add_agent-1

Based on this, you will send a User-Agent with the name "AgentToBlock" with “^$” on the beggining to ensure that empty User-Agent are not allowed too.

Following the same on Apache, what if we need to block several User-Agent? Ok, on "Pattern" we will only need to add as more as we need at the same way that we did on Apache.

Pattern (User-agent Header):

^$|360Spider|80legs|Abonti|Aboundex|Acunetix|ADmantX|AhrefsBot|AIBOT|AiHitBot|Aipbot|Alexibot|Alligator|AllSubmitter|Anarchie|Apexoo|ASPSeek|Asterias|Attach|autoemailspider|BackDoorBot|Backlink-Ceck|backlink-check|BacklinkCrawler|BackStreet|BackWeb|Badass|Bandit|Barkrowler|BatchFTP|Battleztar\ Bazinga|BBBike|BDFetch|BetaBot|Bigfoot|Bitacle|Blackboard|Black\ Hole|BlackWidow|BLEXBot|Blow|BlowFish|Boardreader|Bolt|BotALot|Brandprotect|BrandProtect|Brandwatch|Bubing|Buddy|BuiltBotTough|BuiltWith|Bullseye|BunnySlippers|BuzzSumo|Calculon|CATExplorador|CazoodleBot|CCBot|Cegbfeieh|CheeseBot|CherryPicker|ChinaClaw|Chlooe|Claritybot|Cliqzbot|Cogentbot|cognitiveseo|Collector|com.plumanalytics|Copier|CopyRightCheck|Copyscape|Cosmos|Craftbot|CrazyWebCrawler|CRAZYWEBCRAWLER|Crescent|CSHttp|Curious|Custo|DatabaseDriverMysqli|DataCha0s|DBLBot|demandbase-bot|Demon|Deusu|Devil|Digincore|DIIbot|Dirbuster|Disco|Discobot|Discoverybot|DittoSpyder|DomainAppender|DomainCrawler|DomainSigmaCrawler|DomainStatsBot|Dotbot|Download\ Demon|Download\ Devil|Download\ Wonder|Dragonfly|Drip|DTS\ Agent|EasyDL|Ebingbong|eCatch|ECCP/1.0|Ecxi|EirGrabber|EMail\ Collector|EMail\ Extractor|EMail\ Siphon|EMail\ Wolf|EroCrawler|Evil|Exabot|Express\ WebPictures|Extractor|ExtractorPro|Extreme\ Picture\ Finder|EyeNetIE|Ezooms|FDM|FHscan|Fimap|Firefox/7.0|FlashGet|Flunky|Foobot|fq|Freeuploader|FrontPage|Fyrebot|GalaxyBot|Genieo|GermCrawler|Getintent|GetRight|GetWeb|Gigablast|Gigabot|G-i-g-a-b-o-t|Go-Ahead-Got-It|Gotit|GoZilla|Go!Zilla|Grabber|GrabNet|Grafula|GrapeFX|GrapeshotCrawler|Grid ot|GT::WWW|HaosouSpider|Harvest|Havij|HEADMasterSEO|Heritrix|Hloader|HMView|HTMLparser|HTTP::Lite|HTTrack|Humanlinks|HybridBot|Iblog|IDBot|Id-search|IlseBot|Image\ Fetch|Image\ Stripper|Image\ Sucker|Indy\ Library|InfoNaviRobot|InfoTekies|instabid|Intelliseek|InterGET|Internet\ Ninja|InternetSeer|internetVista\ monitor|Iria|IRLbot|Iskanie|JamesBOT|Jbrofuzz|JennyBot|JetCar|JikeSpider|JOC\ Web\ Spider|Joomla|Jorgee|JustView|Jyxobot|Kenjin\ Spider|Keyword\ Density|Kozmosbot|Lanshanbot|Larbin|LeechFTP|LeechGet|LexiBot|Lftp|LibWeb|Libwhisker|Lightspeedsystems|Likse|Linkdexbot|LinkextractorPro|LinkpadBot|LinkScan|LinksManager|LinkWalker|LinqiaMetadataDownloaderBot|LinqiaRSSBot|LinqiaScrapeBot|Lipperhey|Litemage_walker|Lmspider|LNSpiderguy|Ltx71|lwp-request|LWP::Simple|lwp-trivial|Magnet|Mag-Net|magpie-crawler|Mail.ru|Majestic12|MarkMonitor|MarkWatch|Masscan|Mass\ Downloader|Mata\ Hari|Meanpathbot|mediawords|MegaIndex.ru|Metauri|MFC_Tear_Sample|Microsoft\ Data\ Access|Microsoft\ URL\ Control|MIDown\ tool|MIIxpc|Mister\ PiX|MJ12bot|Mojeek|Morfeus\ Fucking\ Scanner|MSFrontPage|MSIE\ 6.0|MSIECrawler|Msrabot|MS\ Web\ Services\ Client\ Protocol|Musobot|Name\ Intelligence|Nameprotect|Navroad|NearSite|Needle|Nessus|NetAnts|Netcraft|netEstate\ NE\ Crawler|NetLyzer|NetMechanic|NetSpider|Nettrack|Net\ Vampire|Netvibes|NetZIP|NextGenSearchBot|Nibbler|NICErsPRO|Niki-bot|Nikto|NimbleCrawler|Ninja|Nmap|NPbot|Nutch|Octopus|Offline\ Explorer|Offline\ Navigator|Openfind|OpenLinkProfiler|Openvas|OrangeBot|OrangeSpider|OutfoxBot|PageAnalyzer|Page\ Analyzer|PageGrabber|Page\ Grabber|page\ scorer|PageScorer|Panscient|Papa\ Foto|Pavuk|pcBrowser|PECL::HTTP|PeoplePal|PHPCrawl|Picscout|Picsearch|PictureFinder|Pimonster|Pi-Monster|Pixray|PleaseCrawl|plumanalytics|Pockey|POE-Component-Client-HTTP|Probethenet|ProPowerBot|ProWebWalker|Psbot|Pump|PyCurl|QueryN\ Metasearch|Qwantify|RankActive|RankActiveLinkBot|RankFlex|RankingBot|RankingBot2|Rankivabot|RankurBot|RealDownload|Reaper|RebelMouse|Recorder|RedesScrapy|ReGet|RepoMonkey|Ripper|RocketCrawler|Rogerbot|SalesIntelligent|SBIder|ScanAlert|Scanbot|Scrapy|Screaming|Screaming\ Frog\ SEO\ Spider|ScreenerBot|Searchestate|SearchmetricsBot|Semrush|SemrushBot|SEOkicks|SEOkicks-Robot|SEOlyticsCrawler|Seomoz|SEOprofiler|seoscannersSEOstats|Siphon|SISTRIX|SISTRIX\ Crawler|Sitebeam|SiteExplorer|Siteimprove|SiteLockSpider|SiteSnagger|SiteSucker|Site\ Sucker|Sitevigil|Slackbot-LinkExpanding|SlySearch|SmartDownload|Snake|Snapbot|Snoopy|SocialRankIOBot|Sogou\ web\ spider|Sosospider|Sottopop|SpaceBison|Spammen|SpankBot|Spanner|Spbot|Spinn3r|SputnikBot|Sqlmap|Sqlworm|Sqworm|Steeler|Stripper|Sucker|Sucuri|SuperBot|SuperHTTP|Surfbot|SurveyBot|Suzuran|Swiftbot|sysscan|Szukacz|T0PHackTeam|T8Abot|tAkeOut|Teleport|TeleportPro|Telesoft|Telesphoreo|Telesphorep|The\ Intraformant|TheNomad|TightTwatBot|Titan|Toata|Toweyabot|Trendictionbot|True_Robot|Turingos|TurnitinBot|Turnitin\ Bot|Turnitin\ Robot|TwengaBot|Twice|Typhoeus|UnisterBot|URLy.Warning|URLy\ Warning|Vacuum|Vagabondo|VB\ Project|VCI|VeriCiteCrawler|VidibleScraper|Virusdie|VoidEYE|Voil|Voltron|Wallpapers/3.0|WallpapersHD|WASALive-Bot|WBSearchBot|Webalta|WebAuto|Web\ Auto|WebBandit|Web\ Bandit|WebCollage|Web\ Collage|WebCopier|Web\ Copier|WEBDAV|WEBDAV\ Client|WebEnhancer|Web\ Enhancer|WebFetch|Web\ Fetch|WebFuck|Web\ Fuck|WebGo\ IS|WebImageCollector|Web\ Image\ Collector|WebLeacher|WebmasterWorldForumBot|webmeup-crawler|WebPix|Web\ Pix|WebReaper|Web\ Reaper|WebSauger|Web\ Sauger|Webshag|WebsiteExtractor|Website\ Extractor|WebsiteQuester|Website\ Quester|Webster|WebStripper|Web\ Stripper|WebSucker|Web\ Sucker|WebWhacker|Web\ Whacker|WebZIP|WeSEE|Whack|Whacker|Whatweb|Widow|WinHTTrack|WiseGuys\ Robot|WISENutbot|Wonderbot|Woobot|Wotbox|Wprecon|WPScan|WWW-Collector-E|WWW-Mechanize|WWW::Mechanize|WWWOFFLE|x09Mozilla|x22Mozilla|Xaldon_WebSpider|Xaldon\ WebSpider|Xenu|YoudaoBot|Zade|Zermelo|Zeus|Zgrab|Zitebot|ZmEu|ZumBot|ZyBorg

Hope that was clear to you and i'm pretty sure that will save some bandwith too :)

Enjoy it!