AI Agents & Web Crawlers Directory
Comprehensive database of AI agents, web crawlers, and bots that visit websites. Learn about each one and how to manage their access to your content.
Track AI Traffic on Your Website
Get real-time visibility into which AI agents and crawlers are visiting your website. Set up blocking rules and analytics integration in minutes.
Start Free TrialAI Model Trainers
5 botsBots collecting web data for AI model training
AI Model Trainers
Crawlers that index content for use in building AI models
AI Visitors
5 botsAI systems acting directly on behalf of user requests
AI Assistants
Direct AI assistants executing web requests
Crawler
30 botsBots indexing content for search engines and online tools
AI Search Engine Crawlers
Bots that crawl to improve AI search capabilities
Archive Crawlers
Web archival services
Ecommerce Crawlers
Crawlers from online marketplaces or Ecommerce sites
Search Engine Crawlers
Major search engine crawlers
Showing first 12 of 20 bots
SEO Tools
Tools for SEO analysis and monitoring
Scraper
7 botsData collection tools, code and command-line tools, and other items that can be used to scrape web content. These are different than crawlers in that they are typically initiated by an application or a person and do not re-visit sites and pages on a schedule.
Code Libraries
Code modules that can access web pages. Note that it is possible for the application developer to override the user agent, which would cause these tools to not. report correctly.
Command-Line Tools
Visits generated by tools that can be run from a command line, like cURL, wget, and others. These tools are not necessarily malicious, but they are often used for scraping or other data collection, which is why they are listed here.
Data Collection Tools
Known data collection tools
Special Purpose
9 botsBots and tools that support specific applications. Generally these are safe to allow.
Monitoring Tools
Tools for monitoring website availability and performance
Security Tools
Tools for security scanning and monitoring
Social Media Tools
Tools used by social media sites for specific purposes, like generating link previews, fetching page titles, and the like.
Unknown
563 botsUnclassified or ambiguous bots
Unclassified
Bots pending classification
Ready to Take Control of Your AI Traffic?
Join thousands of websites using Spyglasses to detect, analyze, and manage AI agent traffic. Start your free trial today.
Get Started Free