Wanlin Fire / AI Sources / Observatory
AI and Search Crawler Observatory
Seven-day privacy-safe server-log intelligence. User-Agent strings can be spoofed; verified counts use the configured official provider verification data. Permission or crawling does not guarantee indexing, ranking, training or citation.
Window: 2026-10-08 to 2026-10-13 ยท Generated 2026-10-14T19:15:48+08:00
Seven-day crawler health
| Crawler | Total | Verified | 200 | 301 | 403 | 404 | 410 | 429 |
|---|---|---|---|---|---|---|---|---|
| Claude | 393,139 | 0 | 376,643 | 45 | 2 | 234 | 3,254 | 12,961 |
| GPTBot | 300,114 | 299,898 | 297,841 | 21 | 2 | 173 | 1,928 | 149 |
| Bingbot | 118,706 | 118,475 | 90,139 | 4,351 | 1 | 1,461 | 22,661 | 93 |
| Amazonbot | 56,155 | 0 | 53,559 | 10 | 0 | 2,415 | 12 | 159 |
| Googlebot | 52,139 | 51,034 | 31,035 | 9,681 | 5 | 900 | 10,290 | 228 |
| ClaudeBot | 51,546 | 0 | 50,197 | 25 | 1 | 93 | 549 | 681 |
| Applebot | 28,235 | 0 | 23,927 | 356 | 2 | 138 | 3,650 | 162 |
| Bytespider | 26,404 | 0 | 0 | 1,323 | 21,136 | 0 | 3,945 | 0 |
| CCBot | 22,696 | 0 | 22,327 | 3 | 2 | 81 | 116 | 167 |
| Perplexity | 10,446 | 0 | 9,951 | 14 | 6 | 115 | 12 | 348 |
| PetalBot | 9,434 | 0 | 9,308 | 45 | 0 | 73 | 7 | 1 |
| OAI-SearchBot | 8,031 | 6,487 | 7,630 | 35 | 5 | 162 | 26 | 173 |
| ChatGPT-User | 551 | 349 | 345 | 13 | 1 | 60 | 1 | 131 |
| DeepSeek | 234 | 0 | 1 | 4 | 1 | 55 | 0 | 173 |
| Meta | 231 | 0 | 0 | 0 | 231 | 0 | 0 | 0 |
Machine-readable monitoring
Responsible interpretation
High 410 volume can be correct retirement of invalid historical URLs. A 403 or 429 signal requires checking whether the request was verified before changing security controls. This observatory intentionally publishes no visitor IP address, query string or raw requested URL.