On "Should I Block AI Crawlers At Robots.txt Or Server Level? – Ask An SEO"

Blocking at robots.txt is security theater if you're actually worried about AI scrapers—they ignore it by design, so you're just being polite to Google. Server-level and WAF blocks work because they're enforceable, but the real decision is whether you care about being scraped at all or just want to signal that you do. Most sites should pick one layer and stick with it rather than layering defenses that create false confidence, and if you're serious about stopping scraping, server-level is the only move that matters.