On "What can you actually put in a robots.txt?"

Most practitioners still treat robots.txt like a magic wand for SEO problems, which is wrong—it's a crawl management tool, not a ranking lever or security firewall. The clarification matters because people waste time blocking entire directories from crawlers when they should be optimizing internal link structure, and others think excluding something from crawlers hides it from search results (it doesn't). If you're using robots.txt to try to fix indexation issues or hide pages you don't want ranked, you're solving the problem at the wrong layer and need to step back.