What is robots.txt?
A text file placed in a website's root directory that tells search engine bots which sections of the site they are allowed or disallowed to crawl and index.
Robots.txt is a simple text file stored in the root directory of a website (typically at yoursite.com/robots.txt) that communicates with search engine crawlers about which areas of your site they should access. The file uses straightforward directives like "Disallow" to block entire folders or file types, and "Allow" to permit access to specific paths. For example, a Malaysian e-commerce site might block /admin or /cart pages from crawlers.
The key distinction from noindex is timing and scope. Robots.txt operates at the crawling stage, before a page is even indexed. It prevents bots from requesting the page in the first place. The noindex meta tag, by contrast, allows crawlers to visit and read a page but instructs them not to include it in search results. A page blocked by robots.txt will never be evaluated for indexing, whereas a page with noindex is visited but then excluded from the index.
For technical SEO work, robots.txt matters because it preserves crawl budget (the limit of pages Google will process) and keeps private sections off search results. Misconfigurations can accidentally block important pages. Many agencies in Malaysia use it to hide staging environments, duplicate content, or administrative interfaces while relying on noindex for pages that should remain publicly accessible but hidden from search.