How can retailers reduce duplicate pages from indexing?

• A. By coding their main website as a 'canonical' page and foreign language pages as 'alternate'.
• B. By blocking all duplicate pages via robots.txt.
• C. By using identical URLs for all language versions.
• D. By removing language variants entirely.

Answer: (B)

Controlling crawling to manage duplicates is a direct way to keep search results clean. By blocking the duplicate URLs with robots.txt, you tell search engine crawlers not to fetch those pages at all. If a page isn’t crawled, it’s far less likely to be indexed, which reduces the number of duplicate pages that could appear in search results. This approach is especially useful when you have many language variants or clone pages that would create multiple URLs serving the same content. In practice, these duplicates can also be managed with canonical tags and hreflang, which guide engines to the preferred version, but blocking duplicates via robots.txt directly limits what gets crawled and indexed. The other options either don’t prevent indexing as effectively, or involve altering how content is presented in ways that can introduce other complications.