robots.txt

Robots.txt indicates which portions of a website crawlers are allowed to access in order to prevent these pages from being indexed by Search engines

robots.txt must be at the root of the website

Sitemap: <url>
...

User-agent: <name>
Disallow: <path>
Allow: <path>
...

User-agent: <name>
Disallow: <path>
Allow: <path>
...

...

Alternatively, use <meta name="robots"/> or HTTP Header X-Robots-Tag

Resources

RobotsTXT.org
Google Search Central — robots.txt
Wikipedia — robots.txt
MDN Web Docs — robots.txt

Frameworks

Astro Docs — robots.txt
Next.js Docs — robots.txt
TanStack Start Docs — robots.txt