mastodon/public/robots.txt

User-agent: *
Disallow: /
Allow: /@jyasskin/
Allow: /@jyasskin$
Change robots.txt to exclude only media proxy URLs (#10038) * Revert "Change robots.txt to exclude some URLs (#10037)" This reverts commit 80161f43510ad9316c60c9b50dd5c09c2dae4d54. * Let's block media_proxy /media_proxy/ is a dynamic route used for requesting uncached media, so it's probably bad to let crawlers use it * misleading comment 2019-02-14 03:11:47 +01:00			`User-agent: *`
block all robots in robots.txt (#1) 2017-04-09 01:20:24 +02:00			`Disallow: /`
fix: allow jyasskin's toots and profile to be indexed (#182) 2022-11-08 14:51:03 +01:00			`Allow: /@jyasskin/`
			`Allow: /@jyasskin$`