# AI Mastermind LT community site. # Machine-readable description of what is public here: /llms.txt # # The PUBLIC surface is crawlable: the landing page, the daily digests, the Wiki in both # languages, the machine-access page in both languages, pinned posts, the privacy notice, the # RSS feed - and, since the CEO's decision of 2026-09-07, the chat archive at /bendra/chat/, # the digests at /bendra/digest/ and the people pages at /members/. The reader answers those # three with `X-Robots-Tag: index, follow` and sitemap.xml lists them (measured live # 2026-09-16); no Disallow below covers them, and none is needed. What stays out of the indexes # is the reader's query-parameter space (q/from/to/source/topic/page/size/message/media): every # filtered or searched view answers `X-Robots-Tag: noindex, nofollow, noarchive` from the reader # itself, so crawling it yields nothing indexable, only load. /archive/ is the archive's RETIRED # address (a 301 to /bendra/ since 2026-09-07); it stays disallowed so a crawler holding an old # link is not sent through a redirect to a page it can reach directly at its canonical address. # # THE ALLOW LINES ARE A PUBLIC PROMISE, SO THEY NAME ADDRESSES THAT REALLY EXIST. # Corrected 2026-09-18. They had drifted: every group allowed /llm.html, /docs/ and # /editions.html while the site had moved to /llm/, /lt/wiki/ and /en/wiki/, and named neither # /en/llm/ (the English half of the machine-access page, live that day) nor /members/ nor # /apzvalgos/. Nothing was BLOCKED by that - there is no `Disallow: /` here, so an unlisted path # is crawlable by default and these lines are advisory - but a file we publish that describes a # site we no longer run is untrue, and a reader cannot tell which half of it is stale. # Both address forms of a page are listed where both are real: the SHORT one the site links # (/llm/, /lt/wiki/) and the FILE one its own canonical and sitemap.xml declare # (/llm.html, /lt/wiki/index.html). Excluding either would contradict the page itself. # sitemap_test derives the allowed set from shell.SITEMAP_PAGES and fails on a dead line, so # this list cannot drift again without a test going red. # # This is a crawl boundary, not an access-control mechanism. Approval, redaction and the # chat allowlist are enforced in the projection and on the serving path; nothing that is # unapproved becomes reachable by ignoring this file. # # THE AI CRAWLERS ARE NAMED ON PURPOSE, AND EACH GROUP REPEATS THE WHOLE RULE SET. # A named User-agent group REPLACES the '*' group for that bot - it does not add to it. A # named group carrying only Allow lines would therefore drop the /archive/ Disallow for # exactly the crawlers we went out of our way to welcome. Every group # below is complete. Being readable by a model is the point: a non-member who asks an # assistant about Lithuanian AI discussion should be able to be pointed here, and the # assistant should be able to cite the digests rather than invent them. How to cite this # archive without inventing an author is written down in /llms.txt. # Ordinary search crawlers (Bingbot, Googlebot, DuckDuckBot and the rest) are covered by the # '*' group and are deliberately NOT named: naming a bot only to repeat the same rules adds a # place to forget the Disallow. Verified 2026-09-18 that Bingbot and Googlebot both fetch # /, /llm/ and /en/llm/ with HTTP 200 under this file. User-agent: * Crawl-delay: 10 Allow: /$ Allow: /index.html Allow: /editions.html Allow: /pins.html Allow: /llm/ Allow: /llm.html Allow: /en/llm/ Allow: /lt/wiki/ Allow: /en/wiki/ Allow: /privacy/ Allow: /members/ Allow: /bendra/chat/ Allow: /bendra/digest/ Allow: /apzvalgos/ Allow: /diagrams/ Allow: /llms.txt Allow: /rss.xml Allow: /sitemap.xml Allow: /logo.svg Disallow: /archive/ User-agent: GPTBot Crawl-delay: 10 Allow: /$ Allow: /index.html Allow: /editions.html Allow: /pins.html Allow: /llm/ Allow: /llm.html Allow: /en/llm/ Allow: /lt/wiki/ Allow: /en/wiki/ Allow: /privacy/ Allow: /members/ Allow: /bendra/chat/ Allow: /bendra/digest/ Allow: /apzvalgos/ Allow: /diagrams/ Allow: /llms.txt Allow: /rss.xml Allow: /sitemap.xml Allow: /logo.svg Disallow: /archive/ User-agent: OAI-SearchBot Crawl-delay: 10 Allow: /$ Allow: /index.html Allow: /editions.html Allow: /pins.html Allow: /llm/ Allow: /llm.html Allow: /en/llm/ Allow: /lt/wiki/ Allow: /en/wiki/ Allow: /privacy/ Allow: /members/ Allow: /bendra/chat/ Allow: /bendra/digest/ Allow: /apzvalgos/ Allow: /diagrams/ Allow: /llms.txt Allow: /rss.xml Allow: /sitemap.xml Allow: /logo.svg Disallow: /archive/ User-agent: ClaudeBot Crawl-delay: 10 Allow: /$ Allow: /index.html Allow: /editions.html Allow: /pins.html Allow: /llm/ Allow: /llm.html Allow: /en/llm/ Allow: /lt/wiki/ Allow: /en/wiki/ Allow: /privacy/ Allow: /members/ Allow: /bendra/chat/ Allow: /bendra/digest/ Allow: /apzvalgos/ Allow: /diagrams/ Allow: /llms.txt Allow: /rss.xml Allow: /sitemap.xml Allow: /logo.svg Disallow: /archive/ User-agent: PerplexityBot Crawl-delay: 10 Allow: /$ Allow: /index.html Allow: /editions.html Allow: /pins.html Allow: /llm/ Allow: /llm.html Allow: /en/llm/ Allow: /lt/wiki/ Allow: /en/wiki/ Allow: /privacy/ Allow: /members/ Allow: /bendra/chat/ Allow: /bendra/digest/ Allow: /apzvalgos/ Allow: /diagrams/ Allow: /llms.txt Allow: /rss.xml Allow: /sitemap.xml Allow: /logo.svg Disallow: /archive/ User-agent: Google-Extended Crawl-delay: 10 Allow: /$ Allow: /index.html Allow: /editions.html Allow: /pins.html Allow: /llm/ Allow: /llm.html Allow: /en/llm/ Allow: /lt/wiki/ Allow: /en/wiki/ Allow: /privacy/ Allow: /members/ Allow: /bendra/chat/ Allow: /bendra/digest/ Allow: /apzvalgos/ Allow: /diagrams/ Allow: /llms.txt Allow: /rss.xml Allow: /sitemap.xml Allow: /logo.svg Disallow: /archive/ Sitemap: https://grupe.aimastermind.lt/sitemap.xml