CRAWLER POLICY

Public pages are discoverable; private surfaces are not.

Legitimate search and AI crawlers may index the public marketing and documentation site. Internal monitoring and administrative routes are excluded and independently protected.

Named crawler families and policy tokens

Googlebot
Google-Extended (policy token)
Bingbot
GPTBot
OAI-SearchBot
Amazonbot
Amzn-SearchBot
Facebook external hit
Applebot
DuckDuckBot

Bingbot supports Bing indexing and the search foundation used by Microsoft Copilot experiences. Google-Extended is a control token, not a separate crawler user-agent. Other agents receive no explicit endorsement. Public pages remain ordinary web resources, while non-public paths stay excluded and protected.

Public and private boundaries

Public marketing, documentation, trust, and contact pages are available for normal indexing. /internal/, API, build-asset, console, and private operational paths are not public content.

robots.txt is advisory. Private endpoints remain authenticated and return no public navigation or sitemap entries.

Traffic classification

Server-side operational metrics classify crawler families only from the claimed User-Agent string. KFIF Enterprise does not describe that signal as verified identity.