fn robots_allowed(url: &str) -> bool
Applies conservative public robots rules before reading an explicitly configured page. Redirects or unavailable policies fail closed; a missing robots document permits access.