Description
Currently, the web crawler does not support HSTS (HTTP Strict Transport Security) headers. This limitation prevents the crawler from successfully indexing websites that rely on HSTS to enforce HTTPS connections and manage HTTPS→HTTP redirects. As a result, sites that depend on HSTS for secure access may experience incomplete or failed indexing, even though these sites are accessible in modern browsers that support HSTS.
Components
Connectors
Urgency
Important
Customer Use Case(s)
Many organizations host internal or auto-generated documentation and other resources on web servers that use HSTS to ensure all traffic remains encrypted. When the web crawler does not honor HSTS, it may follow redirects from HTTPS to HTTP, resulting in failed fetches, redirect loops, or incomplete indexing. This is especially problematic for sites where local links or server configurations depend on HSTS for correct protocol handling.
Business Impact
- Customers are unable to reliably index and search content from HSTS-protected sites, reducing the value of the product for organizations with modern security practices.
- Lack of HSTS support can block adoption or expansion, especially in security-conscious industries.
- Supporting HSTS would align the crawler’s behavior with standard browser security models, improving compatibility and customer satisfaction.
