Adding Apple Documentation As A Datasource?

Hello Glean Community,

I'm reaching out for some guidance on adding the Apple MacBook documentation website as a custom data source in Glean. We have successfully added the seed URL for the site, but it seems that only a single item has been indexed so far. It's been about 12 hours.

Has anyone experienced similar issues or could provide insights on what might be going wrong? Any tips or recommended configurations specific to the Apple documentation website that could help improve the indexing process would be greatly appreciated.

Thank you in advance for your help!

Find more posts tagged with

Glean for Engineering

Accepted answers

All comments

brettroeder

What seed URL are you using?

gphoenix_aledade

https://support.apple.com/en-ie/guide/macbook-air/welcome/mac

For time's sake, I used Playwright to create PDFs of all the pages in the TOC programmatically and then added them to a google drive folder.

Josh Liebster

You might also be running in to their robots.txt rules: https://support.apple.com/robots.txt

# robots for Inquira throttling

User-agent: IQ-WWW

Request-rate: 5/1  #maximum rate is 5 pages per second


…


User-agent: *

Disallow: /kb/index?*page=search*

Disallow: *src=support_app*

Disallow: /*/docs/product/*

Disallow: /docs/product/*

Disallow: */MANUALS/*.pdf$


…

Getting Started

Events

Help Center

glean.com