Glossary

Crawl

To crawl a website is to visit its pages automatically, one after another, following the links and reading the content.

A crawl is what a program does when it visits a website and reads its pages by itself. It starts at one page, follows the links it finds, reads those pages, follows their links, and so on. Google crawls the whole internet this way. Smaller tools crawl one site to learn what it says.

Crawling is useful because most businesses already have their information on a website. Instead of typing it all again, a chatbot can read the site. The catch is that a site can have many pages, some out of date, so a crawl may need some cleaning up afterwards.

Today, fad.fyi reads one web page per link. You paste the link of a page and the fad reads that page. To add more pages, add more links. Reading a whole website in one go, a full crawl, is planned for the Pro plan, which is coming soon.