DadgogoAI Dictionary

Dictionary entry Trending

AI crawler

noun \ˌā-ˈī ˈkrȯ-lər\

also AI web crawler or AI bot or LLM crawler

TopicBuilding with AI Senses1
Advertisement

Definition of AI crawler

  1. : an automated program that retrieves content from websites to support AI model training, search, retrieval, assistants, or related data collection

    • The publisher allowed the company's search crawler but blocked its AI crawler for model training.
    • A surge of AI crawler requests forced the nonprofit website to add rate limits and bot-management rules.

Two meanings, one word

In everyday English

A web crawler is an automated program that visits and retrieves web pages, traditionally for services such as search indexing.

In AI

An automated program that retrieves content from websites to support AI model training, search, retrieval, assistants, or related data collection.

Test yourself

Which of these is the meaning of AI crawler?

Related terms

llms.txt a proposed plain-text file placed at the root of a website that gives AI models a short… training data the collection of examples (text, images, code, audio, and so on) that a model learns from retrieval-augmented generation a technique in which a system first searches a collection of documents for relevant passages… zero-click search a search where the user gets the answer directly on the results page, increasingly from an… deep research an AI feature or agent that spends several minutes searching many sources, reading them, and… rate limit a cap on how many requests or tokens an account can send to an API in a given time

Cite this entry

"AI crawler." AI Dictionary, Dadgogo, https://dadgogo.com/dictionary/ai-crawler/. Accessed 6 Oct. 2026.

Dictionary entries near AI crawler

Advertisement