DadgogoAI Dictionary

Dictionary entry Trending

AI crawler

noun \ˌā-ˈī ˈkrȯ-lər\

also AI web crawler or AI bot or LLM crawler

TopicBuilding with AI Senses1

Definition of AI crawler

  1. : an automated program that retrieves content from websites to support AI model training, search, retrieval, assistants, or related data collection

    • The publisher allowed the company's search crawler but blocked its AI crawler for model training.
    • A surge of AI crawler requests forced the nonprofit website to add rate limits and bot-management rules.

Two meanings, one word

In everyday English

A web crawler is an automated program that visits and retrieves web pages, traditionally for services such as search indexing.

In AI

An automated program that retrieves content from websites to support AI model training, search, retrieval, assistants, or related data collection.

Advertisement

Test yourself

Which of these is the meaning of AI crawler?

Related terms

llms.txt a proposed plain-text file placed at the root of a website that gives AI models a short… training data the collection of examples, observations, labels, actions, or feedback that a model learns… retrieval-augmented generation a technique in which a system first searches a collection of documents for relevant passages… zero-click search a search where the user gets the answer directly on the results page, increasingly from an… deep research an AI feature or agent that spends several minutes searching many sources, reading them, and… rate limit a cap on how many requests or tokens an account can send to an API in a given time Web Bot Auth an emerging web authentication protocol and IETF working-group effort that lets automated HTTP… agentic traffic website, application, or API requests generated by AI agents that browse, compare, fill forms…

Cite this entry

"AI crawler." AI Dictionary, Dadgogo, https://dadgogo.com/dictionary/ai-crawler/. Accessed 26 Oct. 2026.

Dictionary entries near AI crawler

Advertisement