{"id":115,"date":"2025-11-23T04:07:39","date_gmt":"2025-11-23T04:07:39","guid":{"rendered":"http:\/\/ijeesoo.com\/?page_id=115"},"modified":"2025-11-24T00:52:58","modified_gmt":"2025-11-24T00:52:58","slug":"web-crawling","status":"publish","type":"page","link":"http:\/\/ijeesoo.com\/?page_id=115","title":{"rendered":"Web Crawling"},"content":{"rendered":"\n
\n
\n

Definition and Role<\/h1>\n\n\n\n
<\/div>\n\n\n\n

Definition:<\/strong>Web crawling (or “spidering”) is the process of systematically browsing the World Wide Web, typically for the purpose of web indexing.<\/p>\n\n\n\n

The Crawler:<\/strong>A software bot (also called a spider or web robot) that visits web pages to gather information, starting from a list of “seed” URLs.<\/p>\n\n\n\n

Role in Search Engines:<\/strong><\/p>\n\n\n\n

Discovery:<\/strong>Finds new and updated pages to add to the search engine’s index.<\/p>\n\n\n\n

Indexing:<\/strong>Extracts content (text, links, images) from pages to build the searchable index.<\/p>\n\n\n\n

Freshness:<\/strong>Revisits pages to check for changes, keeping the index up-to-date.<\/p>\n<\/div>\n\n\n\n

\"Building<\/figure>\n<\/div>\n\n\n\n
\n
\n

Core Crawling Strategies<\/h2>\n\n\n\n
<\/div>\n\n\n\n

<\/p>\n<\/div>\n\n\n\n

\n
\n

Deapth-First Crawling<\/h3>\n\n\n\n

<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

<\/p>\n\n\n\n

<\/p>\n<\/div>\n\n\n\n

\n

Breath-First Crawling<\/h3>\n\n\n\n

<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

<\/p>\n\n\n\n

<\/p>\n\n\n\n

<\/p>\n<\/div>\n\n\n\n

\n

Focused Crawling<\/h3>\n\n\n\n

<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

<\/p>\n\n\n\n

<\/p>\n<\/div>\n<\/div>\n\n\n\n

<\/div>\n\n\n\n
\n
\n

Crawling Strategies: Comparison<\/h2>\n\n\n\n
\"\"<\/figure>\n<\/div>\n\n\n\n
<\/div>\n<\/div>\n<\/div>\n\n\n\n
\n
\n
\n

Features a Crawler Must Provide<\/h2>\n<\/div>\n\n\n\n
<\/div>\n\n\n\n
\n
\n

Feature 1: Politeness & Compliance<\/strong><\/h3>\n\n\n\n

Goal:<\/strong>To minimize server load, respect site owner wishes, and maintain a good relationship with the web community.<\/p>\n\n\n\n

    \n
  • robots.txt:<\/strong>The crawler must first check for and obey the Robots Exclusion Protocol. A Disallow: rule tells the crawler which parts of a site *not* to visit.<\/li>\n\n\n\n
  • Crawl-Delay:<\/strong>Adhering to Crawl-Delay directives in robots.txt or using adaptive throttling to avoid overloading a web server. This is critical to avoid being banned.<\/li>\n\n\n\n
  • User-Agent Handling:<\/strong>Clearly identifying the crawler via its User-Agent string (e.g., Googlebot\/2.1) allows site administrators (webmasters) to set specific rules for specific bots.<\/li>\n<\/ul>\n<\/div>\n\n\n\n
    \n

    Feature 2: Robustness and Error Handling<\/h3>\n\n\n\n
      \n
    • Fault Tolerance:<\/strong>The system must be designed to withstand component failures (network outages, machine crashes) without losing track of which URLs have been processed or are pending.<\/li>\n\n\n\n
    • Retry Logic:<\/strong>Implementation of smart, time-delayed retries (often using exponential backoff<\/strong>) for transient HTTP errors (like 503 Service Unavailable) to recover data when the server returns.<\/li>\n\n\n\n
    • Handling Malformed HTML:<\/strong>The parser must be robust enough to handle “tag soup”\u2014broken tags, missing elements, and non-standard markup\u2014to successfully extract content and outgoing links from messy real-world web pages.
      <\/li>\n<\/ul>\n<\/div>\n<\/div>\n\n\n\n
      <\/div>\n\n\n\n
      \n
      \n

      Feature 3: Scalability for the Global Web<\/strong><\/h3>\n\n\n\n
        \n
      • Large-Scale Operation:<\/strong>A modern crawler must be able to fetch billions of pages from millions of different web hosts.<\/li>\n\n\n\n
      • Distributed Systems:<\/strong>This is not feasible on a single machine. Scalability is achieved by distributing the crawl across many machines (nodes).<\/li>\n\n\n\n
      • Key Components:<\/strong>\n
          \n
        • URL Frontier:<\/strong>Manages the list of all URLs to be crawled.<\/li>\n\n\n\n
        • Fetchers:<\/strong>Parallel processes that download pages.<\/li>\n\n\n\n
        • Parsers:<\/strong>Parallel processes that extract new links.
          <\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/div>\n\n\n\n
          \n
          \"Windows<\/figure>\n<\/div>\n<\/div>\n\n\n\n
          \n
          \n

          Feature 4: URL Normalization and Duplication<\/strong><\/h3>\n\n\n\n

          Goal:<\/strong>Avoid redundant fetches of the same content. URLs are normalized to a canonical (standard) form *before* being checked for duplication.<\/p>\n\n\n\n

            \n
          • URL Normalization:<\/strong>Converts syntactically different URLs that point to the same<\/em>resource into a single, canonical form<\/strong>.\n
              \n
            • Examples:<\/em>Converting hostnames to lowercase, removing default ports (:80), and sorting query parameters.<\/li>\n<\/ul>\n<\/li>\n\n\n\n
            • URL Deduplication:<\/strong>Using fast, space-efficient data structures (like Bloom Filters) to check the normalized URL against the set of already crawled pages, preventing redundant fetches and endless loops.<\/li>\n\n\n\n
            • Content Deduplication:<\/strong>Comparing the content of two different URLs to detect and eliminate duplicate copies, ensuring the index is efficient and free of boilerplate content.<\/li>\n<\/ul>\n<\/div>\n\n\n\n
              \n\n\n\n\n\n\n\n
              Normalization Type<\/b><\/td>\nOriginal URL<\/b><\/td>\nNormalized URL<\/b><\/td>\n<\/thead>\n
              Lowercase Host<\/td>http:\/\/ExAmPlE.com\/<\/td>\nhttp:\/\/example.com\/<\/td><\/tr>\n
              Remove Default Port<\/td>\nhttp:\/\/example.com:80\/<\/td>\nhttp:\/\/example.com\/<\/td><\/tr>\n
              Resolve Path Traversal<\/td>\nhttp:\/\/example.com\/a\/b\/..\/c\/<\/td>\nhttp:\/\/example.com\/a\/c\/<\/td><\/tr>\n
              Sort Query Parameters<\/td>\nhttp:\/\/example.com\/?b=2&a=1<\/td>http:\/\/example.com\/?a=1&b=2<\/td><\/tr>\n<\/tbody>\n<\/table>\n\n\n\n
                \n
              • <\/li>\n<\/ul>\n<\/div>\n<\/div>\n\n\n\n
                \n
                \n

                Feature 5: Content parsing and link extraction<\/strong><\/h3>\n\n\n\n

                1-Content Parsing (Goal: Prepare for Indexing) <\/strong><\/p>\n\n\n\n

                The goal here is to isolate the meaningful<\/em> content of the page from the markup and surrounding noise so it can be added to the search engine’s index.<\/p>\n\n\n\n

                  \n

                • <\/p>Extraction<\/strong>:Separating the textual content (the actual articles, descriptions, or information) from the HTML tags, JavaScript code, and CSS styles.<\/li>\n\n\n\n

                • <\/p>Normalization<\/strong>:Cleaning up the extracted text, resolving character encodings, and handling specific document types (like converting a PDF into searchable text).<\/li>\n\n\n\n

                • Metadata Harvesting<\/strong>:Collecting important information about the page, such as the title (<title>), meta description, and keywords, which are crucial for ranking and snippet generation.<\/span><\/p><\/li>\n<\/ul>\n<\/div>\n\n\n\n

                  \n
                  \n
                  \n

                  Content parsing and link extraction<\/h2>\n\n\n\n

                  2-Link Extraction (Goal: Feed the Crawler) <\/strong><\/p>\n\n\n\n

                  The goal here is to identify and process every outgoing hyperlink on the current page to ensure the crawler can continue traversing the web.<\/p>\n\n\n\n

                    \n
                  • Discovery:<\/strong>Finding all anchor tags (<a href=”…”>) and other elements that contain links (like image sources, if needed) within the parsed document.<\/li>\n\n\n\n
                  • Resolution:<\/strong>Converting relative URLs<\/strong>(e.g., \/products\/new.html) into absolute, fully qualified URLs<\/strong> (e.g., http:\/\/example.com\/products\/new.html) so they can be accurately added to the URL frontier.<\/li>\n\n\n\n
                  • Filtering:<\/strong>Checking the extracted links against rules (like those found in robots.txt or nofollowattributes) to ensure the crawler only queues URLs it’s allowed to visit.<\/li>\n<\/ul>\n\n\n\n

                    In essence, parsing and extraction serve as the bottleneck between downloading the raw bytes and making decisions about what to index<\/strong> and where to go next<\/strong>.<\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n

                    <\/p>\n<\/div>\n\n\n\n

                    \n
                    \n
                    \n

                    Features a Crawler Should Provide<\/h2>\n<\/div>\n\n\n\n
                    <\/div>\n\n\n\n
                    \n
                    \n

                    Focused Crawling<\/h3>\n\n\n\n

                    Goal:<\/strong>To selectively crawl and index pages that are relevant to a specific, pre-defined topic (e.g., “health,” “finance”), saving resources.<\/p>\n\n\n\n

                      \n
                    • Mechanism:<\/strong>It replaces the simple FIFO (Breadth-First) queue with a Priority Queue<\/strong>.<\/li>\n\n\n\n
                    • Prioritization:<\/strong>A machine learning classifier<\/strong>scores unvisited URLs based on their predicted relevance to the topic (using features like anchor text, keywords in the link, etc.).<\/li>\n\n\n\n
                    • Process:<\/strong>High-scoring links are prioritized in the queue, guiding the crawler to stay on-topic and avoid irrelevant parts of the web.<\/li>\n\n\n\n

                    • Use Case:<\/strong>Vertical search engines (e.g., a medical search engine), academic research, building specialized knowledge bases.<\/span><\/p><\/li>\n<\/ul>\n<\/div>\n\n\n\n

                      \n

                      Freshness & Recrawl Policies<\/h3>\n\n\n\n
                        \n

                      • <\/p>The Challenge<\/strong>:The web is dynamic. Content is constantly added, updated, and deleted. A “stale” index is a low-quality index.<\/li>\n\n\n\n

                      • <\/p>The Goal<\/strong>:To maintain a “fresh” index by revisiting pages to check for changes.<\/li>\n\n\n\n

                      • <\/p>Problem<\/strong>:It is impossible and impolite to recrawlthe entire web constantly.<\/li>\n\n\n\n

                      • Solution (RecrawlPolicy):<\/strong>\n
                          \n
                        • Uniform Policy:<\/strong>Revisit all pages at the same rate (inefficient).<\/li>\n\n\n\n
                        • Proportional Policy (Smarter):<\/strong>Revisit pages based on their estimated frequency of change<\/strong>. News sites are recrawledfrequently (minutes\/hours), while static pages are recrawledrarely (weeks\/months).<\/li>\n\n\n\n
                        • Prioritization:<\/strong>High-importance pages (e.g., high PageRank) are also prioritized for recrawlingto ensure the most important content is the freshest.

                          <\/p><\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/div>\n<\/div>\n\n\n\n

                          <\/div>\n\n\n\n

                          Spam & Trap Avoidance<\/h3>\n\n\n\n
                          \n
                            \n
                          • The Adversarial Web:<\/strong>Not all web content is benign; some is designed to deceive or trap crawlers.<\/li>\n\n\n\n
                          • Crawler Traps (Spider Traps):<\/strong>\n
                              \n
                            • Infinite Loops:<\/strong>Dynamically generated pages that create an endless “tree” of links (e.g., a calendar with a “next day” link that goes on forever).<\/li>\n\n\n\n
                            • Honeypots:<\/strong>Pages built specifically to trap bots with a massive number of (often invisible) links.<\/li>\n<\/ul>\n<\/li>\n\n\n\n
                            • Heuristics for Avoidance:<\/strong>\n
                                \n
                              • Setting a maximum crawl depth or URL length.<\/li>\n\n\n\n
                              • Detecting path repetition (e.g., \/shop\/shop\/shop\/…).<\/li>\n\n\n\n
                              • Using URL normalization to collapse similar, “spammy” URLs.<\/li>\n<\/ul>\n<\/li>\n\n\n\n
                              • Cloaking:<\/strong>Detecting when a server provides different content to the crawler (based on User<\/li>\n<\/ul>\n<\/div>\n\n\n\n
                                \n
                                \n

                                Metadata Extraction<\/h3>\n\n\n\n

                                Goal<\/strong>:To extract data abouta page, which is often as important as the content onthe page for indexing and ranking.
                                Key Metadata:<\/strong><\/p>\n\n\n\n

                                  \n

                                • <\/p>Anchor Text:<\/strong>The clickable text of an incoming link (e.g., <a href=”page_B.html”>This is anchor text<\/a>). This text, found on Page A, is stored as a powerful, objective descriptor for Page B.<\/li>\n\n\n\n

                                • <\/p>Page Titles<\/strong>:The content of the <title>tag. A primary signal for the page’s main topic.<\/li>\n\n\n\n

                                • <\/p>HTTP Headers<\/strong>:Caching information (Last-Modified, ETag), content type (Content-Type), and redirect status (301, 302) are all stored to manage the crawl.<\/li>\n<\/ul>\n<\/div>\n\n\n\n

                                  \n

                                  Support for Dynamic Content<\/h3>\n\n\n\n
                                    \n
                                  • The Challenge (The Modern Web)<\/strong>:Many pages load as a minimal HTML “shell” and then use JavaScript (JS) to fetch and display all the real content (via AJAX, etc.).<\/li>\n\n\n\n
                                  • Problem<\/strong>:A simple “fetch-and-parse” crawler only sees the initial, often-empty, HTML file. It misses all the content loaded by JavaScript.<\/li>\n\n\n\n

                                  • Solution (JavaScript Rendering):<\/strong><\/p>\n

                                      \n
                                    • The crawler must act like a full web browser.<\/li>\n\n\n\n
                                    • It loads the page into a headless browser<\/strong>(e.g., Chromium).<\/li>\n\n\n\n
                                    • It executes the JavaScript, waits for network calls to complete, and then parses the final, rendered HTML DOM<\/strong>(Document Object Model).<\/li>\n<\/ul>\n<\/li>\n\n\n\n
                                    • Limitation:<\/strong>This is extremely<\/em> resource-intensive (CPU, memory, time) and is a major bottleneck for modern crawlers.
                                      <\/li>\n<\/ul>\n<\/div>\n\n\n\n
                                      \n

                                      The Modern Crawler: Key Challenges<\/h3>\n\n\n\n
                                        \n
                                      • The modern web crawler has evolved from a simple “fetcher” into a complex, intelligent, and resource-intensive system.<\/li>\n\n\n\n
                                      • It must balance multiple competing goals:\n
                                          \n
                                        • Efficiency vs. Completeness:<\/strong> Using simple HTML parsing for most pages but knowing when to deploy expensive JavaScript (JS) rendering.<\/li>\n\n\n\n
                                        • Speed vs. Politeness:<\/strong> Crawling as fast as possible to maintain freshness, but respecting robots.txtand Crawl-Delayto remain a “good” bot.<\/li>\n\n\n\n
                                        • Trust vs. Verification:<\/strong>Ingesting content while actively filtering out spam, traps, and deceptive cloaking techniques.<\/li>\n<\/ul>\n<\/li>\n\n\n\n
                                        • These advanced features are what separate a simple script from a petabyte-scale search engine.
                                          <\/li>\n<\/ul>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n
                                          \n
                                          \n
                                          \n

                                          Crawler Architecture<\/h2>\n\n\n\n
                                          <\/div>\n<\/div>\n\n\n\n
                                          \n
                                          \n
                                          \n

                                          Modular Crawler Architecture<\/h3>\n\n\n\n

                                          Goal:<\/strong>To build a large-scale system that is scalable, fault-tolerant, and maintainable. This is achieved through a “separation of concerns.”<\/p>\n\n\n\n

                                            \n
                                          • \u2022The Four Core Modules:<\/strong><\/li>\n\n\n\n
                                          • \u2022Scheduler (URL Frontier):<\/strong>The “brain.” Decides which<\/em>page to crawl next and when<\/em>.<\/li>\n\n\n\n
                                          • \u2022Fetcher:<\/strong>The “legs and hands.” Responsible for downloading the raw content (HTML, PDF, etc.) from a given URL.<\/li>\n\n\n\n
                                          • \u2022Parser:<\/strong>The “eyes and mind.” Responsible for reading the downloaded content, extracting the usable text, and discovering all new (outgoing) links.<\/li>\n\n\n\n
                                          • \u2022Storage:<\/strong>The “memory.” A set of databases that store the crawled content (document index), link relationships (link database), and metadata.<\/li>\n\n\n\n
                                          • \u2022The Flow:<\/strong>The Scheduler gives a URL to the Fetcher. The Fetcher’s content goes to the Parser. The Parser’s extracted links go back to the Scheduler, and the extracted text goes to Storage.<\/li>\n<\/ul>\n<\/div>\n\n\n\n
                                            \n

                                            The URL Frontier (Scheduler)<\/h3>\n\n\n\n

                                            Definition:<\/strong>The central data structure that manages all URLs the crawler knows about<\/em>but hasn’t crawled yet<\/em>. It can contain billions of URLs.<\/p>\n\n\n\n

                                              \n
                                            • Key Responsibilities:<\/strong>\n
                                                \n
                                              • Prioritization:<\/strong>It’s not a simple queue. It’s a Priority Queue<\/strong>that uses scoring functions<\/strong>to decide what to crawl next.<\/li>\n\n\n\n
                                              • Politeness:<\/strong>It must manage per-host queues to ensure it doesn’t hit the same web server too rapidly (obeying Crawl-Delay).<\/li>\n<\/ul>\n<\/li>\n\n\n\n
                                              • Scoring Inputs (How Priority is Set):<\/strong>\n
                                                  \n
                                                • Importance:<\/strong>Is the page a high PageRank site? (Crawl it sooner).<\/li>\n\n\n\n
                                                • Freshness:<\/strong>Is this a news site that changes often? (Recrawlit often).<\/li>\n\n\n\n
                                                • Topic:<\/strong>Is it relevant to a focused crawl? (Score it higher).<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n
                                                  <\/div>\n\n\n\n
                                                  \n
                                                  \n

                                                  Content Filters (The Gatekeeper)<\/h3>\n\n\n\n
                                                    \n
                                                  • Goal:<\/strong>To save processing (CPU) and storage resources by discarding unwanted content as early as possible.<\/li>\n\n\n\n
                                                  • Filter 1: MIME Type Filtering (Pre-Parse):<\/strong>\n
                                                      \n
                                                    • What:<\/strong>Checks the Content-Typeheader (e.g., text\/html, application\/pdf, image\/jpeg).<\/li>\n\n\n\n
                                                    • Action:<\/strong>If text\/html, send to the HTML parser. If application\/pdf, send to the PDF-to-text converter. If video\/mp4or .zip, discard.<\/li>\n<\/ul>\n<\/li>\n\n\n\n
                                                    • \u2022Filter 2: Language Detection (Post-Parse):<\/strong>\n
                                                        \n
                                                      • What:<\/strong>Analyzes the extracted text to identify its natural language.<\/li>\n\n\n\n
                                                      • Action:<\/strong>If the search engine only serves English-speaking users, pages identified as 100% Russian or Mandarin may be discarded.<\/li>\n<\/ul>\n<\/li>\n\n\n\n
                                                      • Other Filters:<\/strong>Spam detection, duplicate content detection, etc.<\/li>\n<\/ul>\n<\/div>\n\n\n\n
                                                        \n

                                                        Threading & Concurrency (The Engine)<\/h3>\n\n\n\n
                                                          \n
                                                        • The Problem:<\/strong>Crawling is I\/O-bound<\/strong>. The single slowest part is waiting for a remote server to respond over the network. The CPU sits idle 99% of the time.<\/li>\n\n\n\n
                                                        • The Solution: Concurrency.<\/strong>Do thousands of things at once.<\/li>\n\n\n\n
                                                        • Approach 1: Multi-threaded Fetchers:<\/strong>\n
                                                            \n
                                                          • Run hundreds or thousands of “fetcher” threads in parallel.<\/li>\n\n\n\n
                                                          • While Thread 1 is waiting<\/em>for Server A, Thread 2 is requesting<\/em>from Server B, and Thread 3 is processing<\/em>data from Server C.<\/li>\n<\/ul>\n<\/li>\n\n\n\n
                                                          • Approach 2: Asynchronous (Non-Blocking) I\/O:<\/strong>\n
                                                              \n
                                                            • The modern, more efficient model.<\/li>\n\n\n\n
                                                            • A single process can manage thousands of open connections. It issues a request, then immediately moves to another task, handling responses as they arrive (event-driven). This maximizes CPU and network efficiency.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n
                                                              \n
                                                              \n

                                                              Architecture in Motion (The Full Flow)<\/h3>\n\n\n\n

                                                              This slide shows how all the modules work together:<\/p>\n\n\n\n

                                                                \n
                                                              1. The URL Frontier<\/strong> selects a high-priority, polite URL.<\/li>\n\n\n\n
                                                              2. A free Fetcher Thread<\/strong>(using AsyncI\/O) makes the HTTP request and downloads the raw content.<\/li>\n\n\n\n
                                                              3. The MIME Type Filter<\/strong> checks the content. If it’s text\/html…<\/li>\n\n\n\n
                                                              4. A Parser<\/strong>is assigned to read the HTML, extract the clean text, and find all <a> (link) tags.<\/li>\n\n\n\n
                                                              5. The Language Detection Filter<\/strong>checks the clean text. If it’s a valid language…<\/li>\n\n\n\n
                                                              6. The Clean Text & Metadata<\/strong>are sent to Storage<\/strong>(the Index).<\/li>\n\n\n\n
                                                              7. The Extracted Links<\/strong>are sent back to the URL Frontier<\/strong>to be scored, prioritized, and added to the queue for future crawling.<\/li>\n<\/ol>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n
                                                                \n
                                                                \n
                                                                \n

                                                                DNS Resolution<\/h2>\n\n\n\n
                                                                <\/div>\n<\/div>\n\n\n\n
                                                                \n
                                                                \n

                                                                DNS Resolution: The Crawler’s “Address Book”<\/h3>\n\n\n\n
                                                                  \n
                                                                • Role in Crawling:<\/strong>The crawler’s main job is to fetch content from URLs. A URL contains a hostname<\/strong>(e.g., www.example.com), but the internet works on IP addresses<\/strong>(e.g., 93.184.216.34).<\/li>\n\n\n\n
                                                                • The “First Step”:<\/strong>Before a crawler can open a network connection to a web server, it must<\/em>use the Domain Name System (DNS) to resolve the hostname into its corresponding IP address.<\/li>\n\n\n\n
                                                                • Process:<\/strong>\n
                                                                    \n
                                                                  • Crawler takes URL: http:\/\/www.example.com\/page.html<\/li>\n\n\n\n
                                                                  • Extracts hostname: www.example.com<\/li>\n\n\n\n
                                                                  • Performs DNS Lookup: www.example.com-> 93.184.216.34<\/li>\n\n\n\n
                                                                  • Opens connection to 93.184.216.34and requests \/page.html.<\/li>\n<\/ul>\n<\/li>\n\n\n\n
                                                                  • Conclusion:<\/strong>DNS resolution is a fundamental, non-negotiable prerequisite for fetching content from any new host.<\/li>\n<\/ul>\n<\/div>\n\n\n\n
                                                                    \n

                                                                    Caching Strategies to Reduce Latency<\/h3>\n\n\n\n
                                                                      \n
                                                                    • The Bottleneck:<\/strong>Performing a full DNS lookup for every single URL<\/em>would be disastrously slow. At the scale of billions of pages, this latency would kill crawl performance.<\/li>\n\n\n\n
                                                                    • The Solution: Caching.<\/strong>A high-performance crawler maintains its own large, internal DNS cache (often in-memory).<\/li>\n\n\n\n
                                                                    • How it Works:<\/strong>\n
                                                                        \n
                                                                      • The crawler looks up example.comand gets 93.184.216.34. It stores this mapping in its cache.<\/li>\n\n\n\n
                                                                      • For the next 10,000 URLs<\/em>from example.com, the crawler looks in its local cache first, getting the IP address almost instantly (in nanoseconds).<\/li>\n\n\n\n
                                                                      • This bypasses the need for any external network lookups.<\/li>\n<\/ul>\n<\/li>\n\n\n\n
                                                                      • Time-To-Live (TTL):<\/strong>The cache must still respect the TTL (e.g., “this IP is valid for 1 hour”) set by the domain. This ensures the crawler adapts if a site’s IP address changes.<\/li>\n<\/ul>\n<\/div>\n<\/div>\n\n\n\n
                                                                        <\/div>\n\n\n\n
                                                                        \n
                                                                        \n

                                                                        Handling DNS Failures and Timeouts<\/h3>\n\n\n\n
                                                                          \n
                                                                        • The Internet is Unreliable:<\/strong>DNS servers can fail, packets get lost, and domains expire. A robust crawler must handle this gracefully.<\/li>\n\n\n\n
                                                                        • Common Failures:<\/strong>\n
                                                                            \n
                                                                          • Timeout:<\/strong>The DNS server did not respond in time. This is a transient<\/em>error.<\/li>\n\n\n\n
                                                                          • NXDOMAIN<\/strong>(Non-Existent Domain):<\/strong>A permanent<\/em> error. The domain name is invalid or has expired.<\/li>\n\n\n\n
                                                                          • SERVFAIL<\/strong>(Server Failure):<\/strong>The DNS server is misconfigured or down. This is transient<\/em>.<\/li>\n<\/ul>\n<\/li>\n\n\n\n
                                                                          • Crawler’s Retry Logic:<\/strong>\n
                                                                              \n
                                                                            • NXDOMAIN<\/strong>:<\/strong> Mark the domain as “dead” and remove all its URLs from the frontier (or de-prioritize for 1 year).<\/li>\n\n\n\n
                                                                            • Timeouts \/ <\/strong>SERVFAIL<\/strong>:<\/strong> Do not<\/em> delete the URLs. This is a temporary problem. The crawler should log the error, de-prioritize the host, and schedule a retry for several hours or days later.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/div>\n\n\n\n
                                                                              \n

                                                                              Impact on Crawl Throughput<\/h3>\n\n\n\n
                                                                                \n
                                                                              • Throughput = Pages per Second.<\/strong>This is the #1 metric for a crawler’s efficiency.<\/li>\n\n\n\n
                                                                              • Direct Impact:<\/strong>DNS is often the main<\/em> bottleneck before the fetcher threads can even start their work.<\/li>\n\n\n\n
                                                                              • High Throughput (Good):<\/strong>A large, fast, in-memory DNS cache makes hostname-to-IP resolution instantaneous. This allows the fetcher threads to run at full capacity, limited only by network speed and politeness delays.<\/li>\n\n\n\n
                                                                              • Low Throughput (Bad):<\/strong>A small or non-existent cache forces fetchers to wait<\/em>for DNS lookups. The entire crawl grinds to a halt.<\/li>\n\n\n\n
                                                                              • Example:<\/strong>At scale, a crawler’s internal DNS resolver may handle millions of queries per second, while a standard DNS server can only handle thousands.<\/li>\n<\/ul>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n
                                                                                \n
                                                                                \n

                                                                                Impact on Crawl Reliability<\/h3>\n\n\n\n
                                                                                  \n
                                                                                • Reliability = Completeness and Uptime.<\/strong>Does the crawler find all the content and never stop running?<\/li>\n\n\n\n
                                                                                • DNS Failures:<\/strong>How the crawler handles DNS failures directly impacts its reliability.<\/li>\n\n\n\n
                                                                                • Unreliable Crawler (Bad):<\/strong>\n
                                                                                    \n
                                                                                  • Treats a temporary DNS timeout as a permanent failure.<\/li>\n\n\n\n
                                                                                  • Deletes a valid website from its frontier just because the DNS server was down for 10 minutes.<\/li>\n\n\n\n
                                                                                  • The resulting index is incomplete<\/em>and missing large chunks of the web.<\/li>\n<\/ul>\n<\/li>\n\n\n\n
                                                                                  • Reliable Crawler (Good):<\/strong>\n
                                                                                      \n
                                                                                    • Differentiates between permanent (NXDOMAIN) and temporary (timeout) errors.<\/li>\n\n\n\n
                                                                                    • Uses smart retry logic to revisit temporarily failed hosts.<\/li>\n\n\n\n
                                                                                    • The resulting index is more complete<\/em>and accurately reflects the state of the web, even the parts that are temporarily hard to reach.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n
                                                                                      \n
                                                                                      \n
                                                                                      \n

                                                                                      The URL Frontier<\/h2>\n\n\n\n
                                                                                      <\/div>\n<\/div>\n\n\n\n
                                                                                      \n
                                                                                      \n

                                                                                      What is The URL Frontier?<\/h3>\n\n\n\n

                                                                                      Definition:<\/strong>The URL Frontier is the central data structure that manages all the URLs a crawler intends to crawl or recrawl. It’s the crawler’s main “to-do list.”<\/p>\n\n\n\n

                                                                                        \n
                                                                                      • Core Task:<\/strong>It acts as the scheduler, deciding which<\/em>URL a free fetcher thread should crawl next<\/em>.<\/li>\n\n\n\n
                                                                                      • Key Challenges:<\/strong>\n
                                                                                          \n
                                                                                        • Scale:<\/strong>It must manage billions<\/em>of URLs.<\/li>\n\n\n\n
                                                                                        • Efficiency:<\/strong>It must select and dispatch URLs with high throughput.<\/li>\n\n\n\n
                                                                                        • Prioritization:<\/strong>It must decide which pages are more important to crawl first.<\/li>\n\n\n\n
                                                                                        • Politeness:<\/strong>It must ensure the crawler doesn’t hit the same server too quickly.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/div>\n\n\n\n
                                                                                          \n

                                                                                          Frontier Data Structures<\/h3>\n\n\n\n

                                                                                          The “Queue” is not one, but many.<\/strong>A large-scale frontier is a complex set of queues.<\/p>\n\n\n\n

                                                                                            \n
                                                                                          • 1. Per-Host Queues (For Politeness):<\/strong>\n
                                                                                              \n
                                                                                            • The frontier is first divided into many separate FIFO (First-In, First-Out)<\/strong>queues, one for each hostname.<\/li>\n\n\n\n
                                                                                            • When the crawler picks a host, it pulls the next URL from that host’s FIFO queue. This naturally enforces politeness delays (e.g., “wait 1 second before pulling from this queue again”).<\/li>\n<\/ul>\n<\/li>\n\n\n\n
                                                                                            • 2. Priority Queue (For Scheduling):<\/strong>\n
                                                                                                \n
                                                                                              • A main Priority Queue<\/strong>is used to decide which<\/em>per-host queue to visit next.<\/li>\n\n\n\n
                                                                                              • This queue is not based on time, but on a score<\/strong>(e.g., PageRank, freshness). It ensures the crawler’s time is spent on high-value hosts.<\/li>\n<\/ul>\n<\/li>\n\n\n\n
                                                                                              • 3. Bloom Filters (For Memory):<\/strong>\n
                                                                                                  \n
                                                                                                • A probabilistic data structure used to quickly answer: “Have I ever<\/em>seen this URL before?”<\/li>\n\n\n\n
                                                                                                • It prevents duplicate URLs from being added to the frontier, saving memory and preventing re-work, with a very small chance of error.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/div>\n<\/div>\n\n\n\n
                                                                                                  <\/div>\n\n\n\n
                                                                                                  \n
                                                                                                  \n

                                                                                                  URL Scoring & Prioritization<\/h3>\n\n\n\n

                                                                                                  Goal:<\/strong>To crawl a “better” version of the web faster by assigning a priority score to each URL. This score determines its position in the priority queue.<\/p>\n\n\n\n

                                                                                                  Key Scoring Inputs:<\/strong><\/p>\n\n\n\n

                                                                                                    \n
                                                                                                  • Link Popularity (e.g., PageRank):<\/strong>High-PageRank URLs are considered more important and are given a higher priority. This is based on the idea of crawling the “best” pages first.<\/li>\n\n\n\n
                                                                                                  • Freshness:<\/strong>Pages that are known to change frequently (like news homepages or stock tickers) are given a higher priority for recrawling<\/em>.<\/li>\n\n\n\n
                                                                                                  • Relevance (for Focused Crawls):<\/strong>In a topical crawler, a machine learning classifier scores URLs based on their predicted relevance to the topic. On-topic links get a high-priority score.<\/li>\n\n\n\n
                                                                                                  • History:<\/strong>Pages that have historically returned errors or spam are given a very low priority.<\/li>\n<\/ul>\n<\/div>\n\n\n\n
                                                                                                    \n

                                                                                                    Duplicate Detection & Canonicalization<\/h3>\n\n\n\n
                                                                                                      \n
                                                                                                    1. This is Step 0.<\/strong>Before a URL is ever<\/em>added to the frontier, it must be “cleaned” and “checked.”<\/li>\n\n\n\n
                                                                                                    2. Canonicalization (Normalization):<\/strong>\n
                                                                                                        \n
                                                                                                      1. T<\/strong>he process of converting a URL into its single, standard (canonical) form.<\/li>\n\n\n\n
                                                                                                      2. Examples:<\/strong>\n
                                                                                                          \n
                                                                                                        1. HTTP:\/\/Example.com\/-> http:\/\/example.com\/(lowercase host)<\/li>\n\n\n\n
                                                                                                        2. http:\/\/example.com:80\/-> http:\/\/example.com\/(remove default port)<\/li>\n\n\n\n
                                                                                                        3. http:\/\/example.com\/a\/..\/b\/-> http:\/\/example.com\/b\/(resolve path)<\/li>\n<\/ol>\n<\/li>\n<\/ol>\n<\/li>\n\n\n\n
                                                                                                        4. Duplicate Detection:<\/strong>\n
                                                                                                            \n
                                                                                                          • After canonicalization, the crawler checks if this standardized URL already exists<\/li>\n\n\n\n
                                                                                                          • (in the frontier or in the “already crawled” database).<\/li>\n\n\n\n
                                                                                                          • This is where Bloom Filters<\/strong>or Hash Sets<\/strong> are used for a very fast,\u00a0<\/li>\n\n\n\n
                                                                                                          • memory-efficient check to prevent redundant work.<\/li>\n<\/ul>\n<\/li>\n<\/ol>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n
                                                                                                            \n
                                                                                                            \n

                                                                                                            Frontier Expansion Strategies<\/h3>\n\n\n\n
                                                                                                              \n
                                                                                                            • The frontier’s logic <\/strong>defines<\/em><\/strong>the crawl strategy.<\/strong>How you store, prioritize, and retrieve URLs dictates the “shape” of your crawl.<\/li>\n\n\n\n
                                                                                                            • Breadth-First (BFS):<\/strong>\n
                                                                                                                \n
                                                                                                              • How:<\/strong>Using a simple, single FIFO Queue<\/strong>(First-In, First-Out).<\/li>\n\n\n\n
                                                                                                              • Result:<\/strong>The crawler explores the web level by level. It finds all pages 1 click from the seed, then all pages 2 clicks away, etc. This is the most common and “safest” strategy.<\/li>\n<\/ul>\n<\/li>\n\n\n\n
                                                                                                              • Depth-First (DFS):<\/strong>\n
                                                                                                                  \n
                                                                                                                • How:<\/strong>Using a LIFO Stack<\/strong>(Last-In, First-Out).<\/li>\n\n\n\n
                                                                                                                • Result:<\/strong>The crawler follows a single link path as deep as it can go. This is rarely used as it gets “lost” in deep sites and crawler traps.<\/li>\n<\/ul>\n<\/li>\n\n\n\n
                                                                                                                • Focused:<\/strong>\n
                                                                                                                    \n
                                                                                                                  • How:<\/strong>Using a Priority Queue<\/strong>where the score is topic relevance<\/em>.<\/li>\n\n\n\n
                                                                                                                  • Result:<\/strong>The crawler prioritizes links that seem<\/em>relevant, ignoring paths that appear to be off-topic.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n
                                                                                                                    \n
                                                                                                                    \n
                                                                                                                    \n

                                                                                                                    Distributing Indexes<\/h2>\n\n\n\n
                                                                                                                    <\/div>\n<\/div>\n\n\n\n
                                                                                                                    \n
                                                                                                                    \n

                                                                                                                    Why Distribute an Index?<\/h3>\n\n\n\n

                                                                                                                    The Problem:<\/strong>A single machine cannot handle the scale of a modern search index.<\/p>\n\n\n\n

                                                                                                                      \n
                                                                                                                    1. Storage Limits (The “Size” Problem):<\/strong>\n
                                                                                                                        \n
                                                                                                                      • The inverted index for billions (or trillions) of documents is petabytes<\/em> in size.<\/li>\n\n\n\n
                                                                                                                      • This vastly exceeds the disk space, RAM, and CPU of any single server.<\/li>\n<\/ul>\n<\/li>\n\n\n\n
                                                                                                                      • Query Throughput (The “Speed” Problem):<\/strong>\n
                                                                                                                          \n
                                                                                                                        • A large search engine must handle tens of thousands of queries per second (QPS).<\/li>\n\n\n\n
                                                                                                                        • A single machine cannot serve this many requests, especially when each query requires complex-list lookups and scoring.<\/li>\n<\/ul>\n<\/li>\n\n\n\n
                                                                                                                        • The Solution:<\/strong>\n
                                                                                                                            \n
                                                                                                                          • Horizontal Scaling:<\/strong>Instead of one “supercomputer,” use a cluster of thousands of cheaper “commodity” machines working in parallel.<\/li>\n\n\n\n
                                                                                                                          • This requires partitioning<\/strong>(splitting) the index and query work across the cluster.<\/li>\n<\/ul>\n<\/li>\n<\/ol>\n<\/div>\n\n\n\n
                                                                                                                            \n

                                                                                                                            Partitioning Strategies<\/h3>\n\n\n\n

                                                                                                                            Goal: To split the giant inverted index into smaller, manageable pieces.<\/p>\n\n\n\n

                                                                                                                              \n
                                                                                                                            • Strategy 1: Document-Based Partitioning (or “Sharding”)<\/strong>\n
                                                                                                                                \n
                                                                                                                              • How<\/strong>: The collection of documents is split into N partitions (shards). Each shard holds a complete, independent index for its subset of documents.<\/li>\n\n\n\n
                                                                                                                              • Query<\/strong>: The query is sent to all N shards in parallel. A broker node then gathers and merges the results from all shards.<\/li>\n\n\n\n
                                                                                                                              • Pro<\/strong>: Simple, easy to add new documents. This is the model used by Elasticsearch\/Solr.<\/li>\n<\/ul>\n<\/li>\n\n\n\n
                                                                                                                              • Strategy 2: Term-Based Partitioning<\/strong>\n
                                                                                                                                  \n
                                                                                                                                • How<\/strong>: The dictionary of terms is split. Shard A holds terms a-f, Shard B holds g-m, etc. Each shard holds the full posting lists for its terms.<\/li>\n\n\n\n
                                                                                                                                • Query<\/strong>: A query (e.g., “web search”) is split. The “web” term goes to Shard W, “search” goes to Shard S.<\/li>\n\n\n\n
                                                                                                                                • Pro<\/strong>: Very efficient for querying, as you only contact the shards that hold your terms. Con: More complex to manage and update.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/div>\n<\/div>\n\n\n\n
                                                                                                                                  <\/div>\n\n\n\n
                                                                                                                                  \n
                                                                                                                                  \n

                                                                                                                                  Shardingand Replication<\/h3>\n\n\n\n

                                                                                                                                  These are two distinct but related concepts for horizontal scaling.<\/p>\n\n\n\n

                                                                                                                                    \n
                                                                                                                                  • Sharding(Partitioning):<\/strong>\n
                                                                                                                                      \n
                                                                                                                                    • What:<\/strong>The process of splitting your data (the index) across multiple machines. This is what solves the storage (size) problem<\/em>.<\/li>\n\n\n\n
                                                                                                                                    • Benefit:<\/strong>Allows the index to grow to any size by simply adding more machines (shards). It also parallelizes query work.<\/li>\n<\/ul>\n<\/li>\n\n\n\n
                                                                                                                                    • \u2022Replication:<\/strong>\n
                                                                                                                                        \n
                                                                                                                                      • What:<\/strong>The process of making copies<\/em>(replicas) of your shards on different<\/em> machines.<\/li>\n\n\n\n
                                                                                                                                      • Benefit 1: Fault Tolerance:<\/strong>If a machine holding a primary shard fails, its replica on another machine is instantly promoted, and no data or query capability is lost.<\/li>\n\n\n\n
                                                                                                                                      • Benefit 2: High Throughput:<\/strong>Query load can be balanced across all replicas, allowing the cluster to handle a much higher volume of queries per second (QPS).<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/div>\n\n\n\n
                                                                                                                                        \n

                                                                                                                                        MapReducefor Index Construction<\/h3>\n\n\n\n

                                                                                                                                        Problem<\/strong>: How do you build a petabyte-scale inverted index in parallel?<\/p>\n\n\n\n

                                                                                                                                        MapReduce<\/strong>is a programming model designed for this exact task.<\/p>\n\n\n\n

                                                                                                                                          \n
                                                                                                                                        1. MAP Phase (Parallel Tokenization):<\/strong>\n
                                                                                                                                            \n
                                                                                                                                          • Input: (DocID, Document Content)<\/li>\n\n\n\n
                                                                                                                                          • Many “Map” workers run in parallel, each on a small chunk of documents.<\/li>\n\n\n\n
                                                                                                                                          • Each worker tokenizes its documents and emits (key, value) pairs of: (term, DocID)<\/li>\n\n\n\n
                                                                                                                                          • Example: (\u2018search\u2019, D1), (\u2018engine\u2019, D1), (\u2018search\u2019, D2)<\/li>\n<\/ul>\n<\/li>\n\n\n\n
                                                                                                                                          • SHUFFLE Phase:<\/strong>The system automatically sorts and groups all pairs by key (the term).<\/li>\n\n\n\n
                                                                                                                                          • REDUCE Phase (Posting List Generation):<\/strong>\n
                                                                                                                                              \n
                                                                                                                                            • Input: (term, [list_of_DocIDs])<\/li>\n\n\n\n
                                                                                                                                            • Each “Reduce” worker takes one term (e.g., \u2018search\u2019) and its entire list of document IDs ([D1, D2, D50, …]).<\/li>\n\n\n\n
                                                                                                                                            • It then sorts, compresses, and writes the final posting list for that term to disk.<\/li>\n<\/ul>\n<\/li>\n<\/ol>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n
                                                                                                                                              \n
                                                                                                                                              \n

                                                                                                                                              Index Consistency & Synchronization<\/h3>\n\n\n\n

                                                                                                                                              The Challenge:<\/strong>The web is not static. How do you add new documents to a massive, read-only distributed index without<\/em>rebuilding it every time?<\/p>\n\n\n\n

                                                                                                                                                \n
                                                                                                                                              • The “Main + Auxiliary” Model:<\/strong>\n
                                                                                                                                                  \n
                                                                                                                                                • Main Index:<\/strong>The massive, static index built by MapReduce. It is read-only and very fast for queries.<\/li>\n\n\n\n
                                                                                                                                                • Auxiliary Index (or “Delta”):<\/strong>A second, smaller index (often in-memory) that is dynamic and writable. All new<\/em>documents are added here first.<\/li>\n<\/ul>\n<\/li>\n\n\n\n
                                                                                                                                                • Synchronization:<\/strong>\n
                                                                                                                                                    \n
                                                                                                                                                  1. A query must now search both<\/em>the Main Index and the Auxiliary Index, then merge the results.<\/li>\n\n\n\n
                                                                                                                                                  2. Periodically (e.g., every few minutes), the small in-memory Auxiliary Index is “flushed” to a new, permanent, read-only file on disk (called a “segment”).<\/li>\n\n\n\n
                                                                                                                                                  3. In the background, a “compaction” process merges these smaller segments into the Main Index.<\/li>\n<\/ol>\n<\/li>\n<\/ul>\n<\/div>\n\n\n\n
                                                                                                                                                    \n

                                                                                                                                                    The Trade-offs<\/h3>\n\n\n\n
                                                                                                                                                      \n
                                                                                                                                                    • Distributing an index is not free; it’s a constant balance of competing goals.<\/li>\n\n\n\n
                                                                                                                                                    • Latency vs. Update Frequency:<\/strong>\n
                                                                                                                                                        \n
                                                                                                                                                      • A static, read-only index has the lowest query latency<\/em>(fastest response).<\/li>\n\n\n\n
                                                                                                                                                      • Allowing for high-frequency updates (e.g., “near-real-time”) requires the “Main + Auxiliary” model, which adds a small amount of latency to every query<\/em>(the cost of merging).<\/li>\n<\/ul>\n<\/li>\n\n\n\n
                                                                                                                                                      • Throughput vs. Consistency:<\/strong>\n
                                                                                                                                                          \n
                                                                                                                                                        • High query throughput is achieved with many replicas<\/em>.<\/li>\n\n\n\n
                                                                                                                                                        • However, the more replicas you have, the harder and slower it is to synchronize updates across all of them, leading to weaker consistency<\/em>(a query to Replica A might show different results than Replica B for a few seconds).<\/li>\n<\/ul>\n<\/li>\n\n\n\n
                                                                                                                                                        • Cost vs. Performance:<\/strong>More shards and more replicas mean better performance, fault tolerance, and storage, but at a linear (or greater) increase in hardware and operational cost.<\/li>\n<\/ul>\n<\/div>\n<\/div>\n\n\n\n
                                                                                                                                                          \n
                                                                                                                                                          \n

                                                                                                                                                          Real-World Systems<\/h3>\n\n\n\n
                                                                                                                                                            \n
                                                                                                                                                          • Google (Caffeine, GFS, Bigtable):<\/strong>\n
                                                                                                                                                              \n
                                                                                                                                                            • The “web-scale” model. Google continuously crawls and updates its index in small, “percolating” batches rather than one large one.<\/li>\n\n\n\n
                                                                                                                                                            • The index is global and massively distributed, using a hybrid of term and document partitioning.<\/li>\n\n\n\n
                                                                                                                                                            • A single query may touch thousands of machines to get an answer in milliseconds.<\/li>\n<\/ul>\n<\/li>\n\n\n\n
                                                                                                                                                            • Elasticsearch& Apache Solr:<\/strong>\n
                                                                                                                                                                \n
                                                                                                                                                              • The “enterprise-scale” model, both built on the Apache Lucene<\/strong>library.<\/li>\n\n\n\n
                                                                                                                                                              • They primarily use document-based partitioning<\/strong>(“sharding”).<\/li>\n\n\n\n
                                                                                                                                                              • They manage shardingand replication automatically.<\/li>\n\n\n\n
                                                                                                                                                              • They excel at “Near-Real-Time” (NRT) search<\/strong>, using the “Main + Auxiliary” (segment-based) model to make new documents searchable within seconds, not hours.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/div>\n<\/div>\n\n\n\n

                                                                                                                                                                <\/p>\n\n\n\n

                                                                                                                                                                \n
                                                                                                                                                                \n

                                                                                                                                                                Connectivity Servers<\/h2>\n\n\n\n
                                                                                                                                                                <\/div>\n<\/div>\n\n\n\n
                                                                                                                                                                \n
                                                                                                                                                                \n

                                                                                                                                                                Definition & Role of a Connectivity Server<\/h3>\n\n\n\n

                                                                                                                                                                Definition<\/strong>: A Connectivity Server is a specialized, large-scale database designed to store and serve the Web Graph<\/strong>.<\/p>\n\n\n\n

                                                                                                                                                                  \n
                                                                                                                                                                • What is the Web Graph?<\/strong>\n
                                                                                                                                                                    \n
                                                                                                                                                                  • Nodes<\/strong>: Web pages (Documents).<\/li>\n\n\n\n
                                                                                                                                                                  • Directed Edges<\/strong>: Hyperlinks (<a href=”…”>) from one page to another.<\/li>\n<\/ul>\n<\/li>\n\n\n\n
                                                                                                                                                                  • Primary Role:<\/strong>\n
                                                                                                                                                                      \n
                                                                                                                                                                    1. Store: To maintain a massive, up-to-date copy of this link structure (trillions of edges).<\/li>\n\n\n\n
                                                                                                                                                                    2. Serve: To provide other parts of the search engine (like the ranker and crawler) with fast answers to “who links to whom?”<\/li>\n<\/ol>\n<\/li>\n\n\n\n
                                                                                                                                                                    3. It is the search engine’s central “database of links.”.<\/li>\n<\/ul>\n<\/div>\n\n\n\n
                                                                                                                                                                      \n

                                                                                                                                                                      The “Why”: Supporting Link Analysis<\/h3>\n\n\n\n
                                                                                                                                                                        \n
                                                                                                                                                                      • The Core Idea:<\/strong>The content of a page is not the only signal of its quality. The links pointing to it<\/em>are a powerful, objective “vote” or “recommendation” from other authors.<\/li>\n\n\n\n
                                                                                                                                                                      • The Problem:<\/strong>Raw content matching (keyword frequency) is easily spammed and doesn’t understand authority<\/em>.<\/li>\n\n\n\n
                                                                                                                                                                      • The Solution: Link Analysis.<\/strong>By analyzing the entire<\/em>graph structure, a search engine can determine the relative authority, importance, and trustworthiness<\/strong>of every page.<\/li>\n\n\n\n
                                                                                                                                                                      • The Connectivity Server is the system that provides the raw data (the graph) to run these analyses.<\/li>\n<\/ul>\n<\/div>\n<\/div>\n\n\n\n
                                                                                                                                                                        <\/div>\n\n\n\n
                                                                                                                                                                        \n
                                                                                                                                                                        \n

                                                                                                                                                                        Key Algorithms: PageRank, HITS, TrustRank<\/h3>\n\n\n\n

                                                                                                                                                                        The Connectivity Server provides the data to compute:<\/p>\n\n\n\n

                                                                                                                                                                          \n
                                                                                                                                                                        1. PageRank (Google):<\/strong>\n
                                                                                                                                                                            \n
                                                                                                                                                                          • A measure of global, static authority<\/strong>.<\/li>\n\n\n\n
                                                                                                                                                                          • It’s a “random surfer” model: a page is important if other important<\/em>pages link to it. The score is passed recursively through the graph.<\/li>\n<\/ul>\n<\/li>\n\n\n\n
                                                                                                                                                                          • HITS (Hyperlink-Induced Topic Search):<\/strong>\n
                                                                                                                                                                              \n
                                                                                                                                                                            • Identifies two types of pages for a specific query:<\/li>\n\n\n\n
                                                                                                                                                                            • Authorities:<\/strong>Pages with the best information (many in-links).<\/li>\n\n\n\n
                                                                                                                                                                            • Hubs:<\/strong>Pages that are good lists<\/em>of authorities (many out-links).<\/li>\n<\/ul>\n<\/li>\n\n\n\n
                                                                                                                                                                            • TrustRank:<\/strong>\n
                                                                                                                                                                                \n
                                                                                                                                                                              • A spam-fighting algorithm. It starts a PageRank-like “walk” from a seed set of known, trusted<\/em>sites (e.g., universities, government sites) to measure “trust” instead of “popularity.”<\/li>\n<\/ul>\n<\/li>\n<\/ol>\n<\/div>\n\n\n\n
                                                                                                                                                                                \n

                                                                                                                                                                                Anchor Text Indexing<\/h3>\n\n\n\n
                                                                                                                                                                                  \n
                                                                                                                                                                                • The Problem<\/strong>: Page Alinks to Page B. The link looks like this:<\/li>\n\n\n\n
                                                                                                                                                                                • <a href=”http:\/\/pageB.com”>Click here for the best widgets<\/a><\/li>\n\n\n\n
                                                                                                                                                                                • The Solution (Anchor Text): <\/strong>The clickable text (“Click here for the best widgets”) is a description of Page Bwritten by the author of Page A.<\/li>\n\n\n\n
                                                                                                                                                                                • Role of Connectivity Server:<\/strong>\n
                                                                                                                                                                                    \n
                                                                                                                                                                                  • \u2022It doesn’t just store (PageA-> PageB).<\/li>\n\n\n\n
                                                                                                                                                                                  • \u2022It stores the link context: (Source: PageA, Target: PageB, AnchorText: “best widgets”)<\/li>\n<\/ul>\n<\/li>\n\n\n\n
                                                                                                                                                                                  • Impact: <\/strong>This allows Page Bto rank for the term “best widgets” even if that phrase never appears on Page B’s content. It’s a powerful, objective, third-party signal.<\/li>\n<\/ul>\n<\/div>\n<\/div>\n\n\n\n
                                                                                                                                                                                    \n
                                                                                                                                                                                    \n

                                                                                                                                                                                    Efficient Storage of the Link Graph<\/h3>\n\n\n\n
                                                                                                                                                                                      \n
                                                                                                                                                                                    • The Challenge<\/strong>: Storing a graph with billions of nodes and trillions of edges is a massive storage problem.<\/li>\n\n\n\n
                                                                                                                                                                                    • Solution: Adjacency Lists<\/strong>. The graph is stored as a set of lists, not a giant matrix.<\/li>\n\n\n\n
                                                                                                                                                                                    • Two Key Lists are Needed:<\/strong>\n
                                                                                                                                                                                        \n
                                                                                                                                                                                      1. Forward Adjacency List (Out-links):<\/strong>\n
                                                                                                                                                                                          \n
                                                                                                                                                                                        • DocA-> [DocB, DocC, DocF]<\/li>\n\n\n\n
                                                                                                                                                                                        • Use: For HITS (finding Hubs) and crawling (where to go next).<\/li>\n<\/ul>\n<\/li>\n\n\n\n
                                                                                                                                                                                        • Reverse Adjacency List (In-links):<\/strong>\n
                                                                                                                                                                                            \n
                                                                                                                                                                                          • DocB-> [DocA, DocE, DocG]<\/li>\n\n\n\n
                                                                                                                                                                                          • Use: Essential for calculating PageRank and TrustRank(who “votes” for this page?).<\/li>\n<\/ul>\n<\/li>\n<\/ol>\n<\/li>\n\n\n\n
                                                                                                                                                                                          • Compression<\/strong>: These lists are heavily compressed (e.g., delta-encoding sorted DocIDs) to fit as much of the graph as possible in RAM for high-speed lookups.<\/li>\n<\/ul>\n<\/div>\n\n\n\n
                                                                                                                                                                                            \n

                                                                                                                                                                                            Application 1: Ranking & Spam Detection<\/h3>\n\n\n\n
                                                                                                                                                                                              \n
                                                                                                                                                                                            • The Connectivity Server is a primary input to the final ranking formula.<\/li>\n\n\n\n
                                                                                                                                                                                            • Ranking Signals:<\/strong>\n
                                                                                                                                                                                                \n
                                                                                                                                                                                              1. Static Score<\/strong>: The pre-calculated PageRank or TrustRankof a document is used as a strong signal of its general, baseline quality.<\/li>\n\n\n\n
                                                                                                                                                                                              2. Anchor Text<\/strong>: The query processor checks the anchor text index. If a user searches for “best widgets,” any page that has “best widgets” in its incoming anchor text gets a massive ranking boost.<\/li>\n<\/ol>\n<\/li>\n\n\n\n
                                                                                                                                                                                              3. Spam Detection:<\/strong>\n
                                                                                                                                                                                                  \n
                                                                                                                                                                                                • Pages with extremely low TrustRank(far from the trusted seed set) can be heavily demoted or removed from results.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/div>\n<\/div>\n\n\n\n
                                                                                                                                                                                                  \n
                                                                                                                                                                                                  \n

                                                                                                                                                                                                  Application 2: Query Expansion & Disambiguation<\/h3>\n\n\n\n
                                                                                                                                                                                                    \n
                                                                                                                                                                                                  • The link graph helps the search engine understand<\/em> the context<\/em> of a query.<\/li>\n\n\n\n
                                                                                                                                                                                                  • Query Disambiguation:<\/strong>\n
                                                                                                                                                                                                      \n
                                                                                                                                                                                                    • User searches for “Jaguar.”<\/li>\n\n\n\n
                                                                                                                                                                                                    • The server sees two distinct clusters of top pages:\n
                                                                                                                                                                                                        \n
                                                                                                                                                                                                      1. One cluster is linked to by “car,” “Ford,” and “dealership” sites (it’s the car).<\/li>\n\n\n\n
                                                                                                                                                                                                      2. Another cluster is linked to by “zoo,” “big cat,” and “jungle” sites (it’s the animal).<\/li>\n<\/ol>\n<\/li>\n\n\n\n
                                                                                                                                                                                                      3. By analyzing the co-citation<\/strong>(what pages are linked to together<\/em>), the engine can disambiguate the query and show results for both, or pick the most popular context.<\/li>\n<\/ul>\n<\/li>\n\n\n\n
                                                                                                                                                                                                      4. Query Expansion:<\/strong>This same context helps find related terms (e.g., “Jaguar” is related to “Ford,” “Jaguar” is also related to “Leopard”).<\/li>\n<\/ul>\n<\/div>\n\n\n\n
                                                                                                                                                                                                        \n

                                                                                                                                                                                                        Integration with the Full Pipeline<\/h3>\n\n\n\n

                                                                                                                                                                                                        The Connectivity Server is the “glue” between the other components.<\/p>\n\n\n\n

                                                                                                                                                                                                          \n
                                                                                                                                                                                                        1. Crawling (Input):<\/strong>\n
                                                                                                                                                                                                            \n
                                                                                                                                                                                                          • The Crawler<\/strong> is the producer<\/em> of data.<\/li>\n\n\n\n
                                                                                                                                                                                                          • As it fetches and parses pages, it extracts all links and anchor text and sends them as a stream of updates to the Connectivity Server.<\/li>\n<\/ul>\n<\/li>\n\n\n\n
                                                                                                                                                                                                          • Indexing (Offline Process):<\/strong>\n
                                                                                                                                                                                                              \n
                                                                                                                                                                                                            • The Indexer<\/strong>(often using MapReduce) takes a snapshot<\/em>of the entire link graph from the server to run the massive, offline calculations for PageRank and TrustRank.<\/li>\n<\/ul>\n<\/li>\n\n\n\n
                                                                                                                                                                                                            • Querying (Real-time Consumer):<\/strong>\n
                                                                                                                                                                                                                \n
                                                                                                                                                                                                              • The Query Processor<\/strong>is the consumer<\/em>of data.<\/li>\n\n\n\n
                                                                                                                                                                                                              • It requests specific data (e.g., “Give me all anchor text for DocID123”) in real-time to score and rank the final results list.<\/li>\n<\/ul>\n<\/li>\n<\/ol>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n
                                                                                                                                                                                                                \n
                                                                                                                                                                                                                \n
                                                                                                                                                                                                                \n

                                                                                                                                                                                                                Advanced Topics<\/h2>\n\n\n\n
                                                                                                                                                                                                                <\/div>\n<\/div>\n\n\n\n
                                                                                                                                                                                                                \n
                                                                                                                                                                                                                \n

                                                                                                                                                                                                                Semantic Indexing & Latent Semantic Analysis (LSA)<\/h3>\n\n\n\n
                                                                                                                                                                                                                  \n
                                                                                                                                                                                                                • The Problem (Keyword Matching Fails):<\/strong>\n
                                                                                                                                                                                                                    \n
                                                                                                                                                                                                                  • Synonymy:<\/strong>User searches for “laptop” but the best page only says “notebook.” A simple keyword index will fail to match them.<\/li>\n\n\n\n
                                                                                                                                                                                                                  • Polysemy:<\/strong>User searches for “jaguar.” Does it mean the car, the animal, or the operating system? A simple index can’t tell the context.<\/li>\n<\/ul>\n<\/li>\n\n\n\n
                                                                                                                                                                                                                  • The Solution: Semantic Indexing (Index the <\/strong>Meaning<\/em><\/strong>).<\/strong><\/li>\n\n\n\n
                                                                                                                                                                                                                  • Latent Semantic Analysis (LSA):<\/strong>A mathematical technique that analyzes the entire document collection to find which terms co-occur<\/em>(appear together) frequently.\n
                                                                                                                                                                                                                      \n
                                                                                                                                                                                                                    • It learns that “laptop,” “notebook,” “CPU,” and “RAM” all belong to a hidden (“latent”) concept<\/em>of “computing.”<\/li>\n\n\n\n
                                                                                                                                                                                                                    • It creates a mathematical “concept space” where “laptop” and “notebook” are very close together.<\/li>\n<\/ul>\n<\/li>\n\n\n\n
                                                                                                                                                                                                                    • Result:<\/strong>A query for “laptop” can now retrieve documents about “notebooks,” leading to far more relevant results. This is the foundation of modern vector and neural search.<\/li>\n<\/ul>\n<\/div>\n\n\n\n
                                                                                                                                                                                                                      \n

                                                                                                                                                                                                                      Privacy-Preserving & Federated Search<\/h3>\n\n\n\n
                                                                                                                                                                                                                        \n
                                                                                                                                                                                                                      1. Privacy-Preserving Indexing (The “Encrypted Web”):<\/strong>\n
                                                                                                                                                                                                                          \n
                                                                                                                                                                                                                        • The Challenge:<\/strong>How do you search on highly sensitive data (e.g., medical records, financial data) without<\/em>the search engine (or its admin) being able to read the content?<\/li>\n\n\n\n
                                                                                                                                                                                                                        • The Solution:<\/strong>Use Searchable Encryption<\/strong>.<\/li>\n\n\n\n
                                                                                                                                                                                                                        • How:<\/strong>The index itself is built from encrypted data<\/em>. When a user searches, their query is also encrypted. Advanced cryptography (like homomorphic encryption) allows the server to find matching documents while they remain encrypted<\/em>. The server gets the right results but learns nothing about the content.<\/li>\n<\/ul>\n<\/li>\n\n\n\n
                                                                                                                                                                                                                        • Federated Search (The “SiloedWeb”):<\/strong>\n
                                                                                                                                                                                                                            \n
                                                                                                                                                                                                                          • The Challenge:<\/strong>Data is often in separate, “siloed” systems (a hospital DB, a university library, a company’s internal wiki) that cannot be centrally crawled.<\/li>\n\n\n\n
                                                                                                                                                                                                                          • The Solution:<\/strong>A “broker” system. The user sends one<\/em>query. The broker forwards this query to all<\/em>the independent systems, waits for their individual results, and then merges them into a single, unified list for the user. The index remains physically distributed.<\/li>\n<\/ul>\n<\/li>\n<\/ol>\n<\/div>\n<\/div>\n\n\n\n
                                                                                                                                                                                                                            <\/div>\n\n\n\n
                                                                                                                                                                                                                            \n
                                                                                                                                                                                                                            \n

                                                                                                                                                                                                                            Indexing Dynamic & Deep Web Content<\/h3>\n\n\n\n
                                                                                                                                                                                                                              \n
                                                                                                                                                                                                                            • 1.The Dynamic Web (JavaScript-Rendered Content):<\/strong>\n
                                                                                                                                                                                                                                \n
                                                                                                                                                                                                                              • The Problem:<\/strong>Many modern websites load as an empty “shell.” All the content (articles, products) is then loaded dynamically using JavaScript (JS)<\/strong>. A simple crawler only sees the empty shell.<\/li>\n\n\n\n
                                                                                                                                                                                                                              • The Solution:Client-Side Rendering.<\/strong>The crawler must act as a full web browser (using a “headless” browser like Chromium). It loads the page, executes all the JavaScript, waits for the dynamic content to appear, and then indexes the final rendered HTML<\/em>. This is very slow and computationally expensive.<\/li>\n<\/ul>\n<\/li>\n\n\n\n
                                                                                                                                                                                                                              • 2.The Deep Web (Content Behind Forms):<\/strong>\n
                                                                                                                                                                                                                                  \n
                                                                                                                                                                                                                                • The Problem:<\/strong>This is content that is not<\/em> accessible via hyperlinks (e.g., flight search results, library catalogs, your bank account).<\/li>\n\n\n\n
                                                                                                                                                                                                                                • The Solution (Difficult):<\/strong>There is no perfect solution.\n
                                                                                                                                                                                                                                    \n
                                                                                                                                                                                                                                  • Form Submission:<\/strong>Programmatically “guess” and submit common queries to forms (e.g., try all airport codes).<\/li>\n\n\n\n
                                                                                                                                                                                                                                  • Data Feeds\/APIs:<\/strong>Partner with the data provider to get access to their database directly, bypassing the web interface.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/div>\n\n\n\n
                                                                                                                                                                                                                                    \n

                                                                                                                                                                                                                                    Real-World Systems: Caffeine, Elasticsearch, Solr<\/h3>\n\n\n\n
                                                                                                                                                                                                                                      \n
                                                                                                                                                                                                                                    • Google Caffeine (The “Freshness” System):<\/strong>\n
                                                                                                                                                                                                                                        \n
                                                                                                                                                                                                                                      • Advanced Problem:<\/strong>The old web was indexed in massive, slow, batch<\/em> processes (e.g., update the index once a month). This is too slow for breaking news.<\/li>\n\n\n\n
                                                                                                                                                                                                                                      • Caffeine’s Solution:<\/strong>A continuous, incremental<\/em>indexing system. It “percolates” updates, analyzing the web in small, continuous chunks.<\/li>\n\n\n\n
                                                                                                                                                                                                                                      • Result:<\/strong>A new blog post or news story can be crawled, indexed, and made searchable in seconds or minutes<\/em>, not weeks. This is the “real-time” web.<\/li>\n<\/ul>\n<\/li>\n\n\n\n
                                                                                                                                                                                                                                      • Elasticsearch & Apache Solr (The “Enterprise” Systems):<\/strong>\n
                                                                                                                                                                                                                                          \n
                                                                                                                                                                                                                                        • Advanced Problem:<\/strong>How can a normal company get Google-like search?<\/li>\n\n\n\n
                                                                                                                                                                                                                                        • Their Solution:<\/strong>They are open-source platforms<\/em>(built on Apache Lucene) that make distributed indexing “easy.”<\/li>\n\n\n\n
                                                                                                                                                                                                                                        • Key Features:<\/strong>They automatically handle sharding, replication, and fault tolerance. They excel at “Near Real-Time” (NRT) Search<\/strong>, using a segment-based model to make new documents (like application logs or product updates) searchable in under a second.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n

                                                                                                                                                                                                                                          <\/p>\n<\/div>\n\n\n\n

                                                                                                                                                                                                                                          <\/p>\n","protected":false},"excerpt":{"rendered":"

                                                                                                                                                                                                                                          Definition and Role Definition:Web crawling (or “spidering”) is the process of systematically browsing the World Wide Web, typically for the purpose of web indexing. The Crawler:A software bot (also called a spider or web robot) that visits web pages to gather information, starting from a list of “seed” URLs. Role in Search Engines: Discovery:Finds new […]<\/p>\n","protected":false},"author":1,"featured_media":0,"parent":113,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-115","page","type-page","status-publish","hentry"],"_links":{"self":[{"href":"http:\/\/ijeesoo.com\/index.php?rest_route=\/wp\/v2\/pages\/115","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/ijeesoo.com\/index.php?rest_route=\/wp\/v2\/pages"}],"about":[{"href":"http:\/\/ijeesoo.com\/index.php?rest_route=\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"http:\/\/ijeesoo.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/ijeesoo.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=115"}],"version-history":[{"count":65,"href":"http:\/\/ijeesoo.com\/index.php?rest_route=\/wp\/v2\/pages\/115\/revisions"}],"predecessor-version":[{"id":164,"href":"http:\/\/ijeesoo.com\/index.php?rest_route=\/wp\/v2\/pages\/115\/revisions\/164"}],"up":[{"embeddable":true,"href":"http:\/\/ijeesoo.com\/index.php?rest_route=\/wp\/v2\/pages\/113"}],"wp:attachment":[{"href":"http:\/\/ijeesoo.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=115"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}