Project Overview

In this project, you will develop a multithreaded web crawler in C that can efficiently fetch,

parse, and process web pages from the internet. The goal is to utilize multiple threads to

improve the performance and efficiency of the web crawling process.

Problem Statement

The internet is vast and ever$-$growing. Traditional single$-$threaded web crawlers are not

efficient enough to handle the volume of information available. The challenge is to develop

a web crawler that can navigate the web efficiently by fetching multiple pages in parallel,

thus significantly reducing the time required to crawl a set of URLs.