With the former, a collection can have several copies of web pages grouped together based on the crawl they were found in. For the second, only the most recent copy of the web pages needs to be saved. For this, you need to keep track of when the web page changed and how often it was changed. This technique is more efficient than the previous one but requires an indexing module to run alongside the scanning module. The authors conclude that an incremental crawler can bring in new copies of web pages more quickly and keep the storage area fresher than a periodic crawler.III. CRAWLING TERMINOLOGYThe web crawler maintains a list of unvisited URLs called a frontier. The list begins with launch URLs that may be provided by a different user or program. Each crawl cycle involves selecting the next URL to crawl from the frontier, obtaining the web page equivalent to the URL, parsing the retrieved web page to extract the URLs and application-specific information, and finally adding of unvisited URLs at the border. The scanning process may end when a specific number of web pages have been scanned. The WWW is observed as a huge graph with web pages as nodes and links as edges. A crawler starts from some nodes and then follows the edges to get to other nodes. The process of retrieving a web page and extracting the links within it is similar to expanding a node in graph search. A topic crawler attempts to follow edges that should lead to portions of the graph related to a topic. Frontier: The crawl method initializes with a seed URL, extracting links from it and adding them to a list of unvisited URLs. The list of unvisited URLs is known as Frontier. The frontier is base... middle of the card... level) until the entire website is navigated. After creating this list of URLs, the second part of our application will start getting the HTML text of each link in the list and saving it as a new record in the database. There is only one central database for storing all the web pages. The following figure is the snapshot of the user interface of the Web Crawler application, designed in the VB.NET Windows application, for crawling a website or of any web application using this crawler must require an Internet connection and as input use the URL in the format shown in the figure. At each crawling stage, the program selects the top URL from the border and sends the data of this website to a unit that will download the web pages from the website. For this implementation we use multithreading to parallelize the scanning process so we can download many websites in parallel.