How Partitioning Clustering Technique For Implementing Indexing Phase Of Search Engine

For this assignment, I was allowed to improvise on a provided base code to develop a functioning web crawler. The web crawler needed to accept a starting URL and then develop a URL frontier queue of “out links” to be further explored. The crawler needed to track the number of URLs and stop adding them once the queue had reached 500 links. The crawler needed to also extract text and remove HTML tags and formatting. The assignment instructions offered using the BeautifulSoup module to achieve those goals, which I chose to do. Finally, the web crawler program needed to report metrics including the number of documents (web pages), the number of tokens extracted and processed, and the number of unique terms added to the term dictionary.

This section discuss about the common traits or ideas observed in the three research topics. Although, each of the three articles discuss a unique idea, all of them are aimed at utilizing the web data to produce better results. Web data mining is a hot research topic in the current realm of big data. These papers discuss about the utilization of the valuable user generated data from the social media or the browser cookies to provide the best user experience in order to maintain the user interest in the company's product or to take effective decisions by an individual. All the three articles propose an idea to solution the problem stated, compared their results to the existing models and showed significant improvement.

Here we discuss about the common traits or ideas observed in the three research topics. Although, these three papers discuss about different ideas, they all fall under the web data mining domain. web data mining is a hot research topic in the current realm of big data. These papers discuss about the utilisation of the valuable user generated data from the social media or the the browser cookies to provide the best user experience in order to maintain the user interest in the company's product or to take effective decisions by the individual.

With the advent of computer technology in 1990’s the need to search large databases was increasingly becoming vital. The search engines prior to PageRank had limitations, the then most widely used algorithm used text based indexes to provide search results on World Wide Web however had limitations of improper search results as the logic used by the search engines looked at the number of occurrences of the search word in webpage which sometimes resulted in improper search results. Another technique used during the time was based on variations of standard vector space model – i.e. search based on how recent the webpage was updated and/or how close the search terms are to the

More importantly, she mainly covers why Google is the most efficient search engine and how it operates more accurately than other engines and Web browsers. Kraft shares the same positive outlook on Google as the preferred search engine as is evidenced in this paper.

Through the methodology proposed, we aspire to achieve a more efficient technology for generating keywords and finding more accurate data from the search engine. By saving physical memory and storing only what is important rather than all the data from a random website. Also, due to this we may achieve faster response time. So, here we can conclude that the proposed system may be more better than the previous systems

We were introduced to searching and indexing algorithms, we had to cover a lot material related to the different processes to implement these approaches for best results under the assumption of how often a record is accessed.

When comparing the collection of information from a search engine such as Google, and a database such as EBSCO, one can notice many differences and a proper evaluation and assessment is necessary. An evaluation can be done by looking first at the accessibility of an article pertaining to a specific type of information, such as on the

Document clustering is way of automatic organization of documents into clusters so that documents within a cluster have high similarity in comparison to documents in other clusters. It is measuring similarity between documents and grouping similar documents together. The study of similarity measure for clustering is initially motivated by a research on automated text categorization. There was several similarity measures used for document similarity. It provides client representation and visualization of the documents; thus helps in easy navigation also. It has been used intensively because of its wide applicability in various areas such as web mining, search engines, and information retrieval. The key of organizing data in such a way is to improve data availability and to fasten data access, so that web information retrieval and content delivery on the web are improved. The main idea is to improve the accessibility and usability of text mining for various applications. By optimizing similarity measures the optimal clusters can be formed thus performance is improved.

Search engines like Google, Yahoo, Bing and others “crawl” the web for information based on words users type in the search bar. Taking a variety of factors into account, the search engine determines which sites are the

The web is highly dynamic; lots of pages are added, updated and removed everyday and it handle huge set of information hence there is an arrival of many number of problems or issues. Normally, web data is highly dimensional, limited query interfaces, keyword oriented search and limited customization to individual user. Due to this, it is very difficult to find the relevant information from the web which may create new issues. Web mining procedures are classification, clustering and association laws which are used to understand the customer behavior , evaluated a particular website by using traditional data mining parameter. Web mining process is divided into four steps; they are resources finding, data selection and pre-processing, generalization and analysis. Web measurement or web analytics are one of the significant challenged in web mining. The measurement factors are hits, page views, visits or user sessions and find the unique visitor regularly used to measures the user impact of various proposed changes. Large institutions and organizations archived usage data from the web sites. The main problem is that, detecting and/or preventing fraud activities. The web usage mining algorithms are more effective and accurate. But there is a challenge that has to be taken into consideration. Web cleaning is the most significant process but data cleaning becomes difficult when it comes to heterogeneous data. Maintaining accurateness in classifying the

A Web crawler is a type of a computer program that browses the World Wide Web in a logical, automated approach. Cothey (2004) affirms that Web crawlers are used to generate a copy of all the visited web pages (p. 1230). These pages are later processed by a search engine that indexes the downloaded pages to provide quick searches. Crawlers can also be applied to the maintenance of automated tasks on a Web site; such as checking relations or verifying HTML code. Crawlers are employed when gathering a specific type of data from Web pages, such as e-mail addresses collection. Web search engines are becoming gradually more, essential as the main means of tracking relevant information.

Search engine marketing providers have increased in significance and there have been a number of traits enveloping the market. Most web optimization corporations are anticipated to know and perceive the algorithms created for serps, but solely few are in a position to crack it with precision.

Data mining techniques can be mainly divided into three categories: Web structural mining, Web Content mining and web usage mining. Web structural mining is used to discover structure from data available on web like hyperlinks and documents. It can be helpful to the user for navigating within documents as mining can be done to retrieve intra and inter hyperlinks and DOM structure out of documents. Web Content mining can be used to extract information from the data available on web like texts, videos, images, audio files etc. Web usage mining is the application of data mining techniques to discover interesting usage patterns from web usage data, in order to understand and better serve the needs of web-based applications (Srivastava, Cooley, Deshpande, and Tan 2000). Websites or usage data takes user’s available information and browsing history, location of user etc. as input for mining information. Web usage mining can be further divided into three categories depending upon the type of data used for mining: web server logs, application server, application level logs. Web usage mining can be highly helpful in mining the data for web applications and thus helping development in fields like E-Commerce. It can be helpful to discover usage patterns from Web data, thus helping serve better the needs of Web-based applications. Web usage mining can be categorized into three different phases: Preprocessing, Pattern Discovery and Pattern Analysis. I believe, this

This section describes different semantic methodologies being forwarded by the scholars. Currently, various types of search engines are being deployed to access the information required. Each search engine has its own features and uses different algorithms to index, rank and present web documents. Hence the result put forth on information retrieval by the search engines are different from one another. And there is not a definite and unique single technology or architecture that leads to a logical and meaningful search engine. In fact there can be various ways to achieve this.