The world wide web conjures up images of a giant spider web where everything is linked with the rest in a random pattern and you can go collected from one of edge of the web to another by just following the right links. Theoretically, that’s what makes the web distinctive from of typical list system dark web sites: You can follow hyperlinks collected from one of page to another. In the “small world” theory of the web, every web page is perceived as separated from any other Web page by an average of about 19 clicks. In 1968, sociologist Stanley Milgram invented small-world theory for social networks by noting that every human was separated from any other human by only six degree of divorce. On the web, the miscroscopic world theory was supported by early research on a small eating of web sites. But research conducted along by scientists at IBM, Compaq, and Alta Vista found something entirely different. These scientists used a web crawler to spot 200 million Web pages and follow 1. 5 billion links on these pages.
The science tecnistions found that the web was not like a spider web at all, but alternatively like a bow tie. The bow-tie Web had a inches strong connected component” (SCC) composed of about 56 million Web pages. On the right side of the bow tie was a couple of 44 million OUT pages that you could get from the center, but could not get back to the guts from. OUT pages assisted to be corporate intranet and other web sites pages that are designed to trap you at the site when you land. On the left side of the bow tie was a couple of 44 million IN pages from which you can get to the center, but that you could not happen to be from the center. Just read was recently created pages that had not yet been connected to many middle pages. In addition, 43 million pages were classified as inches tendrils” pages that did not hyperlink to the guts and may even not be connected to from the center. However, the tendril pages were sometimes connected to IN and/or OUT pages. Occasionally, tendrils linked to one another without passing through the center (these are called “tubes”). Finally, there were 16 million pages totally turned off from everything.
Further evidence for the non-random and structured nature of the Web is provided in research performed by Albert-Lazlo Barabasi at the University of Notre Dame. Barabasi’s Team found that far from being a random, exponentially exploding network of 50 billion Web pages, activity on the web was actually highly concentrated in “very-connected super nodes” that provided the on-line to less well-connected nodes. Barabasi called this type of network a “scale-free” network and found parallels in the growth of cancers, diseases transmission, and computer worms. As its ends up, scale-free networks are highly at risk of break down: Destroy their super nodes and transmission of messages breaks down rapidly. On the upside, if you are a marketer trying to “spread the message” about your products, place your products on one of the super nodes and watch what is the news spread. Or build super nodes and attract a huge audience.
Thus the picture of the web that emerges from this research is quite distinctive from earlier reports. The notion that most pairs of web pages are separated by a handful of links, almost always under 20, and that the number of connections would grow exponentially with the size of the web, is not supported. In fact, there is a 75% chance that there is no path collected from one of randomly chosen page to another. With this knowledge, it now becomes clear why the most advanced web search engines only list a very small percentage of all web pages, and only about 2% of the overall population of internet hosts(about 400 million). Search engines cannot find most web sites because their pages are not well-connected or for this central core of the web. Another important finding is the identification of a “deep web” composed of over 900 billion web pages are not easy to access to web robots that most search engine companies use. Instead, these pages are either private (not available to robots and non-subscribers) like the pages of (the Wall Street Journal) or are not common from web pages. Within the last few few years newer search engines (such as the medical search engine Mammaheath) and older ones such as yahoo have been revised to search the deep web. Because e-commerce revenues partly depend on customers being able to find a web site using search engines, web site operators need to take measures to ensure their web pages are area of the connected central core, or “super nodes” of the web. One way to do this is to make sure the site has as many links as possible to and from other relevant sites,