Extracting the Hyperlink Graph from the Common Web Crawl
Robert Meusel
Oliver Lehmberg
Christian Bizer
Sebastiano Vigna

This page provides detailed download instruction to obtain the hyperlink graph extracted from the Common Crawl 2012 web corpus. The graph covers 3.5 billion web pages and 128 billion hyperlinks between these pages.
We also provide basic statistics about the hyperlink. Please visit the overview page for more information about the provided file formats.


1. Index/Arc Format

The following table contains the links for downloading the index as the arc file(s) of the graph for each aggregation level.

Data SetIndex FileArc File
Page Graphsee below (45 GB)see below (331 GB)
Subdomain/Host Graphdownload (832 MB)download (9.2 GB)
1st Subdomain Graphdownload (757 MB)download (8.7 GB)
PLD Graphdownload (297 MB)download (2.8 GB)
In case you experience any problems downloading the files please find additional information here.

Downloading the page graph: The page graph (arc and indes files) are, due to their size split into in small files of around 500 MB. These files can be downloaded using wget -i http://webdatacommons.org/hyperlinkgraph/2012-08/data/index.list.txt for the index files and respectively wget -i http://webdatacommons.org/hyperlinkgraph/2012-08/data/arc.list.txt for the arc files.

2. WebGraph Files

We also provide the page graph in the format expected by the WebGraph Framework developed by Sebastiano Vigna. The graph is represented using three files: .graph, .offsets, .properties. All three are necessary to load the network into the library.

Using the WebGraph Framework, which can be downloaded from Maven Central, these files can be loaded using the following line of code: BVGraph graph = BVGraph.loadMapped(baseName, new ProgressLogger()).

3. Additional Material

3.1. PLD Graph (.NET)

We offer the PLD-aggregation of the page graph also in the Pajek NET Format which is understood by various graph analysis tools such as Pajek or Gephi. The format combines the index and the arc list into a single file (example file, 106 nodes, 141 arcs). The PLD graph can be downloaded from Pajek version of PLD Graph (2.3 GB) and can directly be imported into Pajek after unzipping. To process the graph in acceptable time, we recommend to run Pajek with at least 32 GB of RAM.

3.2. Ranking Files

Beside the pure graph files, we also calculated for each host in the host graph the harmonic centrality, indegree centrality, Katz's index, and Page Rank. An interactive version of those ranking is available at wwwranking.webdatacommons.org. The incorporated rating data can also be downloaded using the files listed below. Each line within the files consists of the host and the value for the corresponding measure.

4. License

The extracted data is provided according the same terms of use, disclaimer of warranties and limitation of liabilities that apply to the Common Crawl corpus.

The Web Data Commons extraction framework can be used under the terms of the Apache Software License.

5. Feedback

Please send questions and feedback to the Web Data Commons mailing list or post them in our Web Data Commons Google Group.

More information about Web Data Commons is found here.