Dataset: WikiLinkGraphs' RawWikilinks

This dataset contains wikilinks, i.e. links between Wikipedia articles, extracted by processing each revision of each Wikipedia article (namespace 0) from Wikimedia’s history dumps for the languages de, en, es, fr, it, nl, pl, ru, sv.

WikiLinkGraphs. This dataset is part of the WikiLinkGraphs family, a collection of datasets extracted from Wikipedia history dumps. See the other datasets:

Description

page_id: an integer, the page identifier used by MediaWiki. This identifier is not necessarily progressive, there may be gaps in the enumeration;
page_title: a string, the title of the Wikipedia article;
revision_id: an integer, the identifier of a revision of the article, also called a permanent id, because it can be used to link to that specific revision of a Wikipedia article;
revision_parent_id: an integer, the identifier of the parent revision. In general, each revision as a unique parent; going back in time before 2002, however, we can see that the oldest articles present non-linear edit histories. This is a consequence of the import process from the software previously used to power Wikipedia, MoinMonWiki, to MediaWiki;
revision_timestamp: date and time of the edit that generated the revision under consideration;
user_type: a string ("registered" or "anonymous"), specifying whether the user making the revision was logged-in or not;
user_username: a string, the username of the user that made the edit that generated the revision under consideration;
user_id: an integer, the identifier of the user that made the edit that generated the revision under consideration;
revision_minor: a boolean flag, with value 1 if the edit that generated the current revision was marked as minor by the user, 0 otherwise;
wikilink.link: a string, the page linked by the wikilink;
wikilink.tosection: a string, the name of the section if the link points to a section;
wikilink.anchor: a string, the anchor text of the wikilink;
wikilink.section_name: the name of the section wherein the wikilink appears;
wikilink.section_level: the level of the section wherein the wikilink appears;
wikilink.section_number: the number of the section wherein the wikilink appears;

Sample

Extract of the file enwiki-20180301-pages-meta-history1.xml-p10p2115.7z.revisionlist.csv.gz in enwiki/20180301/:

page_id,page_title,revision_id,revision_parent_id,revision_timestamp,user_type,user_username,user_id,revision_minor,wikilink.link,wikilink.tosection,wikilink.anchor,wikilink.section_name,wikilink.section_level,wikilink.section_number
10,AccessibleComputing,862220,233192,2002-02-25T15:43:11Z,registered,Conversion script,0,1,Accessible Computing,,Accessible Computing,---~--- incipit ---~---,0,0
10,AccessibleComputing,15898945,862220,2003-04-25T22:18:38Z,registered,Ams80,7543,1,Accessible_computing,,Accessible_computing,---~--- incipit ---~---,0,0
10,AccessibleComputing,56681914,15898945,2006-06-03T16:55:41Z,registered,Nzd,516514,1,Computer accessibility,,Computer accessibility,---~--- incipit ---~---,0,0
10,AccessibleComputing,74466685,56681914,2006-09-08T04:16:04Z,registered,Rory096,750223,0,Computer accessibility,,Computer accessibility,---~--- incipit ---~---,0,0
10,AccessibleComputing,133452289,133180268,2007-05-25T17:12:12Z,registered,Gurch,241822,1,Computer accessibility,,Computer accessibility,---~--- incipit ---~---,0,0
10,AccessibleComputing,381202555,381200179,2010-08-26T22:38:36Z,registered,OlEnglish,7181920,1,Computer accessibility,,Computer accessibility,---~--- incipit ---~---,0,0
10,AccessibleComputing,631144794,381202555,2014-10-26T04:50:23Z,registered,Paine Ellsworth,9092818,0,Computer accessibility,,Computer accessibility,---~--- incipit ---~---,0,0
10,AccessibleComputing,767284433,631144794,2017-02-25T00:30:28Z,registered,Godsy,23257138,0,Computer accessibility,,Computer accessibility,---~--- incipit ---~---,0,0
12,Anarchism,18201,332419362,2002-02-25T15:00:22Z,registered,Conversion script,0,1,libertarian socialism,,Libertarian socialists,---~--- incipit ---~---,0,0

Download

This dataset can be downloaded in two different ways:

HTTP (preferred method)

You can find the dataset on: cricca.disi.unitn.it/datasets/wikilinkgraphs-rawwikilinks.

You can download the dataset with the following command:

dataset='wikilinkgraphs-rawwikilinks'; adate=20180301; \
langs=( 'dewiki' 'enwiki'  'eswiki'  'frwiki'  'itwiki'  'nlwiki'  'plwiki'  'ruwiki' 'svwiki' ); \
for lang in "${langs[@]}"; do
  lynx \
    -dump \
    -listonly \
      "http://cricca.disi.unitn.it/datasets/${dataset}/${lang}/${adate}/" | \
  awk '{print $2}' | \
  grep -E "^http://cricca\.disi\.unitn\.it/datasets/${dataset}/" | \
  xargs -L1 -I{} wget -R '\?C=' {}
done

dat (experimental)

(coming soon)

Code

This dataset has been processed with Python, see the wikidump project and the other repositories in the WikiLinkGraphs organization.

Authors

This dataset has been produced by:

Cristian Consonni – DISI, University of Trento, Trento, Italy.
David Laniado – Eurecat, Centre Tecnològic de Catalunya, Barcelona, Spain.
Alberto Montresor – DISI, University of Trento, Trento, Italy.

This dataset has been produced as part of the research related to the ENGINEROOM project. EU ENGINEROOM has received funding from the European Union’s Horizon 2020 research and innovation programme under the Grant Agreement no 780643.

License

This dataset is released under Creative Commons Attribution 4.0 International.

The original dump is released under the GNU Free Documentation License (GFDL) and the Creative Commons Attribution-Share-Alike 3.0 License, see the legal info.

How to cite

If you use this dataset please cite the main WikiLinkGraphs paper:

Consonni, Cristian, David Laniado, and Alberto Montresor. “WikiLinkGraphs: A complete, longitudinal and multi-language dataset of the Wikipedia link networks.”

FAQs

What is the total size of the dataset, the number of files and the largest file in the dataset?

The total dataset size is 1.2TB, divided among the languages like this:

110G dewiki/
638G enwiki/
94G eswiki/
111G frwiki/
77G itwiki/
23G nlwiki/
31G plwiki/
73G ruwiki/
13G svwiki/

The dataset contains 978 files. The average file size is 1.3GB and the largest file is ~13GB (the file from svwiki, which is just one chunk).

How are files organized?

Files are divided in directories, one for each language, like this:

.
├── dewiki
│   └── 20180301
│       ├── dewiki-20180301-pages-meta-history1.xml-p1p3405.7z.features.xml.gz
│       ├── dewiki-20180301-pages-meta-history1.xml-p3406p6349.7z.features.xml.gz
│       ├── ...
│       └── dewiki-20180301-pages-meta-history4.xml-p9990172p10238969.7z.features.xml.gz
├── enwiki
│   └── 20180301
│       ├── enwiki-20180301-pages-meta-history1.xml-p10p2115.7z.features.xml.gz
│       ├── enwiki-20180301-pages-meta-history1.xml-p2116p4236.7z.features.xml.gz
│       ├── ...
│       └── enwiki-20180301-pages-meta-history27.xml-p56237363p56724160.7z.features.xml.gz
├── eswiki
│   └── 20180301
│       ├── eswiki-20180301-pages-meta-history1.xml-p5p3721.7z.features.xml.gz
│       ├── eswiki-20180301-pages-meta-history1.xml-p3722p7536.7z.features.xml.gz
│       ├── ...
│       └── eswiki-20180301-pages-meta-history4.xml-p8089658p8379527.7z.features.xml.gz
├── frwiki
│   └── 20180301
│       ├── frwiki-20180301-pages-meta-history1.xml-p3p3554.7z.features.xml.gz
│       ├── frwiki-20180301-pages-meta-history1.xml-p3555p9507.7z.features.xml.gz
│       ├── ...
│       └── frwiki-20180301-pages-meta-history4.xml-p9743778p10092297.7z.features.xml.gz
├── itwiki
│   └── 20180301
│       ├── itwiki-20180301-pages-meta-history1.xml-p2p12411.bz2.features.xml.gz
│       ├── itwiki-20180301-pages-meta-history1.xml-p12412p29362.bz2.features.xml.gz
│       ├── ...
│       └── itwiki-20180301-pages-meta-history4.xml-p6535051p6847991.bz2.features.xml.gz
├── nlwiki
│   └── 20180301
│       ├── nlwiki-20180301-pages-meta-history1.xml-p1p6842.7z.features.xml.gz
│       ├── nlwiki-20180301-pages-meta-history1.xml-p6843p18427.7z.features.xml.gz
│       ├── ...
│       └── nlwiki-20180301-pages-meta-history4.xml-p4953496p4955587.7z.features.xml.gz
├── plwiki
│   └── 20180301
│       ├── plwiki-20180301-pages-meta-history1.xml-p2p7835.7z.features.xml.gz
│       ├── plwiki-20180301-pages-meta-history1.xml-p7836p27363.7z.features.xml.gz
│       ├── ...
│       └── plwiki-20180301-pages-meta-history4.xml-p3751406p4229714.7z.features.xml.gz
├── ruwiki
│   └── 20180301
│       ├── ruwiki-20180301-pages-meta-history1.xml-p4p5387.7z.features.xml.gz
│       ├── ruwiki-20180301-pages-meta-history1.xml-p5388p15885.7z.features.xml.gz
│       ├── ...
│       └── ruwiki-20180301-pages-meta-history4.xml-p6895239p7350163.7z.features.xml.gz
└── svwiki
    └── 20180301
        └── svwiki-20180301-pages-meta-history.xml.7z.features.xml.gz

Who produced this dataset and why?

This dataset has been produced by Cristian Consonni, David Laniado and Alberto Montresor.
Cristian Consonni and Alberto Montresor are affiliated with the Department of Information Engineering and Computer Science (DISI), University of Trento, Trento, Italy; David is affiliated with Eurecat - Centre Tecnològic de Catalunya, Barcelona, Spain.
This dataset has also been produced as part of the research related to the ENGINEROOM project. EU ENGINEROOM has received funding from the European Union’s Horizon 2020 research and innovation programme under the Grant Agreement no 780643.

Questions?

For further info send me an e-mail.

Cristian Consonni