Web corpus construction

by Schäfer, Roland

Type Book

Rate 0.0 /5

Details

Files

Reviews

Download

Title Statements

Full title :Web corpus construction / Roland Schäfer and Felix Bildhauer.

Summary

The World Wide Web constitutes the largest existing source of texts written in a great variety of languages. A feasible and sound way of exploiting this data for linguistic research is to compile a static corpus for a given language. There are several advantages of this approach: (i) Working with such corpora obviates the problems encountered when using Internet search engines in quantitative linguistic research (such as non-transparent ranking algorithms). (ii) Creating a corpus from web data is virtually free. (iii) The size of corpora compiled from the WWW may exceed by several orders of magnitudes the size of language resources offered elsewhere. (iv) The data is locally available to the user, and it can be linguistically post-processed and queried with the tools preferred by her/him. This book addresses the main practical tasks in the creation of web corpora up to giga-token size. Among these tasks are the sampling process (i. e., web crawling) and the usual cleanups including boilerplate removal and removal of duplicated content. Linguistic processing and problems with linguistic processing coming from the different kinds of noise in web corpora are also covered. Finally, the authors show how web corpora can be evaluated and compared to other corpora (such as traditionally compiled corpora).

Edition Statements

Publication Date:©2013.

Publication Statements

Publisher(s):Springer
Place of Publication:Cham, Switzerland
Language:eng
Publication Date:©2013.

The Physical Object

Description:1 online resource (xv, 129 pages) : : illustrations

Edition Identifiers

ISBN:9781608459841, 1608459845, 1608459837, 9781608459834, 9781627053129, 1627053123, 9783031021527, 3031021525
DOI:10.1007/978-3-031-02152-7
oclc:ocn855857901
LC Classification:P128.C68
Dewey Decimal:410.188

1. Web corpora.

2. Data collection -- 2.1 Introduction -- 2.2 The structure of the web -- 2.2.1 General properties -- 2.2.2 Accessibility and stability of web pages -- 2.2.3 What's in a (national) top level domain? -- 2.2.4 Problematic segments of the web -- 2.3 Crawling basics -- 2.3.1 Introduction -- 2.3.2 Corpus construction from search engine results -- 2.3.3 Crawlers and crawler performance -- 2.3.4 Configuration details and politeness -- 2.3.5 Seed URL generation -- 2.4 More on crawling strategies -- 2.4.1 Introduction -- 2.4.2 Biases and the pagerank -- 2.4.3 Focused crawling.

3. Post-processing -- 3.1 Introduction -- 3.2 Basic cleanups -- 3.2.1 HTML stripping -- 3.2.2 Character references and entities -- 3.2.3 Character sets and conversion -- 3.2.4 Further normalization -- 3.3 Boilerplate removal -- 3.3.1 Introduction to boilerplate -- 3.3.2 Feature extraction -- 3.3.3 Choice of the machine learning method -- 3.4 Language identification -- 3.5 Duplicate detection -- 3.5.1 Types of duplication -- 3.5.2 Perfect duplicates and hashing -- 3.5.3 Near duplicates, Jaccard coefficients, and shingling.

4. Linguistic processing -- 4.1 Introduction -- 4.2 Basics of tokenization, p…

Subjects

computational linguistics Corpora (Linguistics) -- Data processing Corpus (Linguistique) -- Informatique LANGUAGE ARTS & DISCIPLINES -- Linguist… Linguistique informatique Moteurs de recherche sur Internet Web search engines