Towards a Cleaner Document-Oriented Multilingual Crawled Corpus

Abstract

The need for raw large raw corpora has dramatically increased in recent years with the introduction of transfer learning and semi-supervised learning methods to Natural Language Processing. And while there have been some recent attempts to manually curate the amount of data necessary to train large language models, the main way to obtain this data is still through automatic web crawling. In this paper we take the existing multilingual web corpus OSCAR and its pipeline Ungoliant that extracts and classifies data from Common Crawl at the line level, and propose a set of improvements and automatic annotations in order to produce a new document-oriented version of OSCAR that could prove more suitable to pre-train large generative language models as well as hopefully other applications in Natural Language Processing and Digital Humanities.

Publication
In Computation and Language
Julien Abadji
Julien Abadji
Research Engineer

I’m a research engineer at ALMAnaCH research team at Inria

Pedro Ortiz Suarez
Pedro Ortiz Suarez
Researcher

I’m a researcher at the Speech and Language Technology Team at DFKI GmbH Berlin.

Laurent Romary
Laurent Romary
Senior Researcher

Inria Senior Researcher, DARIAH EU infrastructure, director, ISO/TC 37 chair

Benoît Sagot
Benoît Sagot
Senior Researcher

Inria Senior Researcher in Natural Language Processing and Computational Linguistics