Detecting Anatomical and Functional Connectivity Relations in Biomedical Literature via Language Representation Models

IB Ozyurt, J Menke, A Bandrowski, M Martone - … of the Second Workshop on Scholarly …, 2021

… For word embeddings we have used 300 dimensional Common Crawl (840B tokens) trained GloVe (Pennington et al., 2014) vectors. The de- pendency parse trees for the input sentences were generated via Stanford CoreNLP (Manning et al., 2014) package …


Detecting Discrete Emotions in Text Using Neural Networks

Page 1. DETECTING DISCRETE EMOTIONS IN TEXT USING NEURAL NETWORKS by Seyed Armin Seyeditabari A dissertation submitted to the faculty of The University of North Carolina at Charlotte in partial fulfillment of the …


Detecting ethnicity-targeted hate speech in Russian social media texts

E Pronoza, P Panicheva, O Koltsova, P Rosso - Information Processing & …, 2021

JavaScript is disabled on your browser. Please enable JavaScript to use all the features on this page. Skip to main content Skip to article …


Detecting formal thought disorder by deep contextualized word representations

J Sarzynska-Wawer, A Wawer, A Pawlak… - Psychiatry Research, 2021


Detecting Hate Speech on Social Media Using Deep Learning Techniques

… A FastText pre-trained model was obtained from Facebook's FastText website which contained 2 million word vectors with 300 dimensions trained with subword information on Common Crawl (consisting of a total of 600 billion tokens) …


Detecting Online Risks and Supportive Interaction in Instant Messenger Conversations using Czech Transformers

O Sotolář, J Plhák, M Tkaczyk, M Lebedíková… - RASLAN 2021 Recent …, 2021

We present a comparison of state-of-the-art models for text classification of Online Risks and Supportive Interaction in anonymized Instant Messenger conversations held in Czech. We compare the transformer models Czert, RobeCzech, and FERNET-C5 …


Detecting Phishing Sites--An Overview

P Kalaharsha, BM Mehtre - arXiv preprint arXiv:2103.12739, 2021

… Some of the common sources of datasets are shown below in Table 1. Alexa and Common crawl contains names of the legitimate sites which are likely to be used for phishing [62][63] … 2 Common-Crawl [63] 940 million URLs 2.8 billion webpages …


Detecting the target of sarcasm is hard: Really??

P Parameswaran, A Trotman, V Liesaputra, D Eyers - Information Processing & …, 2021


Detection and Morphological Analysis of Novel Russian Loanwords

… is compiled. The first dataset is the CC-100 Russian dataset (Conneau et al., 2020, Wenzek et al., 2020), which includes Common Crawl2 data from January 2018 to February 2019. A second … 3. 2 https://commoncrawl.org/ Page 16. 6 …


Detection of phishing attacks using probabilistic neural network with a novel training algorithm for reduced Gaussian kernels and optimal smoothing parameter …

S Priya, S Selvakumar - International Journal of Ad Hoc and Ubiquitous …, 2021

Page 1. Int. J. Ad Hoc and Ubiquitous Computing, Vol. 36, No. 2, 2021 67 Detection of phishing attacks using probabilistic neural network with a novel training algorithm for reduced Gaussian kernels and optimal …


Detection of Phishing Websites from URLs by using Classification Techniques on WEKA

B Geyik, K Erensoy, E Kocyigit - 2021 6th International Conference on Inventive …, 2021

… We get the dataset from [21]. In this dataset, legitimate sites are collected from common-crawl and Alexa while phishing sites are collected from Phishtank. After some pre-processing steps 122,055 values 85,220 with …


Development and Evaluation of Novel Ophthalmology Domain-Specific Neural Word Embeddings to Predict Visual Prognosis

S Wang, B Tseng, T Hernandez-Boussard - International Journal of Medical …, 2021

… entity recognition, sentence classification, relation extraction, and other general natural language processing tasks,[8] and general domain GloVe (Global Vectors for Word Representation), pre-trained on large corpora of general …


Development of a language model and opinion extraction for text analysis of online platforms

Page 1. Lakehead University Knowledge Commons,http://knowledgecommons. lakeheadu.ca Electronic Theses and Dissertations Electronic Theses and Dissertations from 2009 2021 Development of a language model and opinion …


DFKI SLT at GermEval 2021: Multilingual Pre-training and Data Augmentation for the Classification of Toxicity in Social Media Comments

… It was trained on the Common Crawl corpus in 100 languages using masked language modeling. We choose XLM-RoBERTa instead of Multilingual BERT5 because XLM-RoBERTa outperforms Multilingual BERT on a variety of cross-lingual benchmarks …


Dictionary-based Debiasing of Pre-trained Word Embeddings

M Kaneko, D Bollegala - arXiv preprint arXiv:2101.09525, 2021

… 3M words learned from Google News corpus (Mikolov et al., 2013a)), GloVe4 (300dimensional embeddings for ca. 2.1M words learned from the Common Crawl (Pennington et al., 2014)), and fastText5 (300-dimensional embeddings for ca …


Different Keystrokes for Different Folks: Visualizing Crowdworker Querying Behavior

R Benham, J Mackenzie, JS Culpepper, A Moffat - 2021

… In this paper, we present the Common Crawl News Query Ex- plorer, 1 which enables information retrieval practitioners to ob- serve and compare human query formulation a posteriori … Common Crawl News Query Explorer …


Difficulty-as-improvement: The courage to keep going in the face of life's difficulties

V Yan, D Oyserman, G Kiper, M Atari - 2021

… Our computational linguistic analyses of the “Common Crawl” corpus suggest that when people talk about difficulty they also talk about importance, impossibility, and improvement (Study 1). Studies 2-14 (total N = 2,378) use our brief difficulty-as-improvement …


Digital humanities and web archives: Possible new paths for combining datasets

N Brügger - International Journal of Digital Humanities, 2021

… Only a limited number of web archives are open to the wider public, most notably the US based Internet Archive, but also the Portuguese, the Croatian, the Australian, and the Icelandic web archives, and open collections such …


Discovering prerequisite relations from educational documents through word embeddings

F Gasparetti - Future Generation Computer Systems, 2021


Discovering Representation Sprachbund For Multilingual Pre-Training

Y Fan, Y Liang, A Muzio, H Hassan, H Li, M Zhou… - arXiv preprint arXiv …, 2021

… We use Wikipedia1 corpora (100 languages are included, total size 101GB) and a clean version of Common Crawl (CC)2 (89 languages are included, total size 2500GB) following Liang et al. (2020). 108 languages are included in our multilingual corpora …


Discovering X Degrees of Keyword Separation in a Fine Arts Collection

… Please note that we translate all keywords from German to English for this paper. We use the German fasttext word embedding, which has been trained on about 3 million words from the Wikipediaand 19 million words from …


Discrete and Soft Prompting for Multilingual Models

M Zhao, H Schütze - arXiv preprint arXiv:2109.03630, 2021

… Setup. We conduct all experiments using the pretrained XLM-RoBERTa-base model (Conneau et al., 2020) containing 270M parameters trained on 2.5 TB CommonCrawl data in 100 languages. We use PyTorch (Paszke et al., 2019) and the Hugprompt …


Discrete Cosine Transform as Universal Sentence Encoder

N Almarwani, M Diab - arXiv preprint arXiv:2106.00934, 2021

… Specifically, we used a sample of 1 million parallel sentences from WMT'13 common-crawl data; this subset is the same one used in (Aldarmaki and Diab, 2019).4 To assess efficacy of the DCT models for the cross …


Discriminative Reranking for Neural Machine Translation

… We follow the steps in Ng et al. (2019) for data preprocessing, including sentence deduplication, language identification filtering on all bitext and monolingual data (Joulin et al., 2017) and in- domain filtering (Moore …


Dissociating Semantic and Phonemic Search Strategies in the Phonemic Verbal Fluency Task in early Dementia

H Lindsay, P Mueller, N Linz, R Zeghari, MM Mina… - Proceedings of the Seventh …, 2021

… To obtain semantic word embeddings, the pretrained French fastText model is used. This model is trained on Common Crawl and Wikipedia corpora using the continuous bag of words (CBOW) algorithm with a negative sampling loss function …


DistilBERT-based Argumentation Retrieval for Answering Comparative Questions

A Alhamzeh, M Bouhaouel, E Egyed-Zsigmond… - Working Notes of CLEF, 2021

… doi:10.1007/ 978-3-030-22948-1\_5. [16] J. Bevendorff, B. Stein, M. Hagen, M. Potthast, Elastic ChatNoir: Search Engine for the ClueWeb and the Common Crawl, in: L. Azzopardi, A. Hanbury, G. Pasi, B. Piwowarski (Eds.), Advances in Information Retrieval …


Page 1. DISTRIBUTIONAL INVESTIGATION OF SOME FREQUENT TURKISH DERIVATIONAL AFFIXES FOR EXPLORING THEIR SEMANTICS A THESIS SUBMITTED TO THE GRADUATE SCHOOL OF INFORMATICS OF THE …


Diverse Distributions of Self-Supervised Tasks for Meta-Learning in NLP

T Bansal, K Gunasekaran, T Wang, T Munkhdalai… - arXiv preprint arXiv …, 2021

… This performance drops even further (≈ 10%; r2c4) when the tasks are sampled from a different domain (common crawl) than the training domain … Figure 2 further shows that, for the Cluster tasks, constructing tasks from more diverse domain such …


DLJUST at SemEval-2021 Task 7: Hahackathon: Linking Humor and Offense

… Training the model on larger data set significantly improves the model performance using BookCurpus, English Wikipedia, CC-news with 63 million English news articles, OpenWebText, and Stories; which is a subset of CommonCrawl data …


Do Images really do the Talking? Analysing the significance of Images in Tamil Troll meme classification

SU Hegde, A Hande, R Priyadharshini, S Thavareesan… - arXiv preprint arXiv …, 2021

… To address data imbalance during pretraining, the low-resource data are up- sampled while the high-resourceful data are downsampled to smoothen the data. The model is pretrained from scratch using the Wikipedia2, Common …


Do Syntactic Probes Probe Syntax? Experiments with Jabberwocky Probing

RH Maudslay, R Cotterell - Proceedings of the 2021 Conference of the North …, 2021

Page 1. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 124–131 June 6–11, 2021. ©2021 Association for Computational Linguistics 124 …


DOCmT5: Document-Level Pretraining of Multilingual Language Models

CH Lee, A Siddhant, V Ratnakar, M Johnson - arXiv preprint arXiv:2112.08709, 2021

In this paper, we introduce DOCmT5, a multilingual sequence-to-sequence language model pre-trained with large scale parallel documents. While previous approaches have focused on leveraging sentence-level parallel data, we try to build …


Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus

J Dodge, M Sap, A Marasovic, W Agnew, G Ilharco…

… In addition to C4.EN, we host the “uncleaned” version (C4.EN.NOCLEAN), which is the snapshot of Common Crawl identified as … 5https://github.com/ allenai/c4- documentation/discussions 6https://commoncrawl.org/, where …


Documenting the English Colossal Clean Crawled Corpus

J Dodge, M Sap, A Marasovic, W Agnew, G Ilharco… - arXiv preprint arXiv …, 2021

… been established. In this work we provide the first documentation for the Colossal Clean Crawled Corpus (C4; Raffel et al., 2020), a dataset created by applying a set of filters to a single snapshot of Common Crawl. We begin …


Does Robustness Improve Fairness? Approaching Fairness with Word Substitution Robustness Methods for Text Classification

Y Pruksachatkun, S Krishna, J Dhamala, R Gupta… - arXiv preprint arXiv …, 2021

… We take 97,320 examples from the public leaderboard as the test set. • Bias in Bios (De-Arteaga et al., 2019) 3 is a dataset for occupation classification derived from Common Crawl corpus. It consists of 178,619 train and 91,917 test examples …


Does semantics aid syntax? An empirical study on named entity recognition and classification

X Zhong, E Cambria, A Hussain - Neural Computing and Applications, 2021

Many researchers jointly model multiple linguistic tasks (eg, joint modeling of named entity recognition and named entity classification and joint modeli.


Domain-specific Taxonomy Enrichment based on Meta-Embeddings

In this paper we study the use of meta-embeddings approaches, which combine several source embeddings, for the taxonomy class prediction of new terms. We test the proposed approach in the informationsecurity domain in the task of enriching the …


Don't Stop at the Top: Using Certificate Transparency Logs to Extend Domain Lists for Web Security Studies

F Marquardt, C Schmidt - 2020 IEEE 45th Conference on Local Computer …, 2020

… They propose a system that collects massive amounts of records from various data sources, including data from top lists, blacklists, the Common Crawl project and DNS zone files. From these data sources, they were able to extract 199,110,841 unique domain names …


DRo: A data-scarce mechanism to revolutionize the performance of DL-based Security Systems

M Sewak, SK Sahay, H Rathore - 2021 IEEE 46th Conference on Local Computer …, 2021

… For pre-training a Transformer based TL language models like the OpenAI's GPT-3 [7] or the Google's BERT [5], billions to even trillions of (web) common-crawl documents could be easily and inexpensively made available …


DS-TOD: Efficient Domain Specialization for Task Oriented Dialog

CC Hung, A Lauscher, SP Ponzetto, G Glavaš - arXiv preprint arXiv:2110.08395, 2021

Recent work has shown that self-supervised dialog-specific pretraining on large conversational datasets yields substantial gains over traditional language modeling (LM) pretraining in downstream task-oriented dialog (TOD). These approaches, however …


DSGPT: Domain-Specific Generative Pre-Training of Transformers for Text Generation in E-commerce Title and Review Summarization

X Zhang, Y Jiang, Y Shang, Z Cheng, C Zhang, X Fan… - Proceedings of the 44th …, 2021

… powerful. Pre-training a language model from a general corpus such as Wikipedia or the Common Crawl requires tremendous time and resource commitment, and can be wasteful if the downstream tasks are limited in variety …


Dual-Objective Fine-Tuning of BERT for Entity Matching

… WDC LSPC datasets: We use the training and test sets from the WDC Product Data Corpus for Large-scale Product Matching (WDC LSPC)1 [27] for the evaluation. The datasets were built by ex- tracting product offers from the Common Crawl …


P Demotte, S Ranathunga - arXiv preprint arXiv:2109.04762, 2021

Page 1. Dual-State Capsule Networks for Text Classification Piyumal Demotte, Surangika Ranathunga Department of Computer Science and Engineering, University of Moratuwa Katubedda 10400, Sri Lanka {piyumalanthony.16,surangika}@cse.mrt.ac.lk Abstract …


Z Liu, H Wang, ZY Niu, H Wu, W Che - arXiv preprint arXiv:2109.08877, 2021

Page 1. DuRecDial 2.0: A Bilingual Parallel Corpus for Conversational Recommendation Zeming Liu1∗, Haifeng Wang2, Zheng-Yu Niu2, Hua Wu2, Wanxiang Che1† 1Research Center for Social Computing and Information …


Dynamic Knowledge Capitalization in Big Data Context: Monte Carlo Tree Search Based Reinforcement Learning

B Hirchoua, B Ouhbi, B Frikh - The International Conference on Artificial Intelligence …, 2021

… 1. Fig. 1. Sub-trees examples alongside their local rewards. In order to demonstrate the effectiveness and efficiency of the proposed approach, a Common Crawl big data set is used for evaluation. Texts in this collection …


Dynamic Tuning and Weighting of Meta-learning for NMT Domain Adaptation

Z Song, Z Ma, K Qi, G Liu - International Conference on Artificial Neural Networks, 2021

… 4 Experiments. 4.1 Datasets. Experiments are conducted on English-German parallel datasets. We use 6.4M sentence pairs to get a pre-trained model. The training data are sampled from CommonCrawl, Europarl, NewsCommentary, Wikititles and Rapid2019 …


Easy-to-use combination of POS and BERT model for domain-specific and misspelled terms

… – Oscar [19] is a set of monolingual corpora extracted from Common Crawl. It was selected using a classification model for each language following the approach of [6] based on FastText [12]. The classifier was previously pretrained on Wikipedia …


M Wojcieszak, A Casas, X Yu, J Nagler, JA Tucker - 2021

Page 1. DRAFT Echo chambers revisited: The (overwhelming) sharing of in-group politicians, pundits and media on Twitter Magdalena Wojcieszak a,b , Andreu Casas c , Xudong Yu a , Jonathan Nagler d , and Joshua A. Tucker d …


ECKPN: Explicit Class Knowledge Propagation Network for Transductive Few-shot Learning

C Chen, X Yang, C Xu, X Huang, Z Ma - Proceedings of the IEEE/CVF Conference on …, 2021

… Specifically, we first leverage the GloVe (pretrained on a large text corpora with self-supervised constraint) [33] to obtain the d1-dimensional semantic embeddings of class labels. The Common Crawl version of the GloVe is …


H Zhang, M Zhao, C Wei, D Mantini, Z Li, Q Liu - Journal of Neural Engineering, 2021

Page 1. Journal of Neural Engineering ACCEPTED MANUSCRIPT • OPEN ACCESS EEGdenoiseNet: A benchmark dataset for end-to-end deep learning solutions of EEG denoising To cite this article before publication: Haoming Zhang et al 2021 J. Neural Eng …


Effective Seed-Guided Topic Labeling for Dataless Hierarchical Short Text Classification

… We set the number of iterations to 8 as our model achieves competitive performance since then. For word embeddings, we employ the widely used GloVe Common Crawl6 [21], which contains 840B tokens, 2.2M vocab and 300d vectors …


Efficiency and Scalability of Large Search Architectures

Page 1. Page 2. 4/21/2021 04/25/2021 Page 3. iii Micro lm or other copies of this dissertation are obtainable from UMI Dissertation Publishing ProQuest CSA 789 E. Eisenhower Parkway PO Box 1346 Ann Arbor, MI 48106-1346 Page 4. iv V …


Efficient Large Scale Language Modeling with Mixtures of Experts

M Artetxe, S Bhosale, N Goyal, T Mihaylov, M Ott… - arXiv preprint arXiv …, 2021

Mixture of Experts layers (MoEs) enable efficient scaling of language models through conditional computation. This paper presents a detailed empirical study of how autoregressive MoE language models scale in comparison with dense models …


Efficient Passage Retrieval with Hashing for Open-domain Question Answering

I Yamada, A Asai, H Hajishirzi - arXiv preprint arXiv:2106.00882, 2021

… the index of a common knowledge source (eg, Wikipedia) requires dozens of gigabytes.1 A reduction in the index size is critical for real-world QA that requires large knowledge sources such as scientific databases (eg, PubMed) …


Emerging trends: A gentle introduction to fine-tuning

KW Church, Z Chen, Y Ma - Natural Language Engineering, 2021

The previous Emerging Trends article (Church et al., 2021. Natural Language Engineering 27(5), 631–645.) introduced deep nets to poets. Poets is an imperfect metaphor, intended as a gesture toward inclusion. The future for deep nets will …


Emotionally informed hate speech detection: a multi-target perspective

P Chiril, EW Pamungkas, F Benamara, V Moriceau… - Cognitive Computation, 2021

Hate Speech and harassment are widespread in online communication, due to users' freedom and anonymity and the lack of regulation provided by social me.


Empirical Best Practices On Using Product-Specific Schema. org

M Kejriwal, RK Selvam, CC Ni, N Torzec - Proceedings of the AAAI Conference on …, 2021

… Keywords: E-commerce, Schema.org, Best Practices, Web Data Commons, Common Crawl, Markup, Big Data Abstract. Schema.org has experienced high growth in recent years. Structured descriptions of products embedded …


Page 1. Empirical Regularization for Synthetic Sentence Pairs in Unsupervised Neural Machine Translation Xi Ai, Bin Fang Chongqing University barid.x.ai@gmail.com, fb@cqu.edu.cn Abstract UNMT tackles …


End-to-End Multimodal Clinical Depression Recognition using Deep Neural Networks: A comparative Analysis

M Muzammel, H Salam, A Othmani - Computer Methods and Programs in …, 2021

JavaScript is disabled on your browser. Please enable JavaScript to use all the features on this page. Skip to main content Skip to article …


Energy-saving Cross-layer Optimization of Big Data Transfer Based on Historical Log Analysis

L Rodolph, MD Nine, L Di Tacchio, T Kosar - arXiv preprint arXiv:2104.01192, 2021

… These include: (1) the small file size dataset consisting of 20,000 HTML files derived from the common crawl project [2]; (2) the medium file size dataset consisting of 5,000 image files derived from Flickr [4]; (3) the large …


Engineering Knowledge Graph from Patent Database

L Siddharth, L Blessing, KL Wood, J Luo - arXiv preprint arXiv:2106.06739, 2021

… pdf Page 25. embedding models that were trained on Google News, Wikipedia, gigaword, and Common Crawl as the original data sources. The entries in Table 2 that mention these models are the resultant semantic networks …


Engineering Practical Lempel-Ziv Tries

D Arroyuelo, R Cánovas, J Fischer, D Köppl, M Löbel…

Page 1. Engineering Practical Lempel-Ziv Tries ∗ Diego Arroyuelo1,2 Rodrigo Cánovas3 Johannes Fischer4 Dominik Köppl5 Marvin Löbel4 Gonzalo Navarro 1,6 Rajeev Raman7 1 Millennium Institute for Foundational Research on Data (IMFD), Chile …


Page 1. ENHANCED REPRESENTATIONS AND EFFICIENT ANALYSIS OF SYNTACTIC DEPENDENCIES WITHIN AND BEYOND TREE STRUCTURES A Dissertation Presented to the Faculty of the Graduate School of Cornell University …


Enhanced sentiment extraction architecture for social media content analysis using capsule networks

P Demotte, K Wijegunarathna, D Meedeniya, I Perera - Multimedia Tools and …, 2021

Recent research has produced efficient algorithms based on deep learning for text-based analytics. Such architectures could be readily applied to text-base.


Enhancing Context Through Contrast

K Ambilduke, A Shetye, D Bagade, R Bhagwatkar…

Neural machine translation benefits from semantically rich representations. Considerable progress in learning such representations has been achieved by language modelling and mutual information maximization objectives using …


Enhancing Cross-lingual Semantic Annotations using Deep Network Sentence Embeddings

… RoBERTa. XLM- R uses only MLM as training objective as RoBERTa (ie no TLM) but with multilingual corpus. It is trained on even larger dataset (the CommonCrawl corpus (Wenzek et al., 2019), 2.5 TB) and in 100 languages. XLM …


Enhancing Viewing Experience of Generated Visual Storylines for Promotional Videos

C Liu, H Yu, Z Shen, I Dixon, Y Yu, Z Gao, P Wang… - 2021 IEEE International …, 2021

… ResNet50. Both models are trained on the DeepFashion dataset [21] until convergence2. Word embeddings are obtained from the pre-trained model on the 840B Common Crawl dataset. λi is set to 0.5. α, β, αc and βc are set to 0.4 …


Ensembling of Distilled Models from Multi-task Teachers for Constrained Resource Language Pairs

A Hendy, EA Gad, M Abdelghaffar, JS ElMosalami… - arXiv preprint arXiv …, 2021

… all the available monolingual data, eg NewsCrawl + CommonCrawl + Extended CommonCrawl for Hausa, and Extended CommonCrawl for both Xhosa and Zulu. For relatively … In addition to the NewsCrawl monolingual subset, we add a …


Enterprise Data Integration: On Extracting Data from HTML Tables

… layer is used. The authors learnt the network using 3 567 tables from 200 web sites, and evaluated the results on 60 678 ta- bles from 300 web sites; the documents were selected from the April 2016 Common Crawl. They also …


ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generation

Y Sun, S Wang, S Feng, S Ding, C Pang, J Shang, J Liu… - arXiv preprint arXiv …, 2021

… However, these large-scale pre-trained language models with hundreds of billions of parameters are trained on plain texts. For example, the 175-billion-parameter GPT-3 is trained on a corpus with 570GB filtered texts from Common Crawl …


ERNIE-M: Enhanced Multilingual Representation by Aligning Cross-lingual Semantics with Monolingual Corpora

X Ouyang, S Wang, C Pang, Y Sun, H Tian, H Wu… - arXiv preprint arXiv …, 2020

… XLM-R (Conneau et al., 2019) studies the effects of models when trained on a large scale corpus. It used 2.5T data extracted from Common Crawl (Wenzek et al., 2019) that involves 100 languages for MMLM training. Results …


Error Analysis of using BART for Multi-Document Summarization: A Study for English and German Language

… For instance, single-document summarization of text in German language was done by Parida and Motlicek (2019) who utilized transformer models for abstractive summarization on two datasets — SwissText 20191 and …


Error identification for machine translation with metric embedding and attention

R Rubino, A Fujita, B Marie - Proceedings of the 2nd Workshop on Evaluation and …, 2021

Abstract Quality Estimation (QE) for Machine Translation has been shown to reach relatively high accuracy in predicting sentence-level scores, relying on pretrained contextual embeddings and human-produced quality scores. However, the lack of …


ESPnet-ST IWSLT 2021 Offline Speech Translation System

H Inaguma, B Yan, S Dalmia, P Gu, J Shi, K Duh… - arXiv preprint arXiv …, 2021

… Must-C - 0.68M Must-C v2 0.74M ST-TED (cleaned) 0.40M Europarl 1.82M Commoncrawl 2.39M Paracrawl 34.37M NewsCommentary 0.37M WikiTitles 1.38M RAPID 1.63M WikiMatrix 1.57M Table 1: Corpus statistics data was …


Est-ce que vous compute? Code-switching, cultural identity, and AI

A Falbo, T LaCroix - arXiv preprint arXiv:2112.08256, 2021

… of the Common Crawl dataset, resulting in approximately 14 billion tokens (though they do not provide details of how the Common Crawl … is predominantly English: around 45% of HTML pages in the Common Crawl dataset have English as their …


Establishing Trustworthiness Through Algorithmic Approaches to Qualitative Research

H Nguyen, J Ahn, A Belgrave, J Lee, L Cawelti, HE Kim… - International Conference on …, 2021

HE Kim, Y Prado, R Santagata, A Villavicencio - … 2020, Malibu, CA, USA, February 1-3 …, 2021

… The model contains 300 dimensional word vectors that were trained on a vocabulary of 2 million words from web page data (Common Crawl dataset; GloVe,[21]). We then worked to cluster words together using the word embedding developed with spaCy …


Estimating the Effects of Text Genre, Image Resolution and Algorithmic Complexity needed for Sinhala Optical Character Recognition

I Anuradha, C Liyanage, R Weerasinghe - International Journal on Advances in ICT …, 2021

… 2) 5million+ sentences in Sinhala common crawler: In 2019, Guzman [16] presented two monolingual corpora for Sinhala. Those were a combination of 155k+ sentences of filtered Sinhala Wikipedia and 5178k+ sentences of Sinhala common crawl …


Estimation of Imageability Ratings of English Words Using Neural Networks

VV Bochkarev, AV Savinkov, AV Shevlyakova - Mexican International Conference on …, 2021

… (a set of CommonCrawl vectors). The values of the correlation coefficients obtained in [6] are shown in the last row of the table. We chose the best values from those given in [6] that were obtained also using the fastText vectors trained on the …


Evaluating and Explaining Natural Language Generation with GenX

K Duskin, S Sharma, JY Yun, E Saldanha, D Arendt - … on Data Science with Human in …, 2021

… While we demonstrated utility on datasets with tens of thousands of text examples, the nearest neighbor approach used would become intractable on massive text corpora such as CommonCrawl 7. This limits GenX to …


Evaluating Contextualized Language Models for Hungarian

… It was trained on Webcorpus 2.0 (Nemeskey, 2020), 9- billion-token corpus compiled from the Hungarian subset of Common Crawl3. Its string identifier in Huggingface Transformers is … XLMRoBERTa was trained on 2TB of …


Evaluating Document Coherence Modelling

A Shen, M Mistica, B Salehi, H Li, T Baldwin, J Qi - arXiv preprint arXiv:2103.10133, 2021

Page 1. arXiv:2103.10133v1 [cs.CL] 18 Mar 2021 Evaluating Document Coherence Modelling Aili Shen ♣ , Meladel Mistica ♣ , Bahar Salehi ♣ , Hang Li ♢ , Timothy Baldwin ♣ , Jianzhong Qi ♣ ♣ The University of Melbourne …


Evaluating Off-the-Shelf Machine Listening and Natural Language Models for Automated Audio Captioning

B Weck, X Favory, K Drossos, X Serra - arXiv preprint arXiv:2110.07410, 2021

… We use the publicly available model trained with subword information on the Common Crawl corpus, which contains 600B tokens and is significantly larger than the corpora used for the Glove and word2vec model [38]. We employ BERT as our …


Evaluating Sequence-to-Sequence Modelling for Dialogue State Tracking

… 3.2.1 Word embeddings. We use GloVe [21] pretrained word embeddings, as done in [9]. We use the GloVe-840B-300D embeddings: 300-dimensional embeddings trained on 840B tokens from Common Crawl. 3.2.2 Models …


Evaluating the Evaluation Metrics for Style Transfer: A Case Study in Multilingual Formality Transfer

E Briakou, S Agrawal, J Tetreault, M Carpuat - arXiv preprint arXiv:2110.10668, 2021

While the field of style transfer (ST) has been growing rapidly, it has been hampered by a lack of standardized practices for automatic evaluation. In this paper, we evaluate leading ST automatic metrics on the oft-researched task of formality style …


Evaluating the Text-to-SQL Capabilities of Large Language Models

N Rajkumar, R Li, D Bahdanau - arXiv preprint arXiv:2204.00498, 2022

… Starting from public checkpoints pretrained 062 on Common Crawl, the T5 model is finetuned on 063 Spider to predict the output SQL, conditioned on 064 the question and schema. The 3B parameter T5 065 model is currently the state-of-the-art …


Evaluating the textbugger NLP Attack on

… 1 Introduction The ubiquity of deep learning models which learn from their user-base, like social recommenders or CAPTCHA[4], or other relatively unsupervised datasets, like the Common Crawl, in practice has given rise …


Evaluating Word Embeddings with Categorical Modularity

S Casacuberta, K Halevy, DE Blasi - arXiv preprint arXiv:2106.00877, 2021

… these embeddings. FastText. Monolingual embeddings for 157 languages trained on Common Crawl and Wikipedia that use CBOW with position-weights and character n-grams (Bojanowski et al., 2017). MUSE. Cross-lingual …


Evaluation and Interpretation of Word Embeddings

… 2.3.1 Facebook's Wikipedia and Common Crawl FastText Word Embeddings for 157 languages Produced by Grave et al. [8], the Facebook's Wikipedia and Common Crawl fastText word embeddings may be the most-used and …


Evaluation of contextual embeddings on less-resourced languages

M Ulčar, A Žagar, CS Armendariz, A Repar, S Pollak… - arXiv preprint arXiv …, 2021

… The ELMoForManyLangs project (EFML) [Che et al., 2018] trained ELMo models for several languages but used relatively small datasets of 20 million words randomly sampled from the raw text released by the CONLL 2017 …


Evaluation of Neural Network Transformer Models for Named-Entity Recognition on Low-Resourced Languages

… LANGUAGE MODEL ARCHITECTURE XLM-Roberta (XLM-R) is a transformer-based multilingual masked language model [2]. This language model trained on 100 languages uses 2.5 TB of CommonCrawl (CC) data …


Evaluation of Sentence Representations in Semantic Text Similarity Tasks

… 5. Transformer based encoder, using RoBERTa. For the baseline representation, word embeddings trained on common crawl data was used.[30] A few different approaches were used to construct the sentence vector representation …


Evaluation of Split-and-Rephrase Output of the Knowledge Extraction Tool in the Intelligent Tutoring System

A Grubišić, B Žitko, A Gašpar, D Vasić, A Dodaj - Expert Systems with Applications, 2021

JavaScript is disabled on your browser. Please enable JavaScript to use all the features on this page. Skip to main content Skip to article …


Everything Is All It Takes: A Multipronged Strategy for Zero-Shot Cross-Lingual Information Extraction

M Yarmohammadi, S Wu, M Marone, H Xu, S Ebner… - arXiv preprint arXiv …, 2021

… We focus on the coarsest “Abstract” level, where the goal is to identify events and their agents and patients. The documents come from the news-specific portion of Common Crawl … We use the same Common Crawl corpus as XLM-R for pretraining …


Evolution of diversity and dominance of companies in online activity

PX McCarthy, X Gong, S Eghbal, DS Falster, MA Rizoiu - Plos one, 2021

… We compile the linkage record over time of online websites towards domains using Common Crawl (https://commoncrawl.org/). Common Crawl is the largest open index of the web and which has been shown to represent over …


I Roman, R Santana, A Mendiburu, JA Lozano - Annals of Mathematics and Artificial …, 2021

Page 1. Annals of Mathematics and Artificial Intelligence https://doi.org/10.1007/s10472021-09751-5 Evolution of Gaussian Process kernels for machine translation post-editing effort estimation Ibai Roman1 ·Roberto Santana1 ·Alexander Mendiburu1 ·Jose A. Lozano1 …


Examination Committee.

… evaluates the model on a German Wiki data 9 using data extracted from Common Crawl 10 as synthetic summaries … it delivers a good value on aiding the overall performance assessment. 9https://www.swisstext.org/ 10http://commoncrawl.org/ Page 18 …


Page 1. Examining Media Bias in the 2020 Democratic Primary by Extraction of Entity-Relationships from News Articles Master Thesis submitted by Daniel Färber at University of Konstanz Faculty of Politics, Law and Economics …


Examining the Effects of Preprocessing on the Detection of Offensive Language in German Tweets

… dings. We ultimately settled on on the default 300 dimensional fastText German embeddings (Grave et al., 2018), trained on the German CommonCrawl and Wikpedia, as they yielded the most stable performance. We also …


Executive function & semantic memory impairments in Alzheimer's disease—investigating the decline of executive function and semantic memory in Alzheimer's …

Alzheimer’s Disease (AD) has a huge impact on an ever-aging society in highly developed industrialized countries such as the EU member states: according to the World Alzheimer’s Association the number one risk factor for AD is age. AD patients …


Experience and prediction: a metric of hardness for a novel litmus test

N Isaak, L Michael - Journal of Logic and Computation, 2021

Abstract. In the past decade, the Winograd schema challenge (WSC) has become a central aspect of the research community as a novel litmus test. Consequently, th.


Z Chen, J Liu, W Gu, Y Su, MR Lyu - arXiv preprint arXiv:2107.05908, 2021

Page 1. Experience Report: Deep Learning-based System Log Analysis for Anomaly Detection Zhuangbin Chen, Jinyang Liu, Wenwei Gu The Chinese University of Hong Kong Hong Kong, China {zbchen,jyliu,wwgu}@cse.cuhk.edu.hk …


Experimental Evaluation of Deep Learning models for Marathi Text Classification

A Kulkarni, M Mandhane, M Likhitkar, G Kshirsagar… - arXiv preprint arXiv …, 2021

… It was created by processing January-December 2018 Commoncrawl snapshots … OSCAR: Open Super-large Crawled AL- MAnaCH coRpus (OSCAR) is obtained by filtering and language classifying the Common Crawl corpus (Suárezetal., 2019) …


Experiments with adversarial attacks on text genres

M Lepekhin, S Sharoff - arXiv preprint arXiv:2107.02246, 2021

… It has the same architecture, but uses bigger and more genre diverse corpora and an updated pre-training procedure. In addition, XLM-RoBERTa is a multilingual model trained on Common Crawl data in comparison to multilingual BERT only trained on Wikipedia …


ExpertRank: A Multi-level Coarse-grained Expert-based Listwise Ranking Loss

Z Chen, C Eickhoff - arXiv preprint arXiv:2107.13752, 2021

… For KNRM, ConvKNRM and MatchPyramid, GloVe word embeddings with 300 dimensions [25], trained on the Common Crawl dataset2, are used for vocabulary initialization … 1https://www.tensor ow.org/ 2https://commoncrawl …


Explainable Prediction of Text Complexity: The Missing Preliminaries for Text Simplification

… all positions without data corruption. We use the 12 layer XLNeT base pre-trained model on the English Wikipedia, the Books corpus (similar to BERT), Giga5, ClueWeb 2012-B, and Common Crawl. 4.1.2 Evaluation Metric We …


Exploiting Linguistic Information from Nepali Transcripts for Early Detection of Alzheimer's Disease using Natural Language Processing and Machine Learning …

S Adhikari, S Thapa, U Naseem, P Singh, H Huo… - International Journal of …, 2021

Alzheimer's disease (AD) is considered as progressing brain disease, which can be slowed down with the early detection and proper treatment by identifying the early symptoms. Language change serves as an early sign that a patient's cognitive …


Page 1. Exploiting semantic similarity models to automate transfer credit assessment in academic mobility by Dhivya Chandrasekaran A thesis submitted in partial fulfillment of the requirements for the degree of Master of Science in Computer Science in the …


Exploring Conditional Text Generation for Aspect-Based Sentiment Analysis

SUS Chebolu, F Dernoncourt, N Lipka, T Solorio - arXiv preprint arXiv:2110.02334, 2021

… (2019). The encoder utilizes a denoising goal similar to BERT, while the decoder tries to recreate the original sequence (autoencoder) token by token using the preceding (uncorrupted) tokens and the encoder output. T5 uses common crawl web extracted text …


Exploring deep learning methods for recognizing rare diseases and their clinical manifestations from texts

I Segura-Bedmar, D Camino-Perdonas… - arXiv preprint arXiv …, 2021

… The model contains word embeddings of di- mension 300 for 3 million words. • GloVe [54], a pre-trained word embedding model trained using Common Crawl, an open repository of web crawl data. The model contains …


Exploring intrinsic information content models for addressing the issues of traditional semantic measures to evaluate verb similarity

MKS Prasad, P Sharma - Computer Speech & Language, 2021

JavaScript is disabled on your browser. Please enable JavaScript to use all the features on this page. Skip to main content Skip to article …


Exploring Monolingual Data for Neural Machine Translation with Knowledge Distillation

AF Aji, K Heafield - arXiv preprint arXiv:2012.15455, 2020

… Out- of-domain back-translation data (CommonCrawl– mixed, Open Subtitle–conversation) damages the performance, however … Though, CommonCrawl or Opensub1 conversational monolingual data is still better than not us- ing any monolingual data at all …


Exploring Pre-Trained Transformers and Bilingual Transfer Learning for Arabic Coreference Resolution

B Min - Proceedings of the Fourth Workshop on Computational …, 2021

Abstract In this paper, we develop bilingual transfer learning approaches to improve Arabic coreference resolution by leveraging additional English annotation via bilingual or multilingual pre-trained transformers. We show that bilingual transfer …


Exploring the Possibilities of Applying Transfer Learning Methods for Natural Language Processing in Software Development

Page 1. DEPARTMENT OF INFORMATICS TECHNISCHE UNIVERSITÄT MÜNCHEN Master's Thesis in Informatics: Data Engineering and Analytics Exploring the Possibilities of Applying Transfer Learning Methods for Natural …


Exploring the Predictive Power of News and Neural Machine Learning Models for Economic Forecasting

V Bitetta, I Bordino, A Ferretti, F Gullo, G Ponti… - Mining Data for Financial …

… rule-based procedure that builds on the linguistic features of the spaCy Python library 5 . The NLP pipeline relies on the en_core_web_lg model of spaCy 6 , an English multi-task Convolutional Neural Network trained on OntoNotes …


Exploring Transformers in Emotion Recognition: a comparison of BERT, DistillBERT, RoBERTa, XLNet and ELECTRA

D Cortiz - arXiv preprint arXiv:2104.02041, 2021

… tokens but in random order. XLNet was trained with over 130 GB of textual data. In addition to BERT's two datasets, they included three more corpora: Giga5, ClueWeb, and Common Crawl. XLNet outperformed BERT on 20 …


Exploring Unsupervised Pretraining Objectives for Machine Translation

C Baziotis, I Titov, A Birch, B Haddow - arXiv preprint arXiv:2106.05634, 2021

… (2019). The (pretraining) monolingual data contain 5M sentences from Common Crawl and Wikipedia per language, while the parallel data are approximately 600K sentences from the Bible, Open Subtitles, GNOME/KDE/Ubuntu, and Paracrawl …


Exploring Zero-shot Cross-lingual Aspect-based Sentiment Analysis using Pre-trained Multilingual Language Models

KTK Phan, DN Hao, D Van Thin, NLT Nguyen - 2021 International Conference on …, 2021

… The mBERT model is trained on the Wikipedia data of 104 languages, while the XLM-R model is trained on the CommonCrawl data in 100 … In detail, the XLM-R model is trained on the combination of Common Crawl with Wikipedia data, while …


Exploring Zero-Shot Emotion Recognition in Speech Using Semantic-Embedding Prototypes

X Xu, J Deng, N Cummins, Z Zhang, L Zhao… - IEEE Transactions on …, 2021

Page 1. 1520-9210 (c) 2021 IEEE. Personal use is permitted, but republication/ redistribution requires IEEE permission. See http://www.ieee.org/ publications_standards/publications/rights/index.html for more information. This …


Exposure of occupations to technologies of the fourth industrial revolution

B Meindl, MR Frank, J Mendonça - arXiv preprint arXiv:2110.13317, 2021

… The common crawl corpus comprises texts from various sources available on the web, and therefore the embeddings represent various aspects of language. However, the structure and words used in patent data are more technical than standard …


EXPRESS: Attribute Sentiment Scoring with Online Text Reviews: Accounting for Language Structure and Missing Attributes

I Chakraborty, M Kim, K Sudhir - Journal of Marketing Research, 2021

Page 1. Peer Review Version Attribute Sentiment Scoring with Online Text Reviews: Accounting for Language Structure and Missing Attributes Journal: Journal of Marketing Research Manuscript ID JMR.19.0192.R3 Manuscript …


Page 1. Extending Neural Temporal Tagging Systems with External Knowledge Martin Pömsl Dr. phil. Tobias Thelen Institute of Cognitive Science, Osnabrück M. Sc. Luzian Hahn Fraunhofer Institute for Integrated Circuits, Erlangen Osnabrück University …


Extracting ESG data from business documents

… Common Crawl (Want to use our data? – Common Crawl 2021) for example is an open repository of web crawl data consisting of web page data, metadata extracts and text extracts. It is obtained by scraping web pages and ignoring HTML markup …


Page 1. ABSTRACT GUO, HUI. Extraction and Analysis of Stories: Natural Language Processing for Software Engineering. (Under the direction of Dr. Munindar P. Singh.) Modern software systems are deployed in sociotechnical …


Extremely low-resource neural machine translation for Asian languages

R Rubino, B Marie, R Dabre, A Fujita, M Utiyama… - Machine Translation, 2021

This paper presents a set of effective approaches to handle extremely low-resource language pairs for self-attention based neural machine translation (NMT).


Facebook AI WMT21 News Translation Task Submission

C Tran, S Bhosale, J Cross, P Koehn, S Edunov, A Fan - arXiv preprint arXiv …, 2021

… For each language, we train an n-gram language model (Heafield, 2011) on all available news-domain data (Newscrawl) and a n-gram language model on a similarly sized sample from general-domain data (Commoncrawl) …


Fact-Checking of Claims from the English Wikipedia Using Evidence in the Wild

… architectures. These advancements enable taking advantage of vast amounts of unsupervised text data obtained using the internet, via sources such as Wikipedia and Common Crawl. Using these large corpora, researchers have demonstrated …


Factorising Meaning and Form for Intent-Preserving Paraphrasing

T Hosking, M Lapata - arXiv preprint arXiv:2105.15053, 2021

… English. Although this system was trained on high volumes of data (including Common Crawl), the training data contains relatively few questions, and we would not expect it to perform well in the domain under consideration …


Fast and Compact Set Intersection through Recursive Universe Partitioning

… CCNews is a dataset of news freely available from CommonCrawl; precisely, it consists of the news appeared from 09/01/16 to 30/03/18. Identifiers were assigned to documents according to the lexicographic order of their URLs …


J Kong, W Li, Z Liu, B Liao, J Qiu, CY Hsieh, Y Cai… - arXiv preprint arXiv …, 2021

Page 1. Fast Extraction of Word Embedding from Q-contexts Junsheng Kong ∗† School of Software Engineering, South China University of Technology China sescut_kongjunsheng@mail.scut.edu.cn Weizhao Li ∗† School …


FAST: A carefully sampled and cognitively motivated dataset for distributional semantic evaluation

… Embeddings: word2vec (Mikolov et al., 2013) trained on 100G tokens of Google News data,10 GloVe (Pennington et al., 2014) trained on 6G to- kens of wikipedia + newspapers or 42G tokens of Common Crawl (CC) Web …


Faster Re-translation Using Non-Autoregressive Model For Simultaneous Neural Machine Translation

H Han, S Indurthi, MA Zaidi, NK Lakumarapu, B Lee… - arXiv preprint arXiv …, 2020

… We choose the best system based on the MuST-C dev set and report the results on the MuST-C tst-COMMON test set. The WMT19 dataset further consists of Europarl v9, ParaCrawl v3, Common Crawl, News Commentary …


fastText-based methods for Emotion Identification in Russian Internet Discourse

A Babii, M Kazyulina, A Malafeev - 13th ACM Web Science Conference 2021, 2021

… 0.8. The only exception is the model based on the GRU architecture and pre-trained on the Common Crawl. That … 0.81. The lowest performance was shown by the CNN model pre-trained on the Common Crawl embeddings. We …


FEEL-IT: Emotion and Sentiment Classification for the Italian Language

F Bianchi, D Nozza, D Hovy - Proceedings of the Eleventh Workshop on …, 2021

… In this paper, we use the Italian BERT model UmBERTo trained on Commoncrawl ITA.4 As the first experimental condition, we fine-tune the UmBERTo model for the task of emotion classification with the considered training data (UmBERTo-FT) …


Felix Hamborg, Philipp Meschenmoser, Moritz Schubotz

P Scharpf, B Gipp - Wahrheit und Fake im postfaktisch-digitalen Zeitalter

… These filter criteria are passed to news-please, which subsequently searches for relevant news articles in the Common Crawl News … 2 News-please currently accesses the raw archive provided by the Common Crawl project. To speed up the …


Feminist Curation of Text for Data-centric AI

Abstract Language models are becoming increasingly central to artificial intelligence through their use in online search, recommendation engines and language generation technologies. However, concepts of gender can be deeply embedded in …


Few-shot Learning with Multilingual Language Models

XV Lin, T Mihaylov, M Artetxe, T Wang, S Chen… - arXiv preprint arXiv …, 2021

… Our models are trained on a static multilingual corpus extracted from CommonCrawl, with English text comprising 32.6% of the total number … As such we also note the potential difference in genres between CommonCrawl and the genres used in GPT-3 …


FH-SWF SG at GermEval 2021: Using Transformer-Based Language Models to Identify Toxic, Engaging, & Fact-Claiming Comments

C Gawron, S Schmidt - arXiv preprint arXiv:2109.02966, 2021

… Philipp Reissel and Philip May have published both a German ELECTRA model (Reissel and May, 2020) and a “German colossal, cleaned Common Crawl corpus” (GC4) (Reissel and May, 2021) with about 540 GB of German …


Findings of the WMT 2021 Biomedical Translation Shared Task: Summaries of Animal Experiments as New Test Set

L Yeganova, D Wiemann, M Neves, F Vezzani, A Siu… - Sixth Conference on …, 2021

In the sixth edition of the WMT Biomedical Task, we addressed a total of eight language pairs, namely English/German, English/French, English/Spanish, English/Portuguese, English/Chinese, English/Russian, English/Italian, and English/Basque. Further, our …


Fine and Coarse Granular Argument Classification before Clustering

L Dumani, T Wiesenfeldt, R Schenkel - Proceedings of the 30th ACM International …, 2021

… This dataset consists of eight topics each with 1,000 usergenerated sentences originating from the Common Crawl archive.A key difference to our dataset, besides presumably different appearing frames as well as topics being used as queries, is …


Fine-tuning Distributional Semantic Models for Closely-Related Languages

K Bhatia, D Aggarwal, A Vaidya - Proceedings of the Eighth Workshop on NLP for …, 2021

… We further compare our fine-tuned models with two pre-trained word embedding models, namely fastText (FT-WC), trained on Common Crawl and Wikipedia (Grave et al., 2018) and IndicFastText (I- FT) (Kakwani et al., 2020) …


FinMatcher at FinSim-2: Hypernym Detection in the Financial Services Domain using Knowledge Graphs

J Portisch, M Hladik, H Paulheim - arXiv preprint arXiv:2103.01576, 2021

… DBpedia [10] – but instead on the whole Web: The dataset consists of hypernymy relations extracted from the Common Crawl5, a … 3see https://www.wikidata.org/wiki/Q1881843 4see https://query.wikidata.org …


Focused Attention Improves Document-Grounded Generation

S Prabhumoye, K Hashimoto, Y Zhou, AW Black… - arXiv preprint arXiv …, 2021

… 4.1 Wikipedia Update Generation This task involves generating an update for Wikipedia context given a news article (Prabhumoye et al., 2019). The dataset was collected by parsing Wikipedia articles and Common Crawl for news articles …


FoodChem: A food-chemical relation extraction model

G Cenikj, BK Seljak, T Eftimov - arXiv preprint arXiv:2110.02019, 2021

… Apart from the datasets used for pre-training the original BERT model, 3 additional sources are involved in the pre-training of RoBERTa: the OpenWebText corpus [32], the Stories subset from the Common Crawl dataset [33] and the CommonCrawl News dataset [34] …


Four dimensions characterize attributions from faces using a representative set of English trait words

C Lin, U Keles, R Adolphs - Nature Communications, 2021

People readily (but often inaccurately) attribute traits to others based on faces. While the details of attributions depend on the language available to describe social traits, psychological theories argue that two or three dimensions (such …


From Argument Search to Argumentative Dialogue: A Topic-independent Approach to Argument Acquisition for Dialogue Systems

N Rach, C Schindler, I Feustel, J Daxenberger…

… query. The engine utilizes a web crawl from the year 2016 based on CommonCrawl 2 to retrieve relevant documents and subsequently classify sentences in the documents as either pro, con or no argument (Stab et al., 2018) …


From OpenAPI Fragments to API Pattern Primitives and Design Smells

S Serbout, C Pautasso, U Zdun, O Zimmermann - 2021

In the past few years, the OpenAPI Specification (OAS) has emerged as a standard description language for accurately modeling Web APIs. Today, thousands of OpenAPI descriptions can be found by mining open source repositories. In this paper …


gaBERT--an Irish Language Model

J Barry, J Wagner, L Cassidy, A Cowap, T Lynn… - arXiv preprint arXiv …, 2021

… texts, and official documents. OSCAR The unshuffled Irish portion of the OS- CAR Corpus (Ortiz Suárez et al., 2019) which is a collection of documents from CommonCrawl and sorted by language ID. ParaCrawl The Irish portion …


Gaze-based Multimodal Meaning Recovery for Noisy/Complex Environments

Ö Alaçam, G Malhotra, E Ruppert, C Biemann - Proceedings of the 2021 International …, 2021

… In addition, we also utilized xlm-roberta-base masked language model, which is a large multi-lingual language model, trained on 2.5 TB of filtered CommonCrawl data [8]. Unlike multilingual BERT and RoBERTa, XLM-RoBERTa can infer the language of …


Page 1. Gender Bias in English and German Children's Literature: A Computational Analysis Using Word Embeddings MASTER THESIS Dominique Geißler Master Interaction Technology Faculty of Electrical Engineering, Mathematics …


Gender Stereotypes in Natural Language: Word Embeddings Show Robust Consistency Across Child and Adult Language Corpora of More Than 65 Million Words

TES Charlesworth, V Yang, TC Mann, B Kurdi… - Psychological Science, 2020

… In this case, lemmatization is particularly helpful because the target corpora are smaller than other natural-language corpora, such as the Common Crawl corpus (which we use for validation as described below) with more than 600 billion word tokens (Mikolov, Grave …


Generalization of cyberbullying traces

Page 1. © Marc-André Larochelle, 2020 Generalization of Cyberbullying Traces Mémoire Marc-André Larochelle Maîtrise en informatique - avec mémoire Maître ès sciences (M. Sc.) Québec, Canada Page 2. Generalization of Cyberbullying Traces Mémoire …


Generalized zero-shot object recognition withoutclass-attribute relations

Page 1. GENERALIZED ZERO-SHOT OBJECT RECOGNITION WITHOUT CLASS-ATTRIBUTE RELATIONS A THESIS SUBMITTED TO THE GRADUATE SCHOOL OF NATURAL AND APPLIED SCIENCES OF MIDDLE EAST …


Generating Sense Inventories for Ambiguous Arabic Words

… 4.3. Fasttext Pre-Trained Embeddings Arabic Fasttext embeddings are provided by Grave et al. [12]. These embeddings result from training on Wikipedia and Common Crawl corpus. They use an extension of the Fasttext model with subword information …


Generative Adversarial Networks based on Mixed-Attentions for Citation Intent Classification in Scientific Publications

YS Wang, CY Chen, LH Lee - Proceedings of the 33rd Conference on Computational …, 2021

Abstract We propose the mixed-attention-based Generative Adversarial Network (named maGAN), and apply it for citation intent classification in scientific publication. We select domain-specific training data, propose a mixed-attention mechanism, and …


Generative Chatbot Framework for Cybergrooming Prevention

Cybergrooming refers to the crime of establishing personal close relationships with potential victims, commonly teens, for the purpose of sexual exploitation or abuse via online social media platforms. Cybergrooming has been recognized as a serious …


Generative Question Answering in a Low-Resource Setting

… While the monolingual T5 was pre-trained on the C4 data set which consists of 750GB of English text from Common Crawl, mT5 has been pre-trained on the data likewise collected from Common Crawl but in over 100 languages …


Generic Key Value Extractions from Emails

R Gupta - International Conference on Big Data Analytics, 2020

… sentences. Authors of [14] describe a machine learning-based model trained on common-crawl data with multi-task optimizations to represent each keyword phrase using a 500-dimension vector of real numbers. Cosine similarity …


GenSumm: A Joint Framework for Multi-task Tweet Classification and Summarization using Sentiment Analysis and Generative Modelling

D Bansal, R Grover, N Saini, S Saha - IEEE Transactions on Affective Computing, 2021

Social media platforms like Twitter act as a medium for communication among people, government agencies, NGOs, and other relief providing agencies in widespread humanitarian havoc during a disaster outbreak when other …


Geoscience Language Processing for Exploration

H Denli, HA Chughtai, B Hughes, R Gistri, P Xu - Abu Dhabi International Petroleum …, 2021

Deep learning has recently been providing step-change capabilities, particularly using transformer models, for natural language processing applications such as question answering, query-based summarization, and language translation for …


GeoVectors: A Linked Open Corpus of OpenStreetMap Embeddings on World Scale

N Tempelmeier, S Gottschalk, E Demidova - arXiv preprint arXiv:2108.13092, 2021

… them. Training: Pre-trained word vectors are available at the fastText website6. As most of the OSM keys are in English, we chose the 300dimensional English word vectors trained on the Common Crawl, and Wikipedia [10] …


German Speech Recognition System using DeepSpeech

J Xu, K Matta, S Islam, A Nürnberger - … of the 4th International Conference on Natural …, 2020

… In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 4774–4778. [5] Common crawl 2020. Language model. http://commoncrawl org/. [6] DeepSpeech Deutsch 2020. German dataset of Mozilla Common voice …


Page 1. GFST: Gender-Filtered Self-Training for More Accurate Gender in Translation Prafulla Kumar Choubey∗† Texas A&M University prafulla.choubey@tamu.edu Anna Currey∗, Prashant Mathur, Georgiana Dinu Amazon …


GitTables: A Large-Scale Corpus of Relational Tables

M Hulsebos, Ç Demiralp, P Groth - 2021
MHS Computing, ÇDS Computing

… We use the character-level n-gram FastText model pretrained on the Common Crawl corpus. We experimented with contextual models like BERT [13] and the Universal Sentence Encoder [8] , but found that these models yield worse or at best …


Going Full-TILT Boogie on Document Understanding with Text-Image-Layout Transformer

R Powalski, Ł Borchmann, D Jurkiewicz, T Dwojak… - arXiv preprint arXiv …, 2021

… Library ∗ , and PDF files from Common Crawl. Then, a T5-like masked language model pretraining objective is used, but in a salient span masking scheme, ie, named entities are preferred rather than random tokens [47,15] …


Golos: Russian Dataset for Speech Research

N Karpov, A Denisenko, F Minkin - arXiv preprint arXiv:2106.10161, 2021

… language. Additionally, we built a 3-gram language model on the open Common Crawl dataset and merged it with the train set transcriptions. Using … 8. 7https:// commoncrawl.org 8https://github.com/sberdevices/golos Page 5. 8 …


Governments and the Net: Defense, Control, and Trust in the Fifth Domain

… The by far largest data source is the Common Crawl archive. Every four weeks, Common Crawl generates a new dataset that contains content and metadata of billions of webpages based on following the links on a large sample of websites …


GPT Perdetry Test: Generating new meanings for new words

N Malkin, S Lanka, P Goel, S Rao, N Jojic - Proceedings of the 2021 Conference of …, 2021

… Most of the words were known by none of them. 36 of the 74 words do not appear in the top 2 million words of the Common Crawl corpus, according to the GloVe embedding matrix (Pennington et al., 2014), and the median rank of the other words is 656565 …


GPT3-to-plan: Extracting plans from text using GPT-3

A Olmo, S Sreedharan, S Kambhampati - arXiv preprint arXiv:2106.07131, 2021

… 2020) is the latest version of the GPT models developed by OpenAI1. A 175 billion parameter autoregressive language model with 96 layers trained on a 560GB+ web corpora (Common Crawl2 and WebText2 (Gokaslan …


Grammar Index by Induced Suffix Sorting

H Bannai, M Takeda - String Processing and Information Retrieval: 28th …

T Akagi, D Köppl, Y Nakashima, S Inenaga, H Bannai… - arXiv preprint arXiv …, 2021

… Sizes are in megabytes [MB]. Dataset commoncrawl dna einstein. de Input size GCIS-nep GCIS-uni ESP-index FM-index r-index 221. 180 220. 119 138 … com/nicolaprezza/r-index, respectively. To save space, we renamed …


Graph-based KB and Text Fusion Interaction Network for Open Domain Question Answering

Y Ding, Y Rao, F Yang - 2021 International Joint Conference on Neural …, 2021

… C. Implementation Details Through our experiments, the 300-dimension GloVe embedding trained on the Common Crawl corpus is used to initialize the word embedding. The hidden dimension of LSTM and the dimension of entity embedding are both 100 …


Grounding Linguistic Commands to Navigable Regions

N Rufus, K Jain, UKR Nair, V Gandhi, KM Krishna - 2021

… Input images are resized to 448 × 448 spatial resolution. We use 300d GloVe embeddings pre-trained on Common Crawl 840B tokens [22]. The maximum length of commands is set to T = 40 and for both visual and linguistic features, channel dimension C = 512 …


H2oloo at TREC 2020: When all you got is a hammer... Deep Learning, Health Misinformation, and Precision Medicine

R Pradeep, X Ma, X Zhang, H Cui, R Xu, R Nogueira… - Corpus

… ranking. We adopt all the default settings. We index all the news articles found in the CommonCrawl News crawl from January 1st, 2020 to April 30th, 2020. At inference time, we retrieve the top-k0 (=1000) documents per query …


HAHA at EmoEvalEs 2021: Sentiment Analysis in Spanish Tweets with Cross-lingual Model

K Li - Proceedings of the Iberian Languages Evaluation …, 2021

… It is a large multi-lingual language model, trained on 2.5TB of filtered Common Crawl data … Because the XLM-RoBERTa model increases the number of languages and the number of training data sets, it uses more …


Harmonized System Code Classification Using Transfer Learning with Pre-Trained Weights

Page 1. HARMONIZED SYSTEM CODE CLASSIFICATION USING TRANSFER LEARNING WITH PRE-TRAINED WEIGHTS by Koustav Pain (Tukai) Submitted in partial fulfillment of the requirements for the degree of Master of Computer Science at …


Hate Speech Detection based on Sentiment Knowledge Sharing

X Zhou, Y Yang, X Fan, G Ren, Y Song, Y Diao, L Yang…

… To compare with previous works, We report results of DV using the standard Accuracy and weighted F1. In our experiments, for the input layer, all word vectors are initialized by Glove Common Crawl Embeddings (840B Token), and the dimension is 300 …


HDRS: Hindi Dialogue Restaurant Search Corpus for Dialogue State Tracking in Task-Oriented Environment

S Malviya, R Mishra, SK Barnwal, US Tiwary - IEEE/ACM Transactions on Audio …, 2021

Page 1. 2329-9290 (c) 2021 IEEE. Personal use is permitted, but republication/ redistribution requires IEEE permission. See http://www.ieee.org/ publications_standards/publications/rights/index.html for more information. This …


HerBERT: Efficiently Pretrained Transformer-based Language Model for Polish

R Mroczkowski, P Rybak, A Wróblewska, I Gawlik - … of the 8th Workshop on Balto …, 2021

… CCNet (Wenzek et al., 2020) is a clean monolingual corpus extracted from Common Crawl6 dataset of crawled websites … The first one did not use pretrained embeddings while the latter utilized fastText (Bo- janowski …


H Gao, M Hu, R Cheng, T Gao - arXiv preprint arXiv:2102.00677, 2021

Page 1. Hierarchical Ranking for Answer Selection Hang Gao, Mengting Hu, Renhong Cheng, Tiegang Gao Nankai University knimet@mail.nankai. edu.cn Abstract Answer selection is a task to choose the positive answers …


HIGH PERFORMANCE DOCUMENT STORE IMPLEMENTATION IN RUST

Databases are a core part of any application which requires persistence of data. The performance of applications involving the use of database systems is directly proportional to how fast their database read-write operations are. The aim of this …


HISTOGRAM OF TASK UNCERTAINTY

… 2014) word embedding (300d version on 840B Common Crawl data), and Xavier initialization for the parameters. The mini-batch size is set to 16, including samples of the same task. Other than these, we follow the training procedure for (Søgaard …


HmBlogs: A big general Persian corpus

HM Khansari, M Shamsfard - arXiv preprint arXiv:2111.02362, 2021

… The main part of PRT is obtained from the Persian section of the Common Crawl project (Common Crawl, nd). Common Crawl is a web crawl project that crawls and collects resources available on the web in any language, including some in Persian …


HONEST: Measuring hurtful sentence completion in language models

D Nozza, F Bianchi, D Hovy - Proceedings of the 2021 Conference of the North …, 2021

Page 1. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2398–2406 June 6–11, 2021. ©2021 Association for Computational Linguistics 2398 …


A Hande, R Priyadharshini, A Sampath, KP Thamburaj… - arXiv preprint arXiv …, 2021

Page 1. Advances in Computational Intelligence manuscript No. (will be inserted by the editor) Hope Speech detection in under-resourced Kannada language Adeep Hande · Ruba Priyadharshini · Anbukkarasi Sampath · Kingston …


How do blind people know that blue is cold? Distributional semantics encode color-adjective associations

J van Paridon, Q Liu, G Lupyan - PsyArXiv. February, 2021

… trained on. Using the projection method, we computed color associations in embeddings trained on four different text corpora: The Common Crawl and Wikipedia (Grave, Bojanowski, Gupta, Joulin, & Page 2. Mikolov, 2018 …


How Do Simple Transformations of Text and Image Features Impact Cosine-Based Semantic Match?

… 1. Word-level features: – GloVe6 [40]: We use 300-d vectors pre-trained on the Common Crawl corpus with 840B tokens and a 2.2M-word vocabulary. – word2vec (w2v) [37]: We use the skip-gram 300-d embeddings trained on Wikipedia …


How far can we get with one GPU in 100 hours? CoAStaL at MultiIndicMT Shared Task

R Aralikatte, HRM Bello, M de Lhoneux, D Hershcovich…

… translation task. It is pre-trained on publicly available corpora from Wikipedia and Common Crawl. It … tion task. This model is pre-trained on mC4, a multilingual version of the Common Crawl consisting of text from 101 languages. It …


How good BERT based models are in sentiment analysis of Croatian tweets: comparison of four multilingual BERTs

… It is pre-trained on texts on 100 different languages and more than two terabytes of data filtered from CommonCrawl data. As stated by the authors “XLM-RoBERTa (XLM-R) outperforms mBERT on cross-lingual classification by up to 23% accuracy on low-resource …


How Great is the Great Firewall? Measuring China's DNS Censorship

NP Hoang, AA Niaki, J Dalek, J Knockel, P Lin… - arXiv preprint arXiv …, 2021

… nsarchive.gwu.edu, cs.colorado.edu) are censored but their SLDs (eg, mit.edu, gwu.edu, colorado.edu) are not. We complement our test list by including domains from the Citizen Lab test lists (CLTL) [13], the Tranco list [66], and the Common Crawl project [14] …


How much is a cow like a meow? A novel database of human judgements of

K Wegner-Clemens, GL Malcolm, S Shomstein - Psychological Science

… This model was trained on a total of 650 billion words including Wikipedia from June 2017, two news corpuses (statmt.org news, UMBC news), and corpuses derived from a wide range of websites (Gigagword, Common Crawl). The words …


How to Adapt Your Pretrained Multilingual Model to 1600 Languages

A Ebrahimi, K Kann - arXiv preprint arXiv:2106.02124, 2021

… 2018) who present embeddings for 157 languages trained on additional data from Common Crawl … the larger vocabulary of 250k subwords created using SentencePiece tokenization (Kudo and Richardson, 2018) and the …


How Viable is Password Cracking in Digital Forensic Investigation? Analyzing the Guessability of over 3.9 Billion Real-World Accounts

A Kanta, S Coray, I Coisel, M Scanlon - Forensic Science International: Digital …, 2021

… GloVe [34] was used to automate this process with its Common Crawl 42B 300d, a pre-trained model in English for GloVe15 … This is because they do not have a representation in the Common Crawl and therefore cannot be compared to the categories …


C Koenders, J Filla, N Schneider, V Woloszyn - arXiv preprint arXiv:2107.07970, 2021

Page 1. How Vulnerable Are Automatic Fake News Detection Methods to Adversarial Attacks? Camille Koenders Technische Universität Berlin camille.koenders@gmail.com Johannes Filla Technische Universität Berlin johannesfilla@gmx.de …


HowSumm: A Multi-Document Summarization Dataset Derived from WikiHow Articles

O Boni, G Feigenblat, G Lev, M Shmueli-Scheuer… - arXiv preprint arXiv …, 2021

… a corpus. They es- tablish this approach by extracting answers that are Wikipedia paragraphs from Google Natural Questions (Kwiatkowski et al., 2019) and documents from Common Crawl (Raffel et al., 2020) corpus. In some …


HTLM: Hyper-Text Pre-Training and Prompting of Language Models

A Aghajanyan, D Okhonko, M Lewis, M Joshi, H Xu… - arXiv preprint arXiv …, 2021

… Hyper-text, such as the HTML found in the Common Crawl1, has a number of advantages for pretraining over plain text … Our HyperText Language Model (HTLM) is trained on 23TB of simplified HTML which we automatically extract …


HULTIG-C: NLP Corpus and Services in the Cloud

… [11] presented a Multilingual Web Corpus in over 50 languages with over 10 billion tokens licensed under the Creative Commons license. The texts that made up the corpus were extracted from Common Crawl, with around 2 billion crawled URLs …


F Alam, U Qazi, M Imran, F Ofli - arXiv preprint arXiv:2104.03090, 2021

Page 1. HumAID: Human-Annotated Disaster Incidents Data from Twitter with Deep Learning Benchmarks Firoj Alam, Umair Qazi, Muhammad Imran, Ferda Ofli Qatar Computing Research Institute, HBKU, Qatar {fialam,uqazi,mimran,fofli}@hbku.edu.qa …


Human and Transformer-Based Prosodic Phrasing in Two Speech Genres

J Volín, M Řezáčková, J Matouřek - International Conference on Speech and …, 2021

… We worked with the pre-trained Text-to-Text TransferTransformer (T5) model [18] and used the ability of weights pre-training from a huge amount of unlabeled text in the language [4, 18] – for this task we used Czech …


Human-Centered Financial Summarization

Page 1. ARISTOTLE UNIVERSITY OF THESSALONIKI MASTER'S THESIS Human-Centered Financial Summarization Author: Tatiana PASSALI Supervisor: Dr. Grigorios TSOUMAKAS A thesis submitted in fulfillment of …


Hybrid approach to detecting symptoms of depression in social media entries

A Wołk, K Chlasta, P Holas - arXiv preprint arXiv:2106.10485, 2021

… 2004). To conduct the experiments, we trained our model on the data from the Common Crawl (Bevendorff et al. 2018) project. As the CommonCrawl corpus is very polluted, an algorithmic division into sentences, data de-duplication, and cleaning was necessary …


I Wish I Would Have Loved This One, But I Didn't--A Multilingual Dataset for Counterfactual Detection in Product Reviews

J O'Neill, P Rozenshtein, R Kiryo, M Kubota… - arXiv preprint arXiv …, 2021

Page 1. arXiv:2104.06893v1 [cs.CL] 14 Apr 2021 I Wish I Would Have Loved This One, But I Didn't – A Multilingual Dataset for Counterfactual Detection in Product Reviews James O'Neill‡,∗∗Polina Rozenshtein†,∗ Ryuichi …


IBM MNLP IE at CASE 2021 task 1: Multi-granular and multilingual event detection on protest news

P Awasthy, J Ni, K Barker, R Florian - Proceedings of the 4th Workshop on …, 2021

… Hence we decided to use XLM-R as the text encoder. We use HuggingFace's pytorch im- plementation of transformers (Wolf et al., 2019). XLM-R was pre-trained with unlabeled Wikipedia text and the CommonCrawl Corpus of 100 languages …


iCompass at NLP4IF-2021–Fighting the COVID-19 Infodemic

W Henia, O Rjab, H Haddad, C Fourati - Proceedings of the Fourth Workshop on NLP …, 2021

… Page 3. 117 3.4 Arabic ALBERT Arabic ALBERT2 by (KUIS-AI-Lab) models were pretrained on 4.4 Billion words: Arabic version of OSCAR (unshuffled version of the corpus) filtered from Common Crawl and Recent dump of Arabic Wikipedia …


Identification of Biased Terms in News Articles by Comparison of Outlet-specific Word Embeddings

… We scraped articles from both news outlets, published in the last decade, from 2010 to 2020, from Common Crawl [4]. For preprocessing, we use Genism simple preprocessing and generate n-grams. 3.4 Linear mapping between vector spaces …


J Risch, A Stoll, L Wilms, M Wiegand - 2021

Page 1. Proceedings of the GermEval 2021 Workshop on the Identification of Toxic, Engaging, and Fact-Claiming Comments 17th Conference on Natural Language Processing KONVENS 2021 Heinrich Heine University Düsseldorf …


Identifying communicative functions in discourse with content types

T Caselli, R Sprugnoli, G Moretti - Language Resources and Evaluation, 2021

Texts are not monolithic entities but rather coherent collections of micro illocutionary acts which help to convey a unitary message of content and purpose.


Idiomatic Expression Identification using Semantic Compatibility

Z Zeng, S Bhat - arXiv preprint arXiv:2110.10064, 2021

Idiomatic expressions are an integral part of natural language and constantly being added to a language. Owing to their non-compositionality and their ability to take on a figurative or literal meaning depending on the sentential context, they have been a …


IIITT@ DravidianLangTech-EACL2021: Transfer Learning for Offensive Language Detection in Dravidian Languages

K Yasaswini, K Puranik, A Hande, R Priyadharshini… - Proceedings of the First …, 2021

… 4.3 XLM-RoBERTa XLM-RoBERTa (Conneau et al., 2020) is a large multi-lingual language model, trained on 2.5TB of cleaned CommonCrawl data in 100 languages. It can be recognized as a union of XLM (Lample …


IIITT@ LT-EDI-EACL2021-Hope Speech Detection: There is always hope in Transformers

K Puranik, A Hande, R Priyadharshini, S Thavareesan… - arXiv preprint arXiv …, 2021

… It's trained on over 2TB of filtered CommonCrawl data in 100 different languages. It was an update to the XLM-100 model (Lample and Conneau, 2019) but with in- creased training data … Wikipedia, Common Crawl7, PMINDIA8 and Dakshina9 datasets …


Image Emotion Recognition Using Visual and Semantic Features Reflecting Emotional and Similar Objects

T YAMAMOTO, S TAKEUCHI, A NAKAZAWA - IEICE TRANSACTIONS on Information …, 2021

… GloVe has been used for sentiment analysis [40]; therefore, we use GloVe as the method for obtaining the word vector. It has a 300-dimensional word embedding weight pre-trained by Common Crawl (840 billion tokens), which is publicly available [41] …


Impacts of Homophone Normalization on Semantic Models for Amharic

TD Belay, E Kombolcha, AA Ayele, G Gelaye…

… embedding representation. The only available pretrained static word embedding for Amharic text is the fastText model, which is trained from Wikipedia and web data scraped in the common crawl project [19]. This embedding …


Implicit social cognition: A brief (and gentle) introduction

B Kurdi, MR Banaji - PsyArXiv. January, 2021

… individual-level implicit attitudes toward ten social categories indexed using the IAT are closely aligned with societal-level semantic associations derived from the vast Common Crawl corpus via so- called word embeddings …


Implicitly Abusive Comparisons–A New Dataset and Linguistic Analysis

M Wiegand, M Geulig, J Ruppenhofer - Proceedings of the 16th Conference of the …, 2021

… A comparison was represented by the embedding vector of the word in that comparison whose similarity was highest to the centroid. As embeddings we chose the fastText emeddings (Joulin et al., 2017) induced on Common Crawl.8 Emotions (EMO) …


Improved Automated Classification of Sentences in Data Science Exercises

AM Angelone, A Galassi, P Vittorini - … in Methodologies and intelligent Systems for …, 2021

… In our setting, fastText embeddings are vectors of 300 dimensions, BERT embeddings are vectors of 768 dimensions. As for fastText, we adopted the precomputed Italian language model 1 , trained on Common Crawl and Wikipedia …


Improved Word Representations Via Summed Target and Context Embeddings

… The word2vec and word2vec-PLUS scores here were trained on the corpus of scraped ar- ticles since it resembles the Common Crawl data used to train GloVe [9] more closely than the other, more specialized corpora we explored …


Improving Hierarchical Product Classification using Domain-specific Language Modelling

… Afterwards, we investigate whether it is possible to improve the performance of the transformer models by performing additional self-supervised pre-training using different corpora of product offers, which were extracted from the Common Crawl …


Improving Large-scale Language Models and Resources for Filipino

JCB Cruz, C Cheng - arXiv preprint arXiv:2111.06053, 2021

… 2019) is a massive dataset obtained from language identification and filtering of the Common Crawl dataset. We use the deduplicated version of the Filipino (Tagalog) portion of OSCAR and add it to our pretraining corpus. …


Improving Multilingual Models for the Swedish Language: Exploring CrossLingual Transferability and Stereotypical Biases

… To acquire the natural text needed for pretraining purposes, the authors used common crawl [21]. Common crawl is a non commercial … To justify the choice of Common Crawl the authors say that it is a publicly available repository, allowing any …


Improving question answering for event-focused questions in temporal collections of news articles

J Wang, A Jatowt, M Färber, M Yoshikawa - Information Retrieval Journal

Temporal collections of news articles (or news archives) contain numerous accurate and time-aligned articles, which offer immense value to our society, hel.


Page 1. Improving Sentiment Classification for Arabic Short Text Using Deep Learning Approaches Ali Alwehaibi North Carolina A&T State University A dissertation submitted to the graduate faculty in partial fulfillment of the requirements for the degree of …


Improving the Lexical Ability of Pretrained Language Models for Unsupervised Neural Machine Translation

A Chronopoulou, D Stojanovski, A Fraser - arXiv preprint arXiv:2103.10531, 2021

… We use 3K randomly sampled sentences of SETIMES (Tiedemann, 2012) as validation/test sets. We also use 68M En sentences from NewsCrawl. For Sq and Mk we use all the CommonCrawl corpora from Ortiz Suárez et …


in the Language Technology (LT) group

… The model is trained on the colossal and cleaned version of common crawl (C4) as well as HugeNews which consists of 1.5B news articles from the web. Unsurprisingly, this method achieves state-of-the-art results on many summarization datasets …


indicnlp@ kgp at DravidianLangTech-EACL2021: Offensive Language Identification in Dravidian Languages

K Kedia, A Nandy - arXiv preprint arXiv:2102.07150, 2021

… use the bert-base-multilingual-cased model, mBERT trained on cased text in 104 languages from large Wikipedia articles, and the xlm-roberta-base model, XLM-R (Conneau et al., 2020) trained on 100 languages, using …


IndoNLG: Benchmark and Resources for Evaluating Indonesian Natural Language Generation

S Cahyawijaya, GI Winata, B Wilie, K Vincentio, X Li… - arXiv preprint arXiv …, 2021


Information Extraction from Semi-Structured Websites

… Note the log scale of the x-axis. . . . . 42 3.4 Extraction precision vs. number of extractions on the CommonCrawl dataset at various confidence thresholds; the orange bar indicates a 0.75 threshold yielding 1.25millionextractionsat90%precision. . . . . 47 …


Information Retrieval Model for Social Media Applications

… 2019 IEEE International Conference on Healthcare Informatics (ICHI), pp. 1–3, 2019. [17] S. Nagel, “Cc-news.” https://commoncrawl.org/2016/10/news-datasetavailable/, 2016. [18] M. Heidari and JH Jones, “Using bert to extract …


Information retrieval versus deep learning approaches for generating traceability links in bilingual projects

J Lin, Y Liu, J Cleland-Huang - Empirical Software Engineering, 2022

Abstract Software traceability links are established between diverse artifacts of the software development process in order to support tasks such as compliance analysis, safety assurance, and requirements validation. However, practice has shown that it …


INFOTEC-LaBD at PAN@ CLEF21: Profiling Hate Speech Spreaders on Twitter through Emotion-based Representations

H Cabrera, S Miranda-Jiménez, ES Tellez - CLEF, 2021

… 26] for mapping Tweets to semantic space. In particular, we use the model pre-trained on 600 billion tokens on Common Crawl for the English task and the one from [27] to tackle the Spanish task. We use the NRC Lexicon [12 …


MA Zaidi, S Indurthi, B Lee, NK Lakumarapu, S Kim - arXiv preprint arXiv:2109.03121, 2021

Page 1. Infusing Future Information into Monotonic Attention Through Language Models Mohd Abbas Zaidi∗, Sathish Indurthi∗†, Beomseok Lee Nikhil Kumar Lakumarapu, Sangha Kim NLP Lab, Samsung Research, Seoul, South …


Input Augmentation Improves Constrained Beam Search for Neural Machine Translation: NTT at WAT 2021

K Chousa, M Morishita - arXiv preprint arXiv:2106.05450, 2021

… We also used CommonCrawl provided by the WMT 2020 news shared task (Barrault et al., 2020) as additional monolingual data. For CommonCrawl data, we chose the ten million English and Japanese sentences that are similar …


Integrated Training for Sequence-to-Sequence Models Using Non-Autoregressive Transformer

E Tokarchuk, J Rosendahl, W Wang, P Petrushkov…

… Figure 2: Different variants of the encoder-decoder model integration through the connection interface. 5.1 Data Training data for French→German includes Eu- roparl corpus version 7 (Koehn, 2005), CommonCrawl2 corpus and the newstest2008-2010 …


Intelligent Arabic-Based Healthcare Assistant

T Wael, A Hesham, M Youssef, O Adel, H Hesham… - 2021 3rd Novel Intelligent …, 2021

… It was pre-trained on ∼8.2 billion words from the Arabic version of OSCAR (unshuffled version of the corpus) - filtered from Common Crawl [6], the recent dump of Arabic Wikipedia, and other Arabic resources, which sum up to ∼95GB of text. The corpus …


Intelligent Document Validation

Abstract Processes in organizations over the past few years have been increasingly automated. In order to make them more efficient and practical. However, one area in which manual work is still common is document analysis. In this area, due to the …


Intelligent Methods for Accurately Detecting Phishing Websites

… A. Data set The data set used in this paper is offered by [19] is composed of 5000 phishing websites and 5000 legitimate websites. Phishing websites are collected from PhishTank and OpenPhish, while legitimate websites are collected from Alexa and Common Crawl …


Interactive Clustering of Cooking Recipe Instructions: Towards the Automatic Detection of Events Involving Kitchen Devices

F Ventirozos, M Jacobo-Romero, S Clinch… - 2021 IEEE 15th …, 2021

… Based on these synonym sets, we enriched our list by adding similar words based on GloVe word embedding vectors [17], which were pre-trained on the Common Crawl corpus1 … 1https://commoncrawl.org/ 342 Page 3. Figure …


Interactive query expansion for professional search applications

T Russell-Rose, P Gooch, U Kruschwitz - arXiv preprint arXiv:2106.13528, 2021

… The DBpedia data set describes 4.58 million entities, out of which 4.22 million are classified in a consistent ontology. 2. WebISA (Seitner et al., 2016) is a publicly available database containing hypernymy relations extracted from the CommonCrawl web corpus …


Internet-Augmented Dialogue Generation

M Komeili, K Shuster, J Weston - arXiv preprint arXiv:2107.07566, 2021

… First, we store and utilize the Common Crawl dump of the internet from Wenzek et al … DPR+FAISS-based models We trained DPR+FAISS-based models using either the WoW or WizInt training datasets, and using either …


Interpretability in Word Sense Disambiguation using Tsetlin Machine

RK Yadav, L Jiao, OC Granmo, M Goodwin - 2021

… FastText-Base (FTX- B), FastText-CommonCrawl (FTX-C), 1 neural network (NN) BERT base, and our proposed TM. FTX- B is a fast text linear classifier without pre-trained embeddings and FTX-C is a fast text linear classifier with …


Interpretable bias mitigation for textual data: Reducing gender bias in patient notes while maintaining classification performance

JR Minot, N Cheney, M Maier, DC Elbers, CM Danforth… - arXiv preprint arXiv …, 2021

… and resulting word embeddings. For instance, general purpose language models are often trained on Wikipedia and the Common Crawl collection of web pages (eg, BERT [27], RoBERTa [28]). Training language models on …


Interpreting and Controlling Linguistic Features in Neural Networks' Representations

Abstract In recent years, Neural Networks proved their usefulness in computing and processing vector representations to solve various NLP tasks. The key objective of the thesis is to provide a better explanation of the neural network’s representation of …


Introducing A large Tunisian Arabizi Dialectal Dataset for Sentiment Analysis

C Fourati, H Haddad, A Messaoudi, M BenHajhmida… - Proceedings of the Sixth …, 2021

… In this pa- per, we introduce a large common-crawl-based Tunisian Arabizi dialectal dataset dedicated for Sentiment Analysis … Hence, our contributions can be summarized as: the first re- lease of a Tunizi large-scale Common-Crawl-based Page 3. 228 …


Introducing huBERT

… Abstract This paper introduces the huBERT family of models. The flagship is the eponymous BERT Base model trained on the new Hungarian Webcorpus 2.0, a 9-billion-token corpus of Web text collected from the Common Crawl …


Introduction: digital humanities and the use of web archives

… Also, an ever-increasing amount of data is selected, archived, and presented worldwide in web collections by organizations like the Internet Archive, Common Crawl, national libraries, and other academic research and digital heritage …


Investigating Depression Semantics on Reddit

S Agarwal, P Singh, J Shah, N Sanjeev - International Conference on Neural …, 2021

Major depression is a challenging issue affecting individuals and those of the people around them. This paper investigates the Reddit comments for the automated identification of comments being indicative of depressive behaviour. We …


Investigating Hostile Post Detection in Hindi

V Bhatnagar, P Kumar, P Bhattacharyya - Neurocomputing, 2021

… Wiki + CC : Pretrained Embeddings trained on Wikipedia and Common Crawl Wiki : Pretrained Embeddings trained on Wikipedia TEAP : … Wikipedia pretrained word vectors 25 and Wikipedia + Common Crawl pretrained word vectors 26 for …


Investigating Monolingual and Multilingual BERTModels for Vietnamese Aspect Category Detection

D Van Thin, LS Le, VX Hoang, NLT Nguyen - arXiv preprint arXiv:2103.09519, 2021

… Besides, there are many variants of BERT models which are published for Vietnamese language and multi-lingual model as cross-lingual language model (XLM-R) [10] or multilingual BERT [9]. These models are pre-trained …


Investigation of different decoding algorithms for the automatic generation of alternative language patterns in German to English translation

… Crawled Corpus (C4) to pretrain the model on. They base this data set on the text data from April 2019 of the Common Crawl archive, which provides web extracted text by scraping about 20TB of text from HTML files each month …


Is Domain Adaptation Worth Your Investment? Comparing BERT and FinBERT on Financial Tasks

B Peng, E Chersoni, YY Hsu, CR Huang - Proceedings of the Third Workshop on …, 2021

… The first-generation Transformers were mainly trained on general corpora, such as Wikipedia or Common Crawl. However, considering domain adaptations, many researchers have later injected domain-specific knowledge in such architectures …


Is the Number of Trainable Parameters All That Actually Matters?

A Chatelain, A Djeghri, D Hesslow, J Launay, I Poli - arXiv preprint arXiv:2109.11928, 2021

… Architecture details for the four model sizes considered are available in the supplementary. The model is trained on French data obtained using a Common Crawl dump filtered by CCNet [23]. We use byte-level byte-pair encoding, with a vocabulary of 50,262 tokens …


isiZulu Word Embeddings

S Dlamini, E Jembere, A Pillay, B van Niekerk - 2021 Conference on Information …, 2021

… TABLE 1: THE MODELS TO BE COMPARED ALONG WITH DETAILS OF THE CORPORA THEY WERE TRAINED ON. Model Corpus Corpus Size Vocabulary Word2Vec Google News 6 billion 1 million GloVe Common …


JABER: Junior Arabic BERt

A Ghaddar, Y Wu, A Rashid, K Bibi, M Rezagholizadeh… - arXiv preprint arXiv …, 2021

… 2020) is a 12 layer BERT model trained on 95GB of common crawl, news, and Wikipedia Arabic data. AraBERT (Antoun et al.… • Common Crawl (CC) This data was downloaded from 10 shards of monthly Common Crawl 3 covering March to …


Jigsaw: Large Language Models meet Program Synthesis

N Jain, S Vaidyanath, A Iyer, N Natarajan… - arXiv preprint arXiv …, 2021

Large pre-trained language models such as GPT-3, Codex, and Google's language model are now capable of generating code from natural language specifications of programmer intent. We view these developments with a mixture of optimism and …


JMIR Mental Health

A Wongkoblap, M Vadillo, V Curcin, C Rauschenberg… - Mental Health

… occurrent words as vectors [25]. GloVe was trained on several textual data sets, such as Wikipedia and common crawl (a copy of web content), and supported 50D, 100D, 200D, and 300D vectors. Anaphora resolution is another …


Joint Open Knowledge Base Canonicalization and Linking

Y Liu, W Shen, Y Wang, J Wang, Z Yang, X Yuan - Proceedings of the 2021 …, 2021

… 1 − Simemb (si,sj ) if xij = 0 For instance, the score ofSimemb (“Barack Obama”, “President Obama”) is 0.873 using fastText [17] embeddings trained on Common Crawl via MindSpore Framework1, which indicates these two NPs are likely to refer to the same entity …


Joint source–target encoding with pervasive attention

M Elbayad, L Besacier, J Verbeek - Machine Translation, 2021

The pervasive attention model is a sequence-to-sequence model that addresses the issue of source–target interaction in encoder–decoder models by jointly encoding the two sequences with a two-dimensional convolutional neural network. We …


Juris2vec: Building Word Embeddings from Philippine Jurisprudence

E Peramo, C Cheng, M Cordel - … International Conference on Artificial Intelligence in …, 2021

… utilized a 2010 Wikipedia dump with 1 billion tokens; a 2014 Wikipedia dump with 1.6 billion tokens; Gigaword 5 which has 4.3 billion tokens; the combination Gigaword5 + Wikipedia2014, which has 6 billion tokens; and …


Just Say No: Analyzing the Stance of Neural Dialogue Generation in Offensive Contexts

A Baheti, M Sap, A Ritter, M Riedl - arXiv preprint arXiv:2108.11830, 2021

… For example, OpenAI's GPT-3 (Brown et al., 2020) is a 175 billion parameter neural network, trained on a 570GB subset CommonCrawl that is capable of engaging in impressive open-domain di- alogues with a user, when …


Just What do You Think You're Doing, Dave?'A Checklist for Responsible Data Use in NLP

A Rogers, T Baldwin, K Leins - arXiv preprint arXiv:2109.06598, 2021

Page 1. 'Just What do You Think You're Doing, Dave?' ∗ A Checklist for Responsible Data Use in NLP Anna Rogers Center for Social Data Science University of Copenhagen Denmark arogers@sodas.ku.dk Timothy Baldwin …


KAN: Knowledge-Augmented Networks for Few-Shot Learning

Z Zhu, X Lin - ICASSP 2021-2021 IEEE International Conference on …, 2021

… 3.3. Hyperparameters For the knowledge graph, to represent the initial semantic features of nodes, we use the GloVe model [23] which is the Common Crawl version trained on 840B tokens. The dimension of the semantic features is 300 …


KazNERD: Kazakh Named Entity Recognition Dataset

R Yeshpanov, Y Khassanov, HA Varol - arXiv preprint arXiv:2111.13419, 2021

We present the development of a dataset for Kazakh named entity recognition. The dataset was built as there is a clear need for publicly available annotated corpora in Kazakh, as well as annotation guidelines containing straightforward--but rigorous--rules …


KI2TE: Knowledge-Infused InterpreTable Embeddings for COVID-19 Misinformation Detection

… After cleaning, we were left with around 20K fake news articles. 3.4.3 Real News Dataset. We used the list from the BS Detector Chrome extension3 to pick the reputable sites. We then collected ar- ticles from all of the matching …


KIT's IWSLT 2021 Offline Speech Translation System

TN Nguyen, TS Nguyen, C Huber, M Awiszus… - Proceedings of the 18th …, 2021

… On the other hand, Page 2. 126 Table 2: Text Training Data Dataset Sentences TED Talks (TED) 220K Europarl (EPPS) 2.2MK CommonCrawl 2.1M Rapid 1.21M ParaCrawl 25.1M OpenSubtitles 12.6M WikiTitle 423K Back-translated News 26M …


Page 1. Titel der Dissertation: Knowledge Graphs for Analyzing and Searching Legal Data Dissertation zur Erlangung des akademischen Grades eines Doktors der Sozialund Wirtschaftswissenschaften an der Wirtschaftsuniversität Wien eingereicht bei …


Knowledge-driven Answer Generation for Conversational Search

M Leite, R Ferreira, D Semedo, J Magalhães - arXiv preprint arXiv:2104.06892, 2021

… This model is pre-trained on the large C4 cor- 5 Page 6. pus, which was derived from Common Crawl1. A masked language modelling objective is used, where the model is trained to predict corrupted randomly sampled tokens of varying sizes. 5. Evaluation 5.1 …


Knowledge-enhanced neural grammar Induction

… become the dominant approach. Thanks to the Transformer's superior capability of parallelism, PLMs can be efficiently trained on enormous raw text (eg, common crawl of the internet) containing billions of tokens. Surprisingly, although PLMs contain no …


Page 1. City University of New York (CUNY) CUNY Academic Works Dissertations, Theses, and Capstone Projects CUNY Graduate Center 9-2021 Label Imputation for Homograph Disambiguation: Theoretical and Practical Approaches Jennifer M. Seale …


Label-Descriptive Patterns and their Application to Characterizing Classification Errors

M Hedderich, J Fischer, D Klakow, J Vreeken - arXiv preprint arXiv:2110.09599, 2021

State-of-the-art deep learning methods achieve human-like performance on many tasks, but make errors nevertheless. Characterizing these errors in easily interpretable terms gives insight into whether a model is prone to making systematic …


LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs

C Schuhmann, R Vencu, R Beaumont, R Kaczmarczyk… - arXiv preprint arXiv …, 2021

… To create image-text pairs, we parse through WAT files from Common Crawl and parse out all HTML IMG tags containing an alt-text attribute. We download the raw images from the parsed URLs with asynchronous requests using Trio and Asks libraries. …


LAMBERT: Layout-Aware Language Modeling for Information Extraction

Ł Garncarek, R Powalski, T Stanisławek, B Topolski…

… tuning, respectively. The training was performed on a collection of PDFs extracted from Common Crawl containing a variety of documents (we randomly selected up to 10 documents from any single domain). The documents …


Language Embeddings for Typology and Cross-lingual Transfer Learning

D Yu, T He, K Sagae - arXiv preprint arXiv:2106.02082, 2021

… lang_emb represents language embeddings from our proposed denoising autoencoder trained with language specific MUSE embeddings, using CommonCrawl text. Eng. lang_emb represents language embeddings trained …


Language Models are not Models of Language

C Veres - arXiv preprint arXiv:2112.07055, 2021

… Also mentioned is the Common Crawl web content where "... difficulty of cleaning and filtering the Common Crawl data ..." which has low … Common Crawl data also contains duplicate lines which often signals highly repetitive "boilerplate text", which …


Language Models have a Moral Dimension

P Schramowski, C Turan, N Andersen, C Rothkopf… - arXiv preprint arXiv …, 2021

Page 1. Language Models have a Moral Dimension Patrick Schramowski, Cigdem Turan, Nico Andersen, Constantin Rothkopf & Kristian Kersting Abstract Artificial writing is permeating our lives due to recent advances in large …


Language Representation Models: An Overview

T Schomacker, M Tropmann-Frick - Entropy, 2021

… Since C4 is based on the common crawl project, it not only contains book-like or article-like texts but a much wider variety of English texts. Despite its colossal size [11], p. 7, ensured that, by their used techniques, C4 only consists of reasonably clean …


Language representations for computational argumentation

… CA Computational Argumentation. – , , , CBOW Continuous Bag of Words. – , , , CC Citation Contexts. CC- CommonCrawl- . , CCC Citation Context Identi cation. – CL Computational Linguistics. , , , , CLWE Cross-lingual Word Embe


Language Transfer for Identifying Diagnostic Paragraphs in Clinical

L Di Liello, O Uryupina, A Moschitti - 2021

This paper aims at uncovering the structure of clinical documents, in particular, identifying paragraphs describing “diagnosis” or “procedures”. We present transformer-based architectures for approaching this task in a monolingual setting (English) …


LaoPLM: Pre-trained Language Models for Lao

N Lin, Y Fu, Z Yang, S Jiang - arXiv preprint arXiv:2110.05896, 2021

… On the one hand, we utilize all the Lao data from the OSCAR corpus9, a humongous multilingual corpus whose texts all come from the Common Crawl corpus10. Suárez et al. [23] proposes the goclassy architecture to perform language …


Large Margin Training Improves Language Models for ASR

J Wang, J Huang, KW Church - ICASSP 2021-2021 IEEE International Conference on …, 2021

… on enWiki+bookcorpus. We thus adopt the published model and skip the pretraining phase. On the other hand, GPT, as a realization of transformer decoder model, is pretrained on a larger corpus, common crawl. For a fair …


Large-Scale Contextualised Language Modelling for Norwegian

A Kutuzov, J Barnes, E Velldal, L Øvrelid, S Oepen - arXiv preprint arXiv:2104.06546, 2021

… models for Bokmål and Nynorsk). However, these models were trained on very modestly sized corpora of 20 million words for each language (randomly sampled from Wikipedia dumps and Common Crawl data). In a parallel effort …


Large-Scale Self-and Semi-Supervised Learning for Speech Translation

C Wang, A Wu, J Pino, A Baevski, M Auli, A Conneau - arXiv preprint arXiv …, 2021

… We also make use of monolingual text data in the target language to further improve translation quality. Specifically, we train a language model (LM) on part of the CommonCrawl data3 that is in similar domains as CoVoST …


LARO: Language Agnostic Sentence Representations from finetuned RoBERTa⋆

… The training resulted in the model approximating the meaning and syntax of all languages in the training data. The dataset which XLM-Roberta was trained on consists of 2.5TB CommonCrawl Data. The architecture of …


LaTr: Layout-Aware Transformer for Scene-Text VQA

AF Biten, R Litman, Y Xie, S Appalaraju, R Manmatha - arXiv preprint arXiv …, 2021

We propose a novel multimodal architecture for Scene Text Visual Question Answering (STVQA), named Layout-Aware Transformer (LaTr). The task of STVQA requires models to reason over different modalities. Thus, we first investigate the …


LATS: Low-Resource Abstractive Text Summarization

CB van Yperen, F Frasincar, K Gruber - 2021

… texts from 350 million Web pages. The dataset is based on the Common Crawl Web archive which is publicly available and contains scraped HTML files from which the markup has been removed. The HTML files have been …