'I'm just feeling like it'. On the relationship between the use of the progressive and sentiment polarity in Italian

… art transformer-based machine learning model for emotion and sentiment classification in Italian which employs the Italian BERT model UmBERTo trained on Commoncrawl ITA (Parisi, Francia, and Magnani [2020] 2021). For …


G Lanzolla, S Santoni, C Tucci - Artificial Intelligence for Sustainable Value Creation, 2021

Page 87. 4. Unlocking value from AI in financial services: strategic and organizational tradeoffs vs. media narratives Gianvito Lanzolla, Simone Santoni and Christopher Tucci 1. INTRODUCTION In 1955, McCarthy wrote that …


6 Data Collection and Representation for Similar Languages, Varieties and Dialects

T Samardžic, N Ljubešic - Similar Languages, Varieties, and Dialects: A …, 2021

… Page 146. Data Collection and Representation for Similar Languages 127 Another project that should be mentioned in this brief overview is the CommonCrawl, a project performing crawls over the whole internet for textual data since 2013 with regular data updates …


" Short is the Road that Leads from Fear to Hate": Fear Speech in Indian WhatsApp Groups

P Saha, B Mathew, K Garimella, A Mukherjee - arXiv preprint arXiv:2102.03870, 2021

Page 1. “Short is the Road that Leads from Fear to Hate”: Fear Speech in Indian WhatsApp Groups Punyajoy Saha punyajoys@iitkgp.ac.in Indian Institute of Technology Kharagpur, West Bengal, India Binny Mathew binnymathew …


A common framework for quantifying the learnability of nouns and verbs

Y Zhou, D Yurovsky - Proceedings of the Annual Meeting of the Cognitive …, 2021

… We used pre-trained 300-dimensional semantic vectors derived from the the Common Crawl corpus composed of 840 billion tokens and 2.2 million words. For our analysis, we considered only the words that corresponded to the relevant 434 images. Procedure …


A Comparative Study on Word Embeddings in Deep Learning for Text Classification

… 3https://nlp.stanford.edu/projects/glove/ 4https://commoncrawl.org/ 5https://fasttext.cc/ 6https://allennlp.org/elmo 7We additionally experimented from the fourth-to-last (-4) layer to the last layer … 300s refers to the GloVe …


Page 1. A Comparison Framework for Product Matching Algorithms by Jeremy Foxcroft A Thesis presented to The University of Guelph In partial fulfilment of requirements for the degree of Master of Science in Computer Science Guelph, Ontario, Canada …


A Comparison of Approaches to Document-level Machine Translation

Z Ma, S Edunov, M Auli - arXiv preprint arXiv:2101.11040, 2021

… WMT17 English-German (en-de). For this benchmark, we follow the setup of Müller et al. (2018) whose training data includes the Europarl, Common Crawl, News Commentary and Rapid corpora, totaling nearly 6M sentence pairs …


A Comprehensive Assessment of Dialog Evaluation Metrics

YT Yeh, M Eskenazi, S Mehri - arXiv preprint arXiv:2106.03706, 2021

… RoBERTa, which is employed in USR (Mehri and Eskenazi, 2020b), improves the training techniques in BERT and trains the model on a much larger corpus which includes the CommonCrawl News dataset (Mackenzie et al., 2020) and text ex- tracted from Reddit …


M Wang, X Xu, Q Yue, Y Wang - arXiv preprint arXiv:2101.12631, 2021

Page 1. A Comprehensive Survey and Experimental Comparison of Graph-Based Approximate Nearest Neighbor Search Mengzhao Wang1, Xiaoliang Xu1, Qiang Yue1, Yuxiang Wang1,∗ 1Hangzhou Dianzi University …


A Comprehensive Survey of Grammatical Error Correction

Y Wang, Y Wang, K Dang, J Liu, Z Liu - ACM Transactions on Intelligent Systems and …, 2021

Grammatical error correction (GEC) is an important application aspect of natural language processing techniques, and GEC system is a kind of very important intelligent system that has long been explored both in academic and industrial …


A Computational Framework for Slang Generation

Z Sun, R Zemel, Y Xu - arXiv preprint arXiv:2102.01826, 2021

… To compare with and compute the baseline em- bedding methods M for definition sentences, we used 300-dimensional fastText embeddings (Bo- janowski et al., 2017) pre-trained with subword information on 600 billion …


A coral-reef approach to extract information from HTML tables

P Jiménez Aguirre, JC Roldán Salvador… - Applied Soft Computing …, 2022

P Jiménez, JC Roldán, R Corchuelo - Applied Soft Computing, 2021

… Unfortunately, a recent analysis of the 32.04 million domains in the November 2019 Common Crawl has revealed that only 11.92 million domains provide such semantic hints [10], which argues for a method to deal with the remaining 20.12 …


A COVID-19 news coverage mood map of Europe

F Robertson, J Lagus, K Kajava - Proceedings of the EACL Hackashop on News …, 2021

… Newscrawl is a web crawl provided by the Common Crawl organisation which is updated more frequently and contains only data from news websites2. In order to keep the size of the corpus manageable and the extraction …


A data quality approach to the identification of discrimination risk in automated decision making systems

A Vetrò, M Torchiano, M Mecati - Government Information Quarterly, 2021

… Similarly, a scientific experiment on the search engine Common Crawl (De-Arteaga et al., 2019) revealed an unequal treatment due to gender imbalance in the input data (almost 400,000 biographies): authors compared …


A data-centric review of deep transfer learning with applications to text data

S Bashath, N Perera, S Tripathi, K Manjang, M Dehmer… - Information Sciences, 2021

Abstract In recent years, many applications are using various forms of deep learning models. Such methods are usually based on traditional learning paradigms requiring the consistency of properties among the feature spaces of the training and …


A deep learning-based bilingual Hindi and Punjabi named entity recognition system using enhanced word embeddings

A Goyal, V Gupta, M Kumar - Knowledge-Based Systems, 2021

… Initially, we collect Facebook’s pre-trained FastText embeddings which are trained on Wikipedia and common crawl data with 300 dimensions for our Hindi and Punjabi datasets. But after experiments, we find many of the words in our dataset are …


A Framework for Generating Extractive Summary from Multiple Malayalam Documents

K Manju, S David Peter, SM Idicula - Information, 2021

… Semantically similar words are mapped to nearby points in the vector space. In this work the vectorization of the terms in the document are performed using the pretrained word embedding model FastText for Malayalam, trained on Common Crawl and Wikipedia …


A Framework for Quality Assessment of Semantic Annotations of Tabular Data

R Avogadro, M Cremaschi, E Jiménez-Ruiz, A Rula - International Semantic Web …, 2021

… 1 Introduction. Much information is conveyed within tables. A prominent example is the large set of relational databases or tabular data present on the Web. To size the spread of tabular data, 2.5M tables have been …


A Fusion Approach for Paper Submission Recommendation System

ST Huynh, N Dang, PT Huynh, DH Nguyen, BT Nguyen - International Conference on …, 2021

… Finally, we use crawl-300d-2M 3 as the pre-train embedding matrix, which has 600 billion tokens and 2 million word vectors trained on Common Crawl. It can make using crawl-300d-2M more efficiently in vectorization. As depicted in Fig …


A General Language Assistant as a Laboratory for Alignment

A Askell, Y Bai, A Chen, D Drain, D Ganguli… - arXiv preprint arXiv …, 2021

… For language model pre-training, these models are trained for 400B tokens on a distribution consisting mostly of filtered common crawl … The natural language dataset was composed of 55% heavily filtered common crawl data (220B tokens), 32 …


A Heuristic-driven Ensemble Framework for COVID-19 Fake News Detection

SD Das, A Basak, S Dutta - arXiv preprint arXiv:2101.03545, 2021

… of model-specific special tokens. Each model also has its corresponding vocabulary associated with its tokenizer, trained on a large corpus data like GLUE, wikitext-103, CommonCrawl data etc. During training, each model …


A Heuristic-driven Uncertainty based Ensemble Framework for Fake News Detection in Tweets and News Articles

SD Das, A Basak, S Dutta - arXiv preprint arXiv:2104.01791, 2021

… Each model also has its corresponding vocabulary associated with its tokenizer, trained on a large corpus data like GLUE, wikitext-103, CommonCrawl data etc. During training, each model applies the tokenization …


A Human Being Wrote This Law

… GPT-3 had an impressively large data training set: it was trained on the Common Crawl dataset, a nearly trillion-word dataset,22 which includes everything from traditional news sites like the New York Times to sites like Reddit.The Common …


A Literature Survey of Recent Advances in Chatbots

… This led to the development of pretrained systems such as BERT (Bidirectional Encoder Representations from transformers) [46] and GPT (Generative Pre-trained Transformer), which were trained with huge language datasets, such as Wikipedia …


A Mechanism for Producing Aligned Latent Spaces with Autoencoders

S Jain, A Radhakrishnan, C Uhler - arXiv preprint arXiv:2106.15456, 2021

… 6.1 Alignment of GloVe Embeddings In this section, we apply our theory to align semantic/syntactic directions in GloVe word embeddings [21]. We use 300 dimensional GloVe vectors that were trained on Common Crawl with 840 billion tokens …


A Multi-Platform Analysis of Political News Discussion and Sharing on Web Communities

Y Wang, S Zannettou, J Blackburn, B Bradlyn… - arXiv preprint arXiv …, 2021

… supported types of entities). The model re- lies on Convolutional Neural Networks (CNNs), trained on the OntoNotes dataset [90], as well as Glove vectors [62] trained on the Common Crawl dataset [17]. 2.3 News Stories Identification …


A Multi-Task Learning Model for Multidimensional Relevance Assessment

DGP Putri, M Viviani, G Pasi - International Conference of the Cross-Language …, 2021

… 6 In particular, we focused on the ad-hoc retrieval subtask. The data consist of Web pages crawled by means of CommonCrawl, 7 related to the health-related domain. The data collections consider 50 topics/queries and associated documents …


A Multifactorial Approach to Crosslinguistic Constituent Orderings

… The data for training these LMs was taken from the raw data of the CoNLL 2017 Shared Task on multilingual parsing (Ginter et al. 2017), which contains texts from Common Crawl and Wikipedia. The architecture of the LM was the same for every language …


A Multitask Framework to Detect Depression, Sentiment and Multi-label Emotion from Suicide Notes

S Ghosh, A Ekbal, P Bhattacharyya - Cognitive Computation, 2021

The significant rise in suicides is a major cause of concern in public health domain. Depression plays a major role in increasing suicide ideation among th.


A Novel Corpus of Discourse Structure in Humans and Computers

B Hemmatian, S Feucht, R Avram, A Wey, M Garg… - arXiv preprint arXiv …, 2021

We present a novel corpus of 445 humanand computer-generated documents, comprising about 27,000 clauses, annotated for semantic clause types and coherence relations that allow for nuanced comparison of artificial and natural …


A novel fusion-based deep learning model for sentiment analysis of COVID-19 tweets

ME Basiri, S Nemati, M Abdar, S Asadi, UR Acharrya - Knowledge-Based Systems, 2021

JavaScript is disabled on your browser. Please enable JavaScript to use all the features on this page. Skip to main content Skip to article …


A NOVEL TRILINGUAL DATASET FOR CRISIS NEWS CATEGORIZATION ACROSS LANGUAGES

… Use Common Crawl instead • “the goal of democratizing access to web information by producing and maintaining an open repository of web crawl data that is universally accessible and analyzable” (https://commoncrawl.org/about/, accessed June 1 2021) …


A Primer on Pretrained Multilingual Language Models

S Doddapaneni, G Ramesh, A Kunchukuttan, P Kumar… - arXiv preprint arXiv …, 2021

… and they differ in the architecture (eg, number of layers, parameters, etc), objective functions used for training (eg, monolingual masked language modeling objective, translation language modeling objective, etc), data used …


A Probing Task on Linguistic Properties of Korean Sentence Embedding

A Ahn, BI Ko, D Lee, G Han, M Shin, J Nam - Annual Conference on Human and …, 2021

Abstract 본 연구는 한국어 문장 임베딩 (embedding) 에 담겨진 언어적 속성을 평가 하기 위한 프로빙 태스크 (Probing Task) 를 소개한다. 프로빙 태스크는 임베딩 으로부터 문장의 표층적, 통사적, 의미적 속성을 구분하는 문제로 영어, 폴란드어 …


A Residual Network Architecture for Hindi NER using Fasttext and BERT embedding layers

… It provides word embedding for Hindi (and 157 other languages) and is based on the CBOW (Continuous Bag-of-Words) model. The CBOW model learns by predicting the current word based on its context, and it was trained on Common Crawl and Wikipedia …


F Alam, A Hasan, T Alam, A Khan, J Tajrin, N Khan… - arXiv preprint arXiv …, 2021

Page 1. A Review of Bangla Natural Language Processing Tasks and the Utility of Transformer Models FIROJ ALAM, Qatar Computing Research Institute, HBKU, Qatar MD. ARID HASAN, Cognitive Insight Limited, Bangladesh …


A Review of Public Datasets in Question Answering Research

BB Cambazoglu, M Sanderson, F Scholer, B Croft

… matching the question. The sentences are selected based on their tf-idf similarity to the question. The underlying web page collection contains pages from the July 2018 archive of the Common Crawl web repository. Task. Given a …


A Semi-supervised Multi-task Learning Approach to Classify Customer Contact Intents

… We note that this ALBERT model is trained as a multiclass classification with only positive cases. 3.2.2 SS MT D/TAPT ALBERT The pretrained language models are mostly trained on well-known corpora, such as Wikipedia, Common Crawl, BookCorpus, Reddit, etc …


A Short Survey of LSTM Models for De-identification of Medical Free Text

JL Leevy, TM Khoshgoftaar - 2020 IEEE 6th International Conference on …, 2020

… The training set was obtained from the 2014 i2b2 challenge, while the test set came from the University of Florida (UF) Health Integrated Data Repository 5. Word embeddings were sourced from GoogleNews [54] …


A Simple Post-Processing Technique for Improving Readability Assessment of Texts using Word Mover's Distance

JM Imperial, E Ong - arXiv preprint arXiv:2103.07277, 2021

… technique described in Section 4. For the word embeddings of English, German, and Filipino needed for the technique, we downloaded the resources from the fastText website5. The word embeddings in various …


A Simple Recipe for Multilingual Grammatical Error Correction

S Rothe, J Mallinson, E Malmi, S Krause, A Severyn - arXiv preprint arXiv:2106.03830, 2021

… 2.1 mT5 Pre-training mT5 has been pre-trained on mC4 corpus, a subset of Common Crawl, covering 101 languages and composed of about 50 billion documents. For details on mC4, we refer the reader to the original paper (Xue et al., 2020) …


A Spontaneous Stereotype Content Model: Taxonomy, Properties, and Prediction.

… model trained on the Common Crawl (600 billion words obtained from various internet sources) … the Common Crawl (600 billion words), a Glove model trained using around 840 billion words from the Common Crawl (Pennington, Socher, & Manning, 2014; …


A Study of Analogical Density in Various Corpora at Various Granularity

In this paper, we inspect the theoretical problem of counting the number of analogies between sentences contained in a text. Based on this, we measure the analogical density of the text. We focus on analogy at the sentence level …


A Study of Analogical Density in Various Corpora at Various Granularity. Information 2021, 12, 314

… Table 4 shows the statistics of Multi30K corpus. • CommonCrawl (available at: commoncrawl.org accessed on 20 September 2020) is a crawled web archive and dataset … Table 5 shows the statistics on the CommonCrawl corpus …


A study on performance improvement considering the balance between corpus in Neural Machine Translation

C Park, K Park, H Moon, S Eo, H Lim - Journal of the Korea Convergence Society, 2021

… 1. Concept of Corpus Weight Balance GPT3도 Common Crawl, WebText2, Books1, Books2, Wikipedia 등의 데이터를 합쳐 모델을 훈련하 게 된다. 그러나 말뭉치 간의 특성 및 특징(어투, 문체, 도메인 등)이 다름에도 하나의 데이터로 …


AR Ullah, A Das, A Das, MA Kabir, K Shu - arXiv preprint arXiv:2110.00737, 2021

Page 1. A Survey of COVID-19 Misinformation: Datasets, Detection Techniques and Open Issues AR Sana Ullaha, Anupam Dasa, Anik Dasb, Muhammad Ashad Kabirc,∗, Kai Shud aDepartment of Computer Science and Engineering …


A Survey of Machine Learning-Based Solutions for Phishing Website Detection

L Tang, QH Mahmoud - Machine Learning and Knowledge Extraction, 2021

With the development of the Internet, network security has aroused people's attention. It can be said that a secure network environment is a basis for the rapid and sound development of the Internet. Phishing is an essential class …


A Survey of Recent Abstract Summarization Techniques

D Puspitaningrum - Proceedings of Sixth International Congress on …, 2021

… For C4, taken from Common Crawl scrape from April 2019 and applied some cleansing filters, it results in a very clean 750GB text dataset of large pre-training datasets, more extensive than other pre-training datasets. 3.2 Pegasus-XSum (Pegasus) …


A Survey on Bias in Deep NLP

I Garrido-Muñoz, A Montejo-Ráez, F Martínez-Santiago… - 2021

… 2016 [26] Gender Word2Vec, GloVe GoogleNews corpus (w2vNEWS), Common Crawl English Analogies/Cosine Similarity Vector Space Manipulation After - 2017 [10] Gender, Ethnicity GloVe, Word2Vec Common …


A Survey on Data Augmentation for Text Classification

M Bayer, MA Kaufhold, C Reuter - arXiv preprint arXiv:2107.03158, 2021

… CNN+LSTM/GRU HON RSN-1 RSN-2 Word2Vec Hate Speech FastText Wikipedia GoogleNews W2V GloVe Common Crawl GloVe Common Crawl GloVe Common Crawl -22.7 (Macro F1) +1.0 -3.3 +0.3 -0.2 0 [44] 1. Method …


R Wang, X Tan, R Luo, T Qin, TY Liu - arXiv preprint arXiv:2107.04239, 2021

Page 1. A Survey on Low-Resource Neural Machine Translation Rui Wang, Xu Tan, Renqian Luo, Tao Qin and Tie-Yan Liu Microsoft Research Asia {ruiwa, xuta, t-reluo, taoqin, tyliu}@microsoft.com Abstract Neural approaches …


A Survey On Neural Word Embeddings

E Sezerer, S Tekir - arXiv preprint arXiv:2110.01804, 2021

… ivLBL/vLBL [95] 2013 100-600 Wiki LBL Performance - NCE [47] GloVe [109] 2014 300 Wiki, Gigaword, Commoncrawl LBL+coocurence Matrix Training - - DEPS [69] 2014 300 Wiki CBOW Training Stanford tagger[129] …


A Survey on Statistical Approaches for Abstractive Summarization of Low Resource Language Documents

P Deshpande, S Jahirabadkar - Smart Trends in Computing and Communications, 2022

… German Wiki data is used as real data and synthetic data is a common crawl data. Synthetic data is used to increase size of data. Three settings are considered for generation of summaries: (1) Transformer model using real data for training. …


Page 1. A Synthetic FACS Framework for Expanding Facial Expression Lexicons DISSERTATION Submitted in Partial Fulfillment of the Requirements for the Degree of DOCTOR OF PHILOSOPHY (Computer Science) at the …


A system for proactive risk assessment of application changes in cloud operations

R Batta, L Shwartz, M Nidd, AP Azad, H Kumar - 2021 IEEE 14th International …, 2021

Abstract Change is one of the biggest contributors to service outages. With more enterprises migrating their applications to cloud and using automated build and deployment the volume and rate of changes has significantly increased. Furthermore …


A Systematic Investigation of Commonsense Understanding in Large Language Models

XL Li, A Kuncoro, CM d'Autume, P Blunsom… - arXiv preprint arXiv …, 2021

… 2019), we train our models using the cleaned version of Common Crawl corpus (C4), around 800 GB of data. Our largest model, with 32 transformer layers and 7 billion parameters, has a similar number of parameters to the open-sourced GPT-J model (Wang …


A systems-wide understanding of the human olfactory percept chemical space

J Kowalewski, B Huynh, A Ray - Chemical Senses, 2021

… 2015; spaCy, 2016), and a convolutional neural network previously trained on GloVe Common Crawl (Pennington, Socher, & Manning, 2014) and OntoNotes 5. The training set is comprised of more than 1 million English …


A Targeted Attack on Black-Box Neural Machine Translation with Parallel Data Poisoning

C Xu, J Wang, Y Tang, F Guzmán, BIP Rubinstein… - Proceedings of the Web …, 2021

… 3https://commoncrawl.org/ 4We assume that these poisoned web pages are archived and to be used for parallel data extraction. This assumption is realistic as we find that the crawling services commonly used for parallel …


A unified approach to sentence segmentation of punctuated text in many languages

Page 1. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 3995–4007 August 1–6, 2021 …


A word embedding-based approach to cross-lingual topic modeling

CH Chang, SY Hwang - Knowledge and Information Systems, 2021

The cross-lingual topic analysis aims at extracting latent topics from corpora of different languages. Early approaches rely on high-cost multilingual reso.


ABC: Attention with Bounded-memory Control

H Peng, J Kasai, N Pappas, D Yogatama, Z Wu, L Kong… - arXiv preprint arXiv …, 2021

Page 1. Under review as a conference paper at ICLR 2022 ABC: ATTENTION WITH BOUNDED-MEMORY CONTROL Hao Peng♠ Jungo Kasai♠ Nikolaos Pappas♠ Dani Yogatama♣ Zhaofeng Wu♦∗ Lingpeng Kong♦ Roy Schwartz …


Abuse is Contextual, What about NLP? The Role of Context in Abusive Language Annotation and Detection

S Menini, AP Aprosio, S Tonelli - arXiv preprint arXiv:2103.14916, 2021

… In particular, all vectors are extracted starting from the pre-trained embeddings obtained from the Common Crawl corpus.5 Since SVM takes in input sentence embeddings, we convert the context and the current tweet …


Accelerated execution via eager-release of dependencies in task-based workflows

H Elshazly, F Lordan, J Ejarque, RM Badia - The International Journal of High …, 2021

Task-based programming models offer a flexible way to express the unstructured parallelism patterns of nowadays complex applications. This expressive capability is required to achieve maximum possi...


Accelerating Text Communication via Abbreviated Sentence Input

… For our out-of-domain training set, we used one billion words of web text from Common Crawl1. We only … As shown in Table 1, random sentences from Common Crawl averaged 30 words. The cross-entropy 1https://commoncrawl …


Accurate Word Representations with Universal Visual Guidance

Z Zhang, H Yu, H Zhao, R Wang, M Utiyama - arXiv preprint arXiv:2012.15086, 2020

… WMT'14 EN-DE 4.43M bilingual sentence pairs of the WMT14 dataset were used as training data, including Common Crawl, News Commentary, and Europarl v7. The newstest2013 and newstest2014 datasets were …


Acquiring and Harnessing Verb Knowledge for Multilingual Natural Language Processing

Advances in representation learning have enabled natural language processing models to derive non-negligible linguistic information directly from text corpora in an unsupervised fashion. However, this signal is underused in downstream tasks …


Active Learning for Argument Mining: A Practical Approach

N Solmsdorf, D Trautmann, H Schütze - arXiv preprint arXiv:2109.13611, 2021

… easily discernible argumentative statements. The corpus contains 1,000 sentences per topic, ie, in total 8,000 instances, which were tapped from a Common Crawl snapshot and in- dexed with Elasticsearch. The time-consuming …


Adapting Neural Machine Translation for Automatic Post-Editing

… reference as the output. 3.2 Pre-training on domain-specific data FAIR's WMT'19 NMT model was trained on Newscrawl and Commoncrawl datasets while the source of this year's APE data is Wikipedia. To fix the domain mismatch …


Adaptive Ranking Relevant Source Files for Bug Reports Using Genetic Algorithm

Abstract. Precisely locating buggy files for a given bug report is a cumbersome and time-consuming task, particularly in a large-scale project with thousands of source files and bug reports. An efficient bug localization module is desirable to improve the …


ADEPT: An Adjective-Dependent Plausibility Task

A Emami, I Porada, A Olteanu, K Suleman, A Trischler…

… 4 Dataset To construct ADEPT, we scrape text samples from English Wikipedia and Common Crawl, extracting adjectival modifier-noun pairs that occur with high frequency … We extracted 10 million pairs from English …


ADPBC: Arabic Dependency Parsing Based Corpora for Information Extraction

… Sch. Econ. Res. Pap. No. WP BRP., 2018. [27] A. Panchenko, E. Ruppert, S. Faralli, SP Ponzetto, and C. Biemann, “Building a web-scale dependency-parsed corpus from common crawl,” Lr. 2018 - 11th Int. Conf. Lang. Resour. Eval., pp. 1816–1823, 2019 …


Advances and Trends in Artificial Intelligence. From Theory to Practice: 34th International Conference on Industrial, Engineering and Other Applications of Applied …

Page 1. Hamido Fujita Ali Selamat Jerry Chun-Wei Lin Moonis Ali (Eds.) Advances and Trends in Artificial Intelligence From Theory to Practice 34th International Conference on Industrial, Engineering and Other Applications …


Aggressive and Offensive Language Identification in Hindi, Bangla, and English: A Comparative Study

R Kumar, B Lahiri, AK Ojha - SN Computer Science, 2021

In the present paper, we carry out a comparative study between offensive and aggressive language and attempt to understand their inter-relationship. To car.


AlephBERT: A Hebrew Large Pre-Trained Language Model to Start-off your Hebrew NLP Application With

A Seker, E Bandel, D Bareket, I Brusilovsky… - arXiv preprint arXiv …, 2021

… Oscar: A deduplicated Hebrew portion of the OSCAR corpus, which is “extracted from Common Crawl via language classification, filtering and cleaning” (Ortiz Suárez et al., 2020). • Twitter: Texts of Hebrew tweets collected between 2014-09-28 and 2018-03-07 …


Alignment of Language Agents

Z Kenton, T Everitt, L Weidinger, I Gabriel, V Mikulik… - arXiv preprint arXiv …, 2021

… Large scale unlabeled datasets are collected from the web, such as the CommonCrawl dataset (Raffel et al., 2019). Input data and labels are created by chopping a sentence into … Brown et al. (2020) attempt to im- prove …


All Labels Are Not Created Equal: Enhancing Semi-supervision via Label Grouping and Co-training

I Nassar, S Herath, E Abbasnejad, W Buntine, G Haffari - arXiv preprint arXiv …, 2021

… A detailed description of such relations and examples thereof can be found in the ConceptNet documentation8. On the other hand, GloVe and word2vec are two prominent sets of word embeddings, the former is trained on 840 …


Z Du, Y Qian, X Liu, M Ding, J Qiu, Z Yang, J Tang - arXiv preprint arXiv:2103.10360, 2021

Page 1. All NLP Tasks Are Generation Tasks: A General Pretraining Framework Zhengxiao Du *12 Yujie Qian * 3 Xiao Liu 1 2 Ming Ding 1 2 Jiezhong Qiu 1 2 Zhilin Yang 4 2 Jie Tang 1 2 Abstract There have been various types …


Allocating Large Vocabulary Capacity for Cross-lingual Language Model Pre-training

B Zheng, L Dong, S Huang, S Singhal, W Che, T Liu… - arXiv preprint arXiv …, 2021

… are learned on the reconstructed CommonCrawl corpus (Chi et al., 2021b; Conneau et al., 2020) using SentencePiece (Kudo and Richardson, 2018) with the unigram language model (Kudo, 2018). The unigram distributions …


ALX: Large Scale Matrix Factorization on TPUs

H Mehta, S Rendle, W Krichene, L Zhang - arXiv preprint arXiv:2112.02194, 2021

We present ALX, an open-source library for distributed matrix factorization using Alternating Least Squares, written in JAX. Our design allows for efficient use of the TPU architecture and scales well to matrix factorization problems of O(B) rows/columns …


AMMUS: A Survey of Transformer-based Pretrained Models in Natural Language Processing

KS Kalyan, A Rajasekharan, S Sangeetha - arXiv preprint arXiv:2108.05542, 2021

… mT6 [91], XLM-E [89] CC-Aligned [108] Parallel corpus of 292 million non-English common crawl document pairs and 100 million English common crawl document pairs. XLM-E [89] Dakshina [109] Parallel corpus containing 10K sentences for 12 In- dian languages …


An Alignment-Based Approach to Semi-Supervised Bilingual Lexicon Induction with Small Parallel Corpora

… learning. 6 Experimental Settings Language Corpus # of words English WaCky, BNC, Wikipedia 2.8 B Italian itWac 1.6 B German SdeWaC 0.9 B Spanish News Crawl 2007-2012 386 M Finnish Common Crawl 2016 2.8 B Table …


Page 1. An analysis of full-size Russian complexly NER labelled corpus of Internet user reviews on the drugs based on deep learning and language neuron nets AG Sboeva,b,, SG Sboevac, IA Moloshnikova, AV Gryaznova, RB …


Page 1. An Effective Deep Learning Approach for Extractive Text Summarization Minh-Tuan Luu PhD. Student, School of Information and Communication Technology, Hanoi University of Science and Technology, No.1 Dai Co …


An embedding method for unseen words considering contextual information and morphological information

MS Won, YS Choi, S Kim, CW Na, JH Lee - Proceedings of the 36th Annual ACM …, 2021

… Random embeddings are assigned for OOv words. Glove is implemented by using preǦtrained 300Ǧdimensional Glove embedding, which is trained on Common Crawl with 840B word tokens. Random embeddings are assigned for OOv words …


An empirical evaluation of text representation schemes to filter the social media stream

S Modha, P Majumder, T Mandl - Journal of Experimental & Theoretical Artificial …, 2021

… Glove pre-trained model available with different embed size and trained on the common crawl, Twitter. We have used the Glove pre-trained model with a vocabulary size of 2.2 million and trained on the common crawl. fastText …


An Empirical Exploration in Quality Filtering of Text Data

L Gao - arXiv preprint arXiv:2109.00698, 2021

… (2020), with a Paretodistribution thresholded filtering method and a shallow CommonCrawl-WebText classfier … (2020) has been made public, we instead use the same type of fasttext (Joulin et al., 2017) classifier between unfiltered …


An Empirical Study on Task-Oriented Dialogue Translation

S Liu - ICASSP 2021-2021 IEEE International Conference on …, 2021

… consistent). We valid them with SENT-BASE model on En⇒De task. data in WMT20 news domain, which consists of CommonCrawl and NewsCommentary. We conduct data selection to select similar amount of sentences …


An End-to-end Point of Interest (POI) Conflation Framework

R Low, ZD Tekler, L Cheah - arXiv preprint arXiv:2109.06073, 2021

… words that did not appear in the training data [63]. For this study, the fastText model was pre-trained on 2 million word vectors with subword information from commoncrawl.org. The second advantage of using the fastText library to …


An evaluation dataset for depression detection in Arabic social media

S Elimam, M Bougeussa - International Journal of Knowledge Engineering and …, 2021

Studying depression in Arabic social media has been neglected compared to other languages and the traditional way of dealing with depression (face-to-face medical diagnose) is not enough as the number of people that suffer from depression in …


An Explainable Multi-Modal Hierarchical Attention Model for Developing Phishing Threat Intelligence

Y Chai, Y Zhou, W Li, Y Jiang - IEEE Transactions on Dependable and Secure …, 2021

Phishing website attack, as one of the most persistent forms of cyber threats, evolves and remains a major cyber threat. Various detection methods (eg, lookup systems, fraud cue-based methods) have been proposed to identify phishing websites. The …


Page 1. An Exploration of Alignment Concepts to Bridge the Gap between Phrase-based and Neural Machine Translation Von der Fakultät für Mathematik, Informatik und Naturwissenschaften der RWTH Aachen University zur …


An Exploratory Analysis of Multilingual Word-Level Quality Estimation with Cross-Lingual Transformers

T Ranasinghe, C Orasan, R Mitkov - arXiv preprint arXiv:2106.00143, 2021

… Our architecture relies on the XLM-R transformer model (Conneau et al., 2020) to derive the representations of the input sentences. XLM-R has been trained on a large-scale multilingual dataset in 104 languages, totalling …


An Exploratory Study on Utilising the Web of Linked Data for Product Data Mining

Z Zhang, X Song - arXiv preprint arXiv:2109.01411, 2021

… The Web Data Commons3 (WDC) project extracts such structured data from the CommonCrawl4 as RDF n-quads5, and release them on … to create a very large training dataset for product entity linking using semantic markup data …


An extended analysis of the persistence of persistent identifiers of the scholarly web

M Klein, L Balakireva - International Journal on Digital Libraries, 2021

… These findings were confirmed in a large-scale study by Thompson and Jian [22] based on two samples of the web taken from Common Crawl Footnote 3 datasets. The authors were motivated to quantify the use of HTTP DOIs versus URLs of …


An Intrinsic and Extrinsic Evaluation of Learned COVID-19 Concepts using Open-Source Word Embedding Sources

S Parikh, A Davoudi, S Yu, C Giraldo, E Schriver… - medRxiv

… 8] and GloVe [9] on large corpora of texts including domain-independent texts (eg, internet web pages like Wikipedia and CommonCrawl; social media … Standard GloVe Embeddings Paper Vectors [9] Common Crawl Token 10 …


An Investigation towards Differentially Private Sequence Tagging in a Federated Framework

… 2The hyperparameter settings to train those models are as follows: epochs- 10, batch size - 32, learning rate - 0.15, optimizer - Stochastic gradient descent (SGD) Common Crawl corpus) from spaCy library3, the dimension of which is 300 …


An Overview on Evaluation Labs and Open Issues in Health-related Credible Information Retrieval

… The 2020 Track used a dataset provided by Common Crawl, in particular related to different news collected in the first four months of 2020.4 On … 2https://trec-health-misinfo.github.io/2019.html 3https://lemurproject.org …


D Loureiro, K Rezaee, MT Pilehvar… - Computational Linguistics, 2021

Page 1. Analysis and Evaluation of Language Models for Word Sense Disambiguation Daniel Loureiro∗ LIAAD - INESC TEC Department of Computer Science - FCUP University of Porto, Portugal dloureiro@fc.up.pt Kiamehr …


Analysis of Machine Learning and Deep Learning Frameworks for Opinion Mining on Drug Reviews

F Youbi, N Settouti - The Computer Journal, 2021

… More precisely, GloVe consists of collecting word co- occurrence statistics in a form of a word co-occurrence matrix, in which its developers have provided pre-embed millions of English tokens obtained from Wikipedia data and common crawl data …


Analyzing Hyperonyms of Stack Overflow Posts

… They applied a similar lexico-syntactic pattern-based mining on the dataset obtained from CommonCrawl [17] using a slightly different grammar for NP identification and, therefore, a slightly different set of patterns. Despite the differences …


Analyzing Multimodal Language via Acoustic-and Visual-LSTM with Channel-aware Temporal Convolution Network

S Mai, S Xing, H Hu - IEEE/ACM Transactions on Audio, Speech, and …, 2021

Page 1. 2329-9290 (c) 2021 IEEE. Personal use is permitted, but republication/ redistribution requires IEEE permission. See http://www.ieee.org/ publications_standards/publications/rights/index.html for more information. This …


Analyzing the Forgetting Problem in Pretrain-Finetuning of Open-domain Dialogue Response Models

T He, J Liu, K Cho, M Ott, B Liu, J Glass, F Peng - … of the 16th Conference of the …, 2021

… 4.1 Datasets For pretraining, we use the large-scale CCNEWS data (Bakhtin et al., 2019) which is a de-duplicated subset of the English portion of the CommonCrawl news dataset1 … We tune the 1 http://commoncrawl.org/2016/10/ news-dataset-available Page 5. 1125 …


Analyzing transfer learning impact in biomedical cross-lingual named entity recognition and normalization

RM Rivera-Zavala, P Martínez - BMC Bioinformatics, 2021

… The FastText-2M [52] pre-trained English word embedding model trained with subword information on Common Crawl using the FastText implementation. Finally, the PubMed and PMC [53] pre-trained English word embedding model, trained on a …


Annotation of Fine-Grained Geographical Entities in German Texts

J Moreno-Schneider, M Plakidis, G Rehm - 3rd Conference on Language, Data and …, 2021

… The SpaCy models are trained on Ontonotes 5 and Common Crawl (English; en_core_web_md) and WikiNER and TIGER (German; de_core_news_md). The Stanford models are trained on the CoNLL 2003 data [18]. BERT-NER is trained on WikiNER [11] …


Answering questions about insurance supervision with a Neural Machine Translator

… Conneau et al. [3] pre-train a model based on the RoBERTa architecture to create crosslingual representations. The XLM-R model is pre-trained on a common crawl dataset of 100 languages using the masked multi-lingual language model ap- proach …


Anticipating Attention: On the Predictability of News Headline Tests

N Hagar, N Diakopoulos, B DeWilde - Digital Journalism, 2021

… These embeddings contain 300 dimensions and were trained on English language text from the OntoNotes 5.0 and GloVe Common Crawl corpora. For each headline, we computed the average embedding vector across all tokens. …


Applying and Understanding an Advanced, Novel Deep Learning Approach: A Covid 19, Text Based, Emotions Analysis Study

J Choudrie, S Patil, K Kotecha, N Matta, I Pappas - Information Systems Frontiers, 2021

The pandemic COVID 19 has altered individuals' daily lives across the globe. It has led to preventive measures such as physical distancing to be impo.


Applying Deep Learning Techniques for Sentiment Analysis to Assess Sustainable Transport

A Serna Nocedal, A Soroa Echave, R Agerri Gascón - 2021

A Serna, A Soroa, R Agerri - Sustainability, 2021

… Thus, the multilingual version of BERT [25] was trained for 104 languages. More recently, XLM-RoBERTa [21] distributes a multilingual model which contains 100 languages trained on 2.5 TB of filtered Common Crawl text. To …


Applying Deep Learning Techniques for Sentiment Analysis to Assess Sustainable Transport. Sustainability 2021, 13, 2397


W Zhong, S Wang, D Tang, Z Xu, D Guo, J Wang, J Yin… - arXiv preprint arXiv …, 2021

Page 1. AR-LSAT: Investigating Analytical Reasoning of Text Wanjun Zhong1∗, Siyuan Wang3∗, Duyu Tang2, Zenan Xu1∗, Daya Guo1∗ Jiahai Wang1, Jian Yin1, Ming Zhou4 and Nan Duan2 1 The School of Data and Computer Science, Sun Yat-sen University …


Arabic Offensive Language Detection in Social Media

Page 1. ARABIC OFFENSIVE LANGUAGE DETECTION IN SOCIAL MEDIA by Fatemah Ali Husain A Dissertation Submitted to the Graduate Faculty of George Mason University in Partial Fulfillment of The Requirements for the …


AraCOVID19-SSD: Arabic COVID-19 Sentiment and Sarcasm Detection Dataset

MS Hadj Ameur - Revue de l'Information Scientifique et Technique, 2023

MSH Ameur, H Aliane - arXiv preprint arXiv:2110.01948, 2021

… Multilingual BERT (mBERT)6: A BERT-based model [17] pretrained on the first 104 major Wikipedia languages7. • XLM-Roberta 8: A large multi-lingual language model, trained on 2.5TB of filtered Common Crawl data [19]. 4.2.2 Bag-of-Words Models …


AraStance: A Multi-Country and Multi-Domain Dataset of Arabic Stance Detection for Fact Checking

T Alhindi, A Alabdulkarim, A Alshehri, M Abdul-Mageed… - arXiv preprint arXiv …, 2021

… AraStance and Khoja. This indicates the suitability of the pretraining data of ARBERT that includes Books, Gi- gawords and Common Crawl data primarily from MSA but also a small amount of Egyptian Arabic. Since half of …


ARBERT & MARBERT: Deep Bidirectional Transformers for Arabic

M Abdul-Mageed, AR Elmadany, EMB Nagoudi - arXiv preprint arXiv:2101.01785, 2020

… mBERT (Devlin et al., 2019) and XLM-R (Conneau et al., 2020) use a small Arabic text collection from Wikipedia (153M tokens) and CommonCrawl (2.9B … XLM-R (Conneau et al., 2020) is trained on Common Crawl data, hence …


Are Multilingual Models Effective in Code-Switching?

GI Winata, S Cahyawijaya, Z Liu, Z Lin, A Madotto… - arXiv preprint arXiv …, 2021

… switching tasks. 2.2.2 XLM-RoBERTa XLM-RoBERTa (XLM-R) (Conneau et al., 2020) is a multilingual language model that is pre-trained on 100 languages using more than two terabytes of filtered CommonCrawl data. Thanks to …


Are Multilingual Models the Best Choice for Moderately Under-resourced Languages? A Comprehensive Assessment for Catalan

J Armengol-Estapé, CP Carrino, C Rodriguez-Penagos… - arXiv preprint arXiv …, 2021

… Catalan Government; (2) the Catalan Open Subtitles, a collection of translated movie subtitles (Tiedemann, 2012); (3) the non-shuffled version of the Catalan part of the OSCAR corpus (Suárez et al., 2019), a collection …


Are You Really Complaining? A Multi-task Framework for Complaint Identification, Emotion, and Sentiment Classification

A Singh, S Saha - International Conference on Document Analysis and …, 2021

… For deep learning baseline (MT\(_{\mathrm{GloVe}}\)) we also used pre-trained GloVe 13 [16] word embedding which is trained on Common Crawl (840 billion tokens) corpus to get the word embedding representations. 4.4 Results and Discussion …


ARMAN: Pre-training with Semantically Selecting and Reordering of Sentences for Persian Abstractive Summarization

A Salemi, E Kebriaei, GN Minaei, A Shakery - arXiv preprint arXiv:2109.04098, 2021

… This corpus contains around 2.8M articles and 1.4B words in all of the articles. CC100 (Conneau et al., 2020; Wenzek et al., 2020) is a monolingual dataset for 100+ languages constructed from Commoncrawl snapshots. This …


As Easy as 1, 2, 3: Behavioural Testing of NMT Systems for Numerical Translation

J Wang, C Xu, F Guzman, A El-Kishky, BIP Rubinstein… - arXiv preprint arXiv …, 2021

… Our testing framework facilitates constructing test instances for new domains in the following steps: 1. Obtain a large corpus of text that contains numbers (eg,CommonCrawl); 2. Check if there is a number in the output translation; …


Aspect-based Sentiment Analysis with Graph Convolution over Syntactic Dependencies

A Žunić, P Corcoran, I Spasić - Artificial Intelligence in Medicine, 2021

… individual sentences into dependency graphs. Individual words representing vertices in such graphs were mapped onto their embeddings, which were pretrained 160 on web data from Common Crawl using the GloVe method [31]. Each input …


ASR4REAL: An extended benchmark for speech models

M Riviere, J Copet, G Synnaeve - arXiv preprint arXiv:2110.08583, 2021

… even a language model trained on a dataset as big as Common Crawl does not seem to have significant positive effect which reiterates … For all of theses models we used the a 4-gram LM trained on Common Crawl with the decoding parameters …


Page 1. Assessing reasoning and world knowledge of large language models using questionized counterfactual conditionals Jörg Frohberg apergo UG Leipzig, Germany j.frohberg@apergo.ai Frank Binder Institute for Applied …


Assessing the Extent and Types of Hate Speech in Fringe Communities: A Case Study of Alt-Right Communities on 8chan, 4chan, and Reddit

D Rieger, AS Kümpel, M Wich, T Kiening, G Groh - Social Media+ Society, 2021

… For this article, the fastText word vectors pre-trained on the English Common Crawl dataset were used because it is trained on web data and thus an appropriate basis (Mikolov, Grave, Bojanowski, Puhrsch, & Joulin, 2019). …


HT Madabushi, E Gow-Smith, C Scarton… - arXiv preprint arXiv …, 2021

Page 1. AStitchInLanguageModels: Dataset and Methods for the Exploration of Idiomaticity in Pre-Trained Language Models Harish Tayyar Madabushi, Edward Gow-Smith, Carolina Scarton and Aline Villavicencio Department …


Attention-based model for predicting question relatedness on Stack Overflow

J Pei, Z Qin, Y Cong, J Guan - arXiv preprint arXiv:2103.10763, 2021

… released by Stanford [22]. This word embeddings pre-trained in the Common Crawl corpus, which contains a large amount of data irrelevant to software engineering, may lead to ambiguous results [18]. Therefore, we hope that …


Attention: there is an inconsistency between android permissions and application metadata!

H Alecakir, B Can, S Sen - International Journal of Information Security

Since mobile applications make our lives easier, there is a large number of mobile applications customized for our needs in the application markets. While.


Attentive Excitation and Aggregation for Bilingual Referring Image Segmentation

Q Zhou, T Hui, R Wang, H Hu, S Liu - ACM Transactions on Intelligent Systems and …, 2021

… For English expression, we use GloVe1 pretrained on Common Crawl to embed each word into a 300-d vector. For Chinese expression, existing tools … GloVe word embeddings [34] pretrained on Common Crawl 840B …


Augmenting Poetry Composition with Verse by Verse

D Uthus, M Voitovich, RJ Mical - arXiv preprint arXiv:2103.17205, 2021

… TextSETTR was shown to yield better results in transforming sentiment while preserving fluency (important aspects for our work). As described in the TextSETTR paper, we use the model that had been fine-tuned on English Common Crawl data …


Augmenting semantic lexicons using word embeddings and transfer learning

T Alshaabi, C Van Oort, M Fudolig, MV Arnold… - arXiv preprint arXiv …, 2021

… words. We then pass the token embeddings to a 300dimensional embedding layer. We initialize the embedding layer with weights trained with subword information on Common Crawl and Wikipedia using FastText [59]. In …


AUGVIC: Exploiting BiText Vicinity for Low-Resource NMT

T Mohiuddin, MS Bari, S Joty - arXiv preprint arXiv:2106.05141, 2021

… localization guide, respectively. For some languages, the amount of specific domain monolingual data is limited, where we added additional monolingual data of that language from Common Crawl. Following previous work …


Authorship Weightage Algorithm for Academic publications: A new calculation and ACES webserver for determining expertise

WL Wu, O Tan, KF Chan, NB Ong, D Gunasegaran… - Methods and Protocols, 2021

… the back-end server. These word vectors were trained on Common Crawl (https://commoncrawl.org (last accessed on 28 April 2021)) using fastText [17], and are used to map the processed query to its corresponding values …


Page 1. University of Denver Digital Commons @ DU Electronic Theses and Dissertations Graduate Studies 2020 Automated Change Detection in Privacy Policies Andrick Adhikari Follow this and additional works at: https://digitalcommons.du.edu/etd …


Automated essay scoring: A review of the field

P Lagakis, S Demetriadis - … International Conference on Computer, Information and …, 2021

… Transformer models make use of those huge datasets of existing general text data, such as Wikipedia Corpus and Common Crawl, to pretrain multilayer neural networks with context-sensitive meaning of, and relations between, words, such as …


Automated Grading of Exam Responses: An Extensive Classification Benchmark

A Farazouli, Z Lee, P Papapetrou, U Fors - … Science: 24th International Conference, DS 2021 …

J Ljungman, V Lislevand, J Pavlopoulos, A Farazouli… - International Conference on …, 2021

… This method proves that training BERT with alternative design choices and with more data, including the CommonCrawl News dataset, improves the performance on downstream tasks. Open image in new window Fig …


Automated identification of bias inducing words in news articles using linguistic and context-oriented features

T Spinde, L Rudnitckaia, J Mitrović, F Hamborg… - Information Processing & …, 2021

JavaScript is disabled on your browser. Please enable JavaScript to use all the features on this page. Skip to main content Skip to article …


Automated methods for Question-Answering in Icelandic

… The source of the data is the open internet, made accessible to those with relatively modest computing resources and disk storage through the targeted use of the Common Crawl datasets that comprise petabytes of data. Prior work has focused on the …


Automatic Detection of Fake

… For the data, they created the RealNews dataset, a large corpus of news articles from Common Crawl1. Fake News Detection Using Deep Learning Techniques [11] compared Logistic Regression (LR), Naive Bayes (NB) …


Automatic Difficulty Classification of Arabic Sentences

N Khallaf, S Sharoff - arXiv preprint arXiv:2103.04386, 2021

… corpus (Common Crawl and Wikipedia for ArabicBERT vs Common Crawl XML-R vs Wikipedia for BERT, AraBert and UCS) used to train the Arabic … The corpus will be classified on the ba- sis of how difficult the sentences are …


Automatic Fully-Contextualized Recommendation Extraction from Radiology Reports

J Steinkamp, C Chambers, D Lalevic, T Cook - Journal of Digital Imaging, 2021

… We evaluated a simple long short-term memory (LSTM) architecture [12] on the task. We used a combination of custom-trained fastText vectors, trained on our institution's entire repository of radiology reports, with Global …


Page 1. 1 Page 2. 2 Automatic Generic Web Information Extraction at Scale Master Thesis Computer Science, Data Science and Technology University of Twente. Enschede, The Netherlands An attempt to bring some structure …


Automatic Sexism Detection with Multilingual Transformer Models

S Mina, B Jaqueline, L Daria, S Djordje, K Armin… - arXiv preprint arXiv …, 2021

… XLM-R is a multilingual model trained on 100 languages, similar to mBERT. Unlike the latter, XLM-R is not trained on Wikipedia data but on monolingual CommonCrawl data. The model shows improved cross-lingual language …


Automatic Stress Detection from Facial Videos

… , leading to the development of pretrained systems such as BERT(Bidirectional Encoder Representations from Transformers) and GPT (Generative Pre-trained Transformer), which were trained with large language datasets, such as Wikipedia …


Automatically Detecting Cyberbullying Comments on Online Game Forums

HHP Vo, HT Tran, ST Luu - arXiv preprint arXiv:2106.01598, 2021

… on Wikipedia and Gigaword corpora. The fastText 3 is trained on Common Crawl and Wikipedia datasets using CBOW with position-weight in 300 dimensions with 5-grams features. D. Traditional machine learning models Logistic …


Autonomous Writing Futures

AH Duin, I Pedersen - Writing Futures: Collaborative, Algorithmic …, 2021

… GPT-3 is 175 billion parameters. GPT-3 is trained on the Common Crawl data set, a corpus of almost a trillion words of texts scraped from the Web. “The dataset and model size are about two orders of magnitude larger than those used for GPT-2,” the authors write …


Auxiliary Bi-Level Graph Representation for Cross-Modal Image-Text Retrieval

X Zhong, Z Yang, M Ye, W Huang, J Yuan, CW Lin - 2021 IEEE International …, 2021

… The scene graph features Soi and Srij are transformed by a learnable embedding layer which is initialized by GloVe [18] pre-trained on the Common-Crawl dataset, and maps Ioi and Irij into a vector of same dimension: Soi = WoIoi , Srij = WrIrij , (1) …


Auxiliary Learning for Relation Extraction

S Lyu, J Cheng, X Wu, L Cui, H Chen, C Miao - IEEE Transactions on Emerging …, 2020

… 7https://catalog.ldc.upenn.edu/LDC2018T24 8http://semeval2.fbk.eu/semeval2. php?location=data 9Following previous work, we choose GloVe word vectors with 300 dimensions (Common Crawl) https://nlp.stanford.edu/projects/glove …


Background Knowledge in Schema Matching: Strategy vs. Data

J Portisch, M Hladik, H Paulheim - arXiv preprint arXiv:2107.00001, 2021

… used. WebIsALOD is a large hypernymy graph based on the WebIsA database [37]. The latter is a dataset which consists of hypernymy relations extracted from the Common Crawl, a large set of crawled Web pages. The extraction …


Bambara Language Dataset for Sentiment Analysis

M Diallo, C Fourati, H Haddad - arXiv preprint arXiv:2108.02524, 2021

… In this paper, we present the first common-crawl-based Bambara dialectal dataset dedicated for Sentiment Analysis, available freely for Natural Language Processing research purposes … Bambara V1 dataset represents …


Bandits Don't Follow Rules: Balancing Multi-Facet Machine Translation with Multi-Armed Bandits

J Kreutzer, D Vilar, A Sokolov - arXiv preprint arXiv:2110.06997, 2021

Training data for machine translation (MT) is often sourced from a multitude of large corpora that are multi-faceted in nature, eg containing contents from multiple domains or different levels of quality or complexity. Naturally, these facets do not …


BARTpho: Pre-trained Sequence-to-Sequence Models for Vietnamese

NL Tran, DM Le, DQ Nguyen - arXiv preprint arXiv:2109.09701, 2021

… rization. Here, mBART is pre-trained on a Common Crawl dataset of 25 languages, which contains 137 GB of syllablelevel Vietnamese texts. We employ the single-document summarization dataset VNDS (Nguyen et al. 2019 …


belabBERT: a Dutch RoBERTa-based language model applied to psychiatric classification

J Wouts, J de Boer, A Voppel, S Brederoo… - arXiv preprint arXiv …, 2021

… 3.1.1. Pre-training For the pre-training of belabBERT we used the OSCAR corpus which consists of a set of monolingual corpora extracted from Common Crawl snapshots … belabBERT Common Crawl Dutch (non-shuffled) BytePairEncoding 95.92 ∗ …


Benchmarking Differential Privacy and Federated Learning for BERT Models

P Basu, TS Roy, R Naidu, Z Muftuoglu, S Singh… - arXiv preprint arXiv …, 2021

… It uses 160 GB of text for pre-training, including 16GB of Books Corpus and English Wikipedia used in BERT. The additional data included CommonCrawl News dataset, Web text corpus and Stories from Common Crawl. For …


BERT, mBERT, or BiBERT? A Study on Contextualized Embeddings for Neural Machine Translation

H Xu, B Van Durme, K Murray - arXiv preprint arXiv:2109.04588, 2021

… on 145G German text data portion of OSCAR (Or- tiz Suárez et al., 2020), a huge multilingual corpus extracted from Common Crawl … Vaswani et al., 2017) masked language model trained on 100 languages, using more than …


MV Koroteev - arXiv preprint arXiv:2103.11943, 2021

Page 1. BERT: A Review of Applications in Natural Language Processing and Understanding Koroteev MV, Financial University under the government of the Russian Federation, Moscow, Russia mvkoroteev@fa.ru Abstract: In …


Bertinho: Galician BERT Representations

D Vilares, M Garcia, C Gómez-Rodríguez - arXiv preprint arXiv:2103.13799, 2021

Page 1. Bertinho: Galician BERT Representations Bertinho: Representaciones BERT para el gallego David Vilares,1 Marcos Garcia,2 Carlos Gómez-Rodr´ıguez 1 1Universidade da Coru˜na, CITIC, Galicia, Spain 2CiTIUS, Universidade …


Better Neural Machine Translation by Extracting Linguistic Information from BERT

HS Shavarani, A Sarkar - arXiv preprint arXiv:2104.02831, 2021

… clack, clack.”). 9Europarl+CommonCrawl+NewsCommentary https://www.statmt. org/wmt14/translation-task.html, please note that in the later years this training set remained the same, but ParaCrawl data was added to it. We …


E Briakou, M Carpuat - arXiv preprint arXiv:2105.15087, 2021

Page 1. Beyond Noise: Mitigating the Impact of Fine-grained Semantic Divergences on Neural Machine Translation Eleftheria Briakou and Marine Carpuat Department of Computer Science University of Maryland College Park …


Beyond the English Web: Zero-Shot Cross-Lingual and Lightweight Monolingual Classification of Registers

L Repo, V Skantsi, S Rönnqvist, S Hellström… - arXiv preprint arXiv …, 2021

… FreCORE and SweCORE are random samples of the 2017 CoNLL datasets (Ginter et al., 2017) originally drawn from Common Crawl … XLM-R is trained on 2.5TB of filtered Common Crawl (Wenzek et al., 2020) data comprising …


Bias Silhouette Analysis: Towards Assessing the Quality of Bias Metrics for Word Embedding Models

… Word Embedding Models. As biased and unbiased models, we use GloVe CommonCrawl [Pennington et al., 2014] trained on 840 billion English tokens and the English ConceptNet Numberbatch 19.08 [Speer et al., 2017] (referred to as NBatch below), respectively …


M Shah Jahan, HU Khan, S Akbar, M Umar Farooq… - Scientific Programming, 2021

Page 1. Review Article Bidirectional Language Modeling: A Systematic Literature Review Muhammad Shah Jahan ,1 Habib Ullah Khan ,2 Shahzad Akbar ,3 Muhammad Umar Farooq ,1 Sarah Gul ,4 and Anam Amjad 1 1Department …


Bilingual Lexical Induction for Sinhala-English using Cross Lingual Embedding Spaces

A Liyanage, S Ranathunga, S Jayasena - 2021 Moratuwa Engineering Research …, 2021

… Using pre-trained fastText embeddings trained on Wikipedia and Common crawl data using two different evaluation dictionaries as a preliminary experiment to identify the performance of embeddings created from non-comparable corpora …


H Shi, L Zettlemoyer, SI Wang - arXiv preprint arXiv:2101.00148

Page 1. Bilingual Lexicon Induction via Unsupervised Bitext Construction and Word Alignment Haoyue Shi ∗ TTI-Chicago freda@ttic.edu Luke Zettlemoyer University of Washington Facebook AI Research lsz@fb.com …


BitextEdit: Automatic Bitext Editing for Improved Low-Resource Machine Translation

E Briakou, SI Wang, L Zettlemoyer, M Ghazvininejad - arXiv preprint arXiv …, 2021

Mined bitexts can contain imperfect translations that yield unreliable training signals for Neural Machine Translation (NMT). While filtering such pairs out is known to improve final model quality, we argue that it is suboptimal in low-resource conditions …


Blank spots, critical information needs and local journalism fund-ing

Abstract A global business model crisis in journalism, fuelled by loss in advertising revenue, challenges the survival of local news production. In Sweden, it has led to the closure of several newspapers across the country, and the concentration of …


Book genre and author's gender recognition based on titles

A Pawłowski, E Herden, T Walkowiak - … and Text: Data, models, information and …, 2021


J Heo, SY Lee, S Min, Y Park, SJ Jung, TJ Ham…

Page 1. BOSS: Bandwidth-Optimized Search Accelerator for Storage-Class Memory Jun Heo, Seung Yul Lee, Sunhong Min, Yeonhong Park, Sung Jun Jung, Tae Jun Ham, Jae W. Lee Seoul National University {j.heo, triomphant1 …


S Yang, M Xia, G Li, HY Zhou, Y Yu - Proceedings of the IEEE/CVF Conference on …, 2021

Page 1. Bottom-Up Shift and Reasoning for Referring Image Segmentation Sibei Yang1∗† Meng Xia2∗ Guanbin Li2 Hong-Yu Zhou3 Yizhou Yu3,4† 1ShanghaiTech University 2Sun Yat-sen University 3The University of Hong Kong 4Deepwise AI Lab Abstract …


bradleypallen/keras-quora-question-pairs

… Model, Source of Word Embeddings, Accuracy. "BiMPM model" [5], GloVe Common Crawl (840B tokens, 300D), 0.88 … "Decomposable attention" [6], "Quora's text corpus", 0.86. "LDC" [5], GloVe Common Crawl (840B tokens, 300D), 0.86 …


Building a File Observatory for Secure Parser Development

T Allison, W Burke, C Mattmann, A Mensikova…

… 3196–3200. [Online]. Available: http://www.lrec-conf.org/proceedings/lrec2012/pdf/534 Paper.pdf [10] “Common Crawl,” https://commoncrawl.org. [11] P. Wyatt, “Stressful PDF corpus grows!” https://www.pdfa.org/ stressful-pdf-corpus-grows/, November 2020.


Building a Question and Answer System for News Domain

S Basu, A Gaddala, P Chetan, G Tiwari, N Darapaneni… - arXiv preprint arXiv …, 2021

… We have used two approaches for building the Embedding Layers for the models 3. GloVe Embedding: we used the 300 Dimension Common Crawl for the English language 4. Universal Sentence Encoder: we used the 512 …


Building Accountable Natural Language Processing Models: on Social Bias Detection and Mitigation

Natural Language Processing (NLP) plays an important role in many applications, including resume filtering, text analysis, and information retrieval. Despite the remarkable accuracy enabled by the advances of machine learning methods, recent …


But how robust is RoBERTa actually?: A Benchmark of SOTA Transformer Networks for Sexual Harassment Detection on Twitter

P Basu, TS Roy, A Singhal - 2021 Fifth International Conference on I-SMAC (IoT in …, 2021

Harassment, which is of sexual/physical in nature, is defined as any unwanted sexual misconduct, including the unwarranted and ill-suited promise of benefit in exchange for sexual indulgence. It also includes a span of actions from verbal …


Page 1. Can I Take Your Subdomain? Exploring Same-Site Attacks in the Modern Web Marco Squarcina1 Mauro Tempesta1 Lorenzo Veronese1 Stefano Calzavara2 Matteo Maffei1 1 TU Wien 2 Università Ca' Foscari Venezia & OWASP …


Can Language Models Encode Perceptual Structure Without Grounding? A Case Study in Color

M Abdou, A Kulmizev, D Hershcovich, S Frank… - arXiv preprint arXiv …, 2021

… Word-type FastText embeddings trained on Common Crawl (Bojanowski et al., 2017) … These S contexts are either randomly sampled from common crawl (RC), or deterministically generated to allow for control over contextual variation (CC) …


NF Liu, T Lee, R Jia, P Liang - arXiv preprint arXiv:2102.01065, 2021

Page 1. Can Small and Synthetic Benchmarks Drive Modeling Innovation? A Retrospective Study of Question Answering Modeling Approaches Nelson F. Liu Tony Lee Robin Jia Percy Liang Computer Science Department, Stanford …


CausalBERT: Injecting Causal Knowledge Into Pre-trained Models with Minimal Supervision

Z Li, X Ding, K Liao, T Liu, B Qin - arXiv preprint arXiv:2107.09852, 2021

… ambiguity and precise causal patterns to extract word level causeeffect pairs from the preprocessed English Common Crawl corpus (5.14 … (2016) for creating a causal lexical knowledge base, we reproduce a variant of their …


CCQA: A New Web-Scale Question Answering Dataset for Model Pre-Training

P Huber, A Aghajanyan, B Oğuz, D Okhonko, W Yih… - arXiv preprint arXiv …, 2021

… Consequently, we propose a novel QA dataset based on the Common Crawl project in this paper. Using the readily available schema.org annotation, we extract around 130 million multilingual question-answer pairs, including about 60 million …


CDA: a Cost Efficient Content-based Multilingual Web Document Aligner

… CommonCrawl Sextet Previous datasets share the same domains that are heavily biased toward French content (see Table 3). We leverage a monthly crawl from CommonCrawl, specifically … Table 4: Parallel English tokens …


Censorship of Online Encyclopedias: Implications for NLP Models

E Yang, ME Roberts - arXiv preprint arXiv:2101.09294, 2021

… Word embeddings are also useful because they can be pre-trained on large corpuses of text like Wikipedia or Common Crawl, and these pre-trained embeddings can then be used as an initial layer in applications that may have less training data …


Challenges for cognitive decoding using deep learning methods

AW Thomas, C Ré, RA Poldrack - arXiv preprint arXiv:2108.06896, 2021

… learning in 251 the target domain. Transfer learning has been especially successful in CV and NLP, where large 252 publicly available datasets exist (eg, [72,73] and http://www.commoncrawl.org). Here, DL 253 models are first …


Changing the World by Changing the Data

A Rogers - arXiv preprint arXiv:2105.13947, 2021

… The use of uncontrolled samples (like the Common-Crawl-based corpora) would have to be justified by arguing either that the above types of bias can be safely ignored, or that the benefits outweigh the risks. 2.2.3 Might not be the best approach …


Characterizing and addressing the issue of oversmoothing in neural autoregressive sequence modeling

I Kulikov, M Eremeev, K Cho - arXiv preprint arXiv:2112.08914, 2021

… We use the subset of WMT’19 training set consisting of news commentary v12 and common crawl resulting in slightly more than 1M and 2M training sentence pairs for Ru→En and De↔En pairs, respectively. We fine-tuned single model checkpoints …


Characterizing Network Infrastructure Using the Domain Name System

Page 1. CHARACTERIZING NETWORK INFRASTRUCTURE USING THE DOMAIN NAME SYSTEM A Dissertation Presented to The Academic Faculty By Panagiotis Kintis In Partial Fulfillment of the Requirements for the Degree …


Charformer: Fast Character Transformers via Gradient-based Subword Tokenization

Y Tay, VQ Tran, S Ruder, J Gupta, HW Chung, D Bahri… - arXiv preprint arXiv …, 2021

… In addition, we compare to the byte-level models from §3.1, which we pre-train on multilingual data. Setup We pre-train CHARFORMER as well as the Byte-level T5 and Byte-level T5+LASC baselines on multilingual …


ChineseBERT: Chinese Pretraining Enhanced by Glyph and Pinyin Information

Z Sun, X Li, X Sun, Y Meng, X Ao, Q He, F Wu, J Li - arXiv preprint arXiv:2106.16038, 2021

… We collected our pretraining data from CommonCrawl5. After pre-processing (such as removing the data with too much … ERNIE BERT-wwm MacBERT ChineseBERT Data Source Heterogeneous Wikipedia Heterogeneous …


Claim Detection in Biomedical Twitter Posts

A Wührl, R Klinger - arXiv preprint arXiv:2104.11639, 2021

… The first three models (NB, LG, BiLSTM) use 50-dimensional FastText (Bojanowski et al., 2017) embeddings trained on the Common Crawl corpus (600 billion tokens) as input6. NB. We use a (Gaussian) naive Bayes with …


Classification of Emotions Based on Text and Qualitative Variables

… Experiments were done both with Common Crawl GloVe pretrained vectors with the dimensionality of 300, and without pretrained vectors. Fifteen percent of learning samples were used for validation. We constructed six neural networks models: CNN …


Classification of Horror Stories from Reddit

… We further hypothesize that performance would still increase a little if we used the larger pre-trained vectors such as Common Crawl or Twitter sets, but they come with increased download sizes (>1 GB) and increased training time …


Classification of Texts Using a Vocabulary of Synonyms

A Giliazova - 2021 14th International Conference Management of …, 2021

… This is a Transformer-based masked language model trained on one hundred languages, including Russian language, using more than two terabytes of filtered CommonCrawl data. The XLM-R model significantly outperforms multilingual BERT (mBERT) …


Page 1. CLASSIFICATION OF TWEETS USING MULTIPLE THRESHOLDS WITH SELF-CORRECTION AND WEIGHTED CONDITIONAL PROBABILITIES A thesis submitted to The University of Manchester for the degree of Doctor of Philosophy …


Classification-based Quality Estimation: Small and Efficient Models for Real-world Applications

S Sun, A El-Kishky, V Chaudhary, J Cross, F Guzmán… - arXiv preprint arXiv …, 2021

… Current state of the art QE systems (Fomicheva et al., 2020b; Ranasinghe et al., 2020a; Sun et al., 2020). are built on XLM-R (Conneau et al., 2019), a contextualized language model pre-trained on more than 2 terabytes of …


Classifying Fake and Real Neurally Generated News

A Govindaraju, J Griffith - 2021 Swedish Workshop on Data Science (SweDS), 2021

… In order to train and test the model, 3 datasets have been created: One containing real news extracted from a common crawl; the second comprises a neural fake news dataset generated using language modelling techniques; the third comprises a …


CLEF eHealth Evaluation Lab 2021

L Kelly, LA Alemany, N Brew-Sam, V Cotik, D Filippo…

… This collection consists of Web pages acquired from Common Crawl,14 which is augmented with additional pages collected from a number of known reliable health Websites and other known unreliable health Websites [9]. The topics …


Click This, Not That: Extending Web Authentication with Deception

T Barron, J So, N Nikiforakis - Proceedings of the 2021 ACM Asia Conference on …, 2021

… after creation. References. 2020. Common Crawl. https://commoncrawl.org/the-data/ get-started/Google Scholar Google Scholar; 2020. Mouseflow: Session Replay, Heatmaps, Funnels, Forms & User Feedback. https://mouseflow …


ClimateBert: A Pretrained Language Model for Climate-Related Text

N Webersinke, M Kraus, JA Bingler, M Leippold - arXiv preprint arXiv:2110.12010, 2021

… 2019), and a subset of CommonCrawl that is said to resemble the storylike style of WINOGRAD schemas (Trinh and Le, 2019). While these sources are valuable to build a model working on general language, it has been shown that domain-specific …


CLIP2StyleGAN: Unsupervised Extraction of StyleGAN Edit Directions

R Abdal, P Zhu, J Femiani, NJ Mitra, P Wonka - arXiv preprint arXiv:2112.05219, 2021

… The CLIP image encoder [34] is trained on the common-crawl dataset, an internet-scale set of images that encompasses a broad range of visual concepts. However, a typical high-quality GAN would be trained on a more specific set of images, for example …


Cluster analysis of agricultural household production of self-employed

AV Plotnikov - IOP Conference Series: Earth and Environmental …, 2021

… To train this model, we used a sample of Russian-language documents from the CommonCrawl dump, balanced by geography, compiled by Jonathan Dunn and Ben Adams; the corpus Size is 2.1 billion words. Page 5. AGRITECH-IV-2020 IOP Conf …


Cluster-Based Antiphishing (CAP) Model for Smart Phones

M Faisal, S Abed - Scientific Programming

… latest techniques tested on UCI datasets. 4.4.2. Dataset Taken from Mendeley. Source Phishing web page: Phish Tank, Legitimate web page source: Alexa, Common Crawl (1) Dataset Information. In this scenario, the dataset …


Code-Mixing on Sesame Street: Dawn of the Adversarial Polyglots

S Tan, S Joty - arXiv preprint arXiv:2103.09593, 2021

… However, the latter trend is replicated for BUMBLEBEE if we remove this constraint (Table 14 in Appendix G). A possible explanation is that XLM-R and Unicoder were trained on monolingual CommonCrawl (CC) data, while …


CoDesc: A Large Code–Description Parallel Dataset

M Hasan, T Muttaqueen, A Al Ishtiaq, KS Mehrab…

… 9052–9065, Online. Association for Computational Linguistics. CommonCrawl Common crawl. https:// commoncrawl.org/. Accessed: 2021-01-31. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019 …


Page 1. PHD PROGRAM IN SMART COMPUTING DIPARTIMENTO DI INGEGNERIA DELL'INFORMAZIONE (DINFO) Combining Natural Language Processing and Machine Learning for Profiling and Fake News Detection Alessandro Bondielli …


Combining Pre-trained Word Embeddings and Linguistic Features for Sequential Metaphor Identification

R Mao, C Lin, F Guerin - arXiv preprint arXiv:2104.03285, 2021

… For instance, GloVe was trained on Common Crawl2, from billions of web pages (840 billion tokens); ELMo was trained on WMT 2011 News Crawl data3 (800 million tokens); BERT was trained on Wikipedia4 (2.5 billion tokens) …


Combining word embeddings as a tool for subject identification

A Hamm - Wissensaustauschworkshop" Maschinelles Lernen VII", 2021

This talk shows ungoing work aiming at finding subject matter relations between text documents and word clouds. A number of increasingly successful semantic word embedding procedures - learning semantic relations from contextual distributions …


Comparative Analysis of Bengali Stop Word Detection Using Different Approaches

RJ Rupa, JF Sohana, M Rahman - … on Automation, Control and Mechatronics for …, 2021

… In this paper, the pretrained Bengali FastText CBOW model is utilized to produce word vectors trained on the common crawl and Wikipedia [27] and both logistic regression and support vector machine classifiers acquire a performance score of 86%. TABLE XII …


Comparative Analysis of Different Transformer Based Architectures Used in Sentiment Analysis

K Pipalia, R Bhadja, M Shukla - 2020 9th International Conference System Modeling …, 2020

… Distill BERT Base:66 BookCorpus wiki BERT Distillation T5 Base:220 large:770 Colossal Clean Crawled Corpus (C4) Text Infilling XLNet Base:~110 Large:~340 BookCorpus Wiki, Giga5 ClueWeb, Common Crawl …


Comparing Apples and Oranges: Human and Computer Clustered Affinity Diagrams Under the Microscope

P Borlinghaus, S Huber - 26th International Conference on Intelligent User …, 2021

… training corpora WSD no OOV LSI [9] − NMF [18] − LDA [5] − GloVe [26] Wiki word2vec [22] Google News corpus doc2vec [17] 900k sentences from qualitative survey ◦ fastText [6] Common Crawl, Wiki • … FastText was trained on Common Crawl and Wikipedia corpus …


Comparing Contextualised Embeddings for Predicting the (Graded) Effect of Context in Word Similarity

… As data set XLM-RoBERTa uses CommonCrawl instead of Wikipedia, which provides limited scale for low resource languages. 4 Page 5 … The CommonCrawl data set is designed to be more diverse than other data sets, which mainly use Wikipedia and books …


Comparing Encoder-Decoder Architectures for Neural Machine Translation: A Challenge Set Approach

Machine translation (MT) as a field of research has known significant advances in recent years, with the increased interest for neural machine translation (NMT). By combining deep learning with translation, researchers have been able to deliver …


Comparing general and specialized word embeddings for biomedical named entity recognition

RE Ramos-Vargas, I Román-Godínez, S Torres-Ramos - PeerJ Computer Science, 2021

… 01-14 Received 2020-11-05 Academic Editor Susan Gauch Subject Areas Bioinformatics, Artificial Intelligence, Computational Linguistics Keywords Word embeddings, BioNER, BiLSTM-CRF, DrugBank, MedLine, Pyysalo …


MZ Khan, J Mitrovic, JMPDM Granitzer - 2021

Page 1. Lehrstuhl für Data Science Comparing the Performance of NLP Toolkits and Evaluation measures in Legal Tech Masterarbeit von Muhammad Zohaib Khan Supervised By: Prof. Dr. Jelena Mitrovic 1. Prüfer 2. Prüfer …


Comparing Traditional and Neural Approaches for Detecting Health-Related Misinformation

D Elsweiler - … IR Meets Multilinguality, Multimodality, and Interaction …

M Fernández-Pichel, DE Losada, JC Pichel…

… Table 1 reports the main statistics of the resulting datasets. We also tested classifiers for the task of distinguishing between useful documents for non-expert end users (ie, trustworthy and readable) and non-useful …


Comparison of Czech Transformers on Text Classification Tasks

J Lehečka, J Švec - arXiv preprint arXiv:2107.10042, 2021

… Researchers from Facebook have published multilingual XLM-RoBERTa model [3] pre-trained on one hundred languages (including Czech), using more than two terabytes of filtered Common Crawl data … 2. 1https …


Compilation and Validation of a Large Fake News Dataset in Hungarian

M Gencsi, Z Bodó, A Szenkovits - 2021 IEEE 19th International Symposium on …, 2021

… The huBERT model was trained on the Hungarian subset of the Common Crawl and a snapshot of the Hungarian Wikipedia, while the multilingual model was trained on the top 104 languages with the largest Wikipedias, among them also …


Comprehensive analysis of embeddings and pre-training in NLP

JK Tripathy, SC Sethuraman, MV Cruz, A Namburu… - Computer Science Review, 2021

JavaScript is disabled on your browser. Please enable JavaScript to use all the features on this page. Skip to main content Skip to article …


Comprehensive Evaluation of Word Embeddings for Highly Inflectional Language

P Drozda, K Sopyla, J Lewalski - International Conference on Computational …, 2021

… The obtained results showed that in terms of accuracy the Facebook fasttext model learned on the Common Crawl collection should be considered the best model under assumptions of experimental session. Keywords. Word …


Comprehensive Multi-Modal Interactions for Referring Image Segmentation

K Jain, V Gandhi - arXiv preprint arXiv:2104.10412, 2021

… 576. At 448 × 448 resolution, H = W = 14 and at 576 × 576 resolution, H = W = 18. We use GLoVe embeddings [17] pre-trained on Common Crawl 840B tokens to initialize word embedding for words in the expressions. The …


M Deng, B Tan, Z Liu, EP Xing, Z Hu - arXiv preprint arXiv:2109.06379, 2021

Page 1. Compression, Transduction, and Creation: A Unified Framework for Evaluating Natural Language Generation Mingkai Deng1∗, Bowen Tan1∗, Zhengzhong Liu1,2, Eric P. Xing1,2,3, Zhiting Hu4 1Carnegie Mellon University …


Computational analysis and synthesis of song lyrics

… It uses CMUdict for phonetic transcription, analyzes CommonCrawl8 web data repository for forced rhymes, Google Books Ngrams (Weiss [2015]) for building language model, and WordNet 3.0 (Pearson et al. [2005]) for semantic relations …


Computational Challenges for Artificial Intelligence and Machine Learning in Environmental Research

M Werner, G Dax, M Laass - INFORMATIK 2020, 2021

… This includes news streams, social media messages, human-curated knowledge such as OpenStreetMap and Wikipedia, opinionated data sources such as blog posts from certain platforms, or blind web scale data collections such as common crawl …


Computational filling of curatorial gaps in a fine arts exhibition

… Please note that we translate all keywords from German to English for this paper. We use the German fasttext5 word em- bedding, which has been trained on about 3 million words from the Wikipediaand 19 million words …


Computational methods to understand the association between emojis and emotions

Page 1. © 2021 Abu Awal Md Shoeb ALL RIGHTS RESERVED Page 2. COMPUTATIONAL METHODS TO UNDERSTAND THE ASSOCIATION BETWEEN EMOJIS AND EMOTIONS By ABU AWAL MD SHOEB A dissertation submitted to the School of Graduate Studies …


Computer Science Review

JK Tripathy, SC Sethuraman, MV Cruz, V Vijayakumar - 2021

abstract The amount of data and computing power has drastically increased over the last decade, which leads to the development of several new fronts in the field of Natural Language Processing (NLP). In addition to that, the entanglement of …


CoMSum and SIBERT: A Dataset and Neural Model for Query-Based Multi-document Summarization

S Kulkarni, S Chammas, W Zhu, F Sha, E Ie - International Conference on Document …, 2021

… We use the cleaned Common Crawl (CC) corpus [32] to source relevant web documents that are diverse and multi-faceted for generating Natural Questions (NQ) (long-form) answers [21]. Figure 1 illustrates the overall procedure …


Concept-Based Label Embedding via Dynamic Routing for Hierarchical Text Classification

X Wang, L Zhao, B Liu, T Chen, F Zhang, D Wang

… Hyper-parameters are tuned on a validation set by grid search. We take Stanford's publicly available GloVe 300-dimensional embeddings trained on 42 billion tokens from Common Crawl (Pennington et al., 2014) as initialization for word em- beddings …


Confused by Path: Analysis of Path Confusion Based Attacks

… 93 iii Page 10. Page 11. List of Tables 4.1 Sample Grouped Web pages. . . . . 29 4.2 Narrowing down the Common Crawl to the candidate set used in our analysis (from left to right). . . . 36 4.3 Vulnerable pages and sites in the candidate set …


ConRPG: Paraphrase Generation using Contexts as Regularizer

Y Meng, X Ao, Q He, X Sun, Q Han, F Wu, J Li - arXiv preprint arXiv:2109.00363, 2021

… We implement the above models, ie p(−→ci|c<i,c>i), p(←−ci|c<i,c>i), p(c<i|ci,c>i), p(c>i|c<i,ci) based on the SEQ2SEQ structure on a subset of CommonCrawl containing 10 billion tokens in total. We use Transformers …


Consumer Health Search at CLEF eHealth 2021

L Goeuriot, G Pasi, H Suominen, E Bassani… - CLEF 2021 Evaluation Labs …, 2021

… The task generated a new representative web corpus including web pages acquired from a 2021 CommonCrawl and social media content from Twitter and Reddit, along with a new collection of 55 manually generated layperson …


Contingency of semantic generalization on episodic specificity varies across development

CT Ngo, SL Benear, H Popal, IR Olson, NS Newcombe - Current Biology, 2021


Contrastive finetuning of generative language models for informal premise selection

JM Han, T Xu, S Polu, A Neelakantan, A Radford

… contrastive training. • GPT-3 style pretraining. The model is pretrained for 300B tokens on the same data (a mix of filtered CommonCrawl, WebText, books, and Wikipedia) as GPT-3 [2]. • WebMath pretraining. Starting from the …


Control Prefixes for Text Generation

J Clive, K Cao, M Rei - arXiv preprint arXiv:2110.08329, 2021

Prompt learning methods adapt pre-trained language models to downstream applications by using a task-specific prompt together with the input. Most of the current work on prompt learning in text generation relies on a shared dataset-level …


W Sun, C Meng, Q Meng, Z Ren, P Ren, Z Chen… - 2021

Page 1. Conversations Powered by Cross-Lingual Knowledge Weiwei Sun1* Chuan Meng1* Qi Meng2 Zhaochun Ren1† Pengjie Ren1† Zhumin Chen1 Maarten de Rijke3,4 1Shandong University, Qingdao, China 2Microsoft …


CopyCat: Near-Duplicates within and between the ClueWeb and the Common Crawl

M Fröbe, J Bevendorff, L Gienapp, M Völske, B Stein… - 2021

The amount of near-duplicates in web crawls like the ClueWeb or Common Crawl demands from their users either to develop a preprocessing pipeline for deduplication, which is costly both computationally and in person hours, or accepting …


Corpulyzer: A Novel Framework for Building Low Resource Language Corpora

… Leveraging dataset from Common Crawl Corpus (CCC), first, we prepare a list of seed URLs by filtering the Urdu language webpages … INDEX TERMS Common crawl, web crawling, text corpus, corpus analysis, regional languages corpora …


Corpus and Baseline Model for Domain-Specific Entity Recognition in German

S Torge, W Hahn, R Jäkel, WE Nagel - 2020 6th IEEE Congress on Information …, 2021

… Deepset AI4 FastText Wikipedia 100 Deepset AI GloVe Wikipedia 300 FastText [26] FastText Wikipedia, 300 Common Crawl Kyubyong5 Word2Vec Wikipedia 300 SB Tweets s[27] FastText 50 Mio Tweets 100 SB Tweets l[27] FastText 50 Mio Tweets 300 …


COVIDSenti: A Large-Scale Benchmark Twitter Data Set for COVID-19 Sentiment Analysis

U Naseem, I Razzak, M Khushi, PW Eklund, J Kim - IEEE Transactions on …, 2021

… Term frequency-inverse document frequency (TF-IDF) has been used for vectorization. Similarly, for word embeddings, pretrained Word2Vec, GloVe, and fastText embeddings trained on Common Crawl and Wikipedia are used and have 300-D vectors …


CRASS: A Novel Data Set and Benchmark to Test Counterfactual Reasoning of Large Language Models

J Frohberg, F Binder - arXiv preprint arXiv:2112.11941, 2021

We introduce the CRASS (counterfactual reasoning assessment) data set and benchmark utilizing questionized counterfactual conditionals as a novel and powerful tool to evaluate large language models. We present the data set design …


Creation, Enrichment and Application of Knowledge Graphs

Page 1. CREATION, ENRICHMENT AND APPLICATION OF KNOWLEDGE GRAPHS Von der Fakultät für Elektrotechnik und Informatik der Gottfried Wilhelm Leibniz Universität Hannover zur Erlangung des Grades DOKTOR …


CrisisBench: Benchmarking Crisis-related Social Media Datasets for Humanitarian Information Processing

F Alam, H Sajjad, M Imran, F Ofli - 2021

… 2017). fastText: For the fastText (Joulin et al. 2017), we used pretrained embeddings trained on Common Crawl, which is re- leased by fastText for English. Transformer models: Pre-trained models have achieved …


Cross-Domain Transfer of Generative Explanations Using Text-to-Text Models

KF Erliksson, A Arpteg, M Matskin, AH Payberah - International Conference on …, 2021

… An analysis of BERT'S attention. In: ACL Blackbox NLP Workshop (2019)Google Scholar. 7. Common Crawl. https://www.commoncrawl.org. 8. Danilevsky, M., et al.: A survey of the state of explainable AI for natural language processing …


Cross-language Information Retrieval

P Galuščáková, DW Oard, S Nair - arXiv preprint arXiv:2111.05988, 2021

Two key assumptions shape the usual view of ranked retrieval: (1) that the searcher can choose words for their query that might appear in the documents that they wish to see, and (2) that ranking retrieved documents will suffice because the searcher …


Cross-Language Learning for Entity Matching

R Peeters, C Bizer - arXiv preprint arXiv:2110.03338, 2021

… We further use a German BERT model pre-trained on a di- verse set of German data ('bert-base-german-dbmdz-uncased'), including the German Wikipedia, parts of the Common Crawl, and EU Bookshop corpus, which …


Cross-lingual Capsule Network for Hate Speech Detection in Social Media

A Jiang, A Zubiaga - arXiv preprint arXiv:2108.03089, 2021

… mBERT. Multilingual BERT6 (mBERT) is a variant of BERT [14] that was trained on 104 languages of Wikipedia; XLM-R. XLM-RoBERTa is a scaled cross-lingual sentence en- coder across 100 languages from Common Crawl [11]; …


Cross-lingual Fine-tuning for Abstractive Arabic Text Summarization

… We only kept ar- ticles from the “Main/Top Stories” section, and filtered out all articles where either the main article text or the lead was too short or missing and items where the text is shorter than 4 times the 2https://www …


Cross-lingual Hate Speech Detection using Transformer Models

T Tiţa, A Zubiaga - arXiv preprint arXiv:2111.00981, 2021

Hate speech detection within a cross-lingual setting represents a paramount area of interest for all medium and large-scale online platforms. Failing to properly address this issue on a global scale has already led over time to morally questionable real-life …


Cross-lingual Language Model Pretraining for Retrieval

P Yu, H Fei, P Li - Proceedings of the Web Conference 2021, 2021

… (MLM) and next sentence prediction (NSP) objectives [13] on the top 102 languages with the largest Wikipedia dumps. (v) XLM-R. We use the public checkpoint9 [8]. It was originally pretrained with the MLM objective on the CommonCrawl corpus in 100 languages …


Cross-Lingual Text Classification of Transliterated Hindi and Malayalam

J Krishnan, A Anastasopoulos, H Purohit, H Rangwala - arXiv preprint arXiv …, 2021

… The improvement produced by our model is due to the fact that the pre-trained models have likely not seen that many transliterations in these languages. mBERT is trained using data from Wikipedia, while XLM-R uses Common Crawl …


Cross-lingual Transfer for Text Classification with Dictionary-based Heterogeneous Graph

N Chairatanakul, N Sriwatanasakdi… - arXiv preprint arXiv …, 2021

… FastText (Bojanowski et al., 2017): a taskindependent word embedding trained on Wikipedia and Common Crawl only of a target language. • RCSLS (Joulin et al., 2018): a CLWE trained on Wikipedia. We used public FastText4 version …


Cross-lingual transfer of abstractive summarizer to less-resource language

A Žagar, M Robnik-Šikonja - Journal of Intelligent Information Systems, 2021

Automatic text summarization extracts important information from texts and presents the information in the form of a summary. Abstractive summarization app.


Z Li, K Parnow, H Zhao, Z Zhang, R Wang, M Utiyama… - arXiv preprint arXiv …, 2021

Page 1. ARXIV, DECEMBER 2020 1 Cross-lingual Transferring of Pre-trained Contextualized Language Models Zuchao Li, Kevin Parnow, Hai Zhao, Zhuosheng Zhang, Rui Wang, Masao Utiyama, and Eiichiro Sumita Abstract …


Cross-Lingual Word Embedding Refinement by $\ell_ {1} $ Norm Optimisation

X Peng, C Lin, M Stevenson - arXiv preprint arXiv:2104.04916, 2021

… Tibetan family). News-Embs (Artetxe et al., 2018a): embeddings trained on a multilingual News text collection, ie, Page 5. the WaCKy Crawl of {EN, DE, IT}, the Common Crawl of FI, and the WMT News Crawl of ES. News-Embs …


Cross-Lingual Word Embedding Refinement by ℓ1 Norm Optimisation Norm Optimisation

X Peng, C Lin, M Stevenson - Proceedings of the 2021 Conference of the North …, 2021

… family). News-Embs (Artetxe et al., 2018a): embeddings trained on a multilingual News text collection, ie, Page 5. 2694 the WaCKy Crawl of {EN, DE, IT}, the Common Crawl of FI, and the WMT News Crawl of ES. News-Embs …


Cross-lingual word embedding refinement by ℓ1 norm optimisation

X Peng, C Lin, M Stevenson - arXiv, 2021

… Tibetan family). News-Embs (Artetxe et al., 2018a): embeddings trained on a multilingual News text collection, ie, Page 6. the WaCKy Crawl of {EN, DE, IT}, the Common Crawl of FI, and the WMT News Crawl of ES. News-Embs …


Cross-Modal Progressive Comprehension for Referring Segmentation

S Liu, T Hui, S Huang, Y Wei, B Li, G Li - IEEE Transactions on Pattern Analysis and …, 2021

Page 1. 0162-8828 (c) 2021 IEEE. Personal use is permitted, but republication/ redistribution requires IEEE permission. See http://www.ieee.org/ publications_standards/publications/rights/index.html for more information. This …


CTNet: Conversational Transformer Network for Emotion Recognition

Z Lian, B Liu, J Tao - IEEE/ACM Transactions on Audio, Speech, and …, 2021

… utterances. Specifically, we convert transcripts into the concatenation of vectors of the constituent words. These vectors are the public available word vectors trained on the Common Crawl and Wikipedia datasets [44]. Totally …


CUNI systems for WMT21: Terminology translation Shared Task

J Jon, M Novák, JP Aires, D Variš, O Bojar - arXiv preprint arXiv:2109.09350, 2021

… 3.1 Data We used all English-French corpora allowed by the organizers, aside from Paracrawl (with the exception of one model, which is marked). Namely this means Europarl v10, Common Crawl, UN Parallel Corpus …


Current Advances in Neural Networks

Abstract This article reviews current advances and developments in neural networks. This requires recalling some of the earlier work in the field. We emphasize Bayesian approaches and their benefits compared to more standard maximum likelihood …


Data Augmentation for Speech Recognition in Maltese: A Low-Resource Perspective

C Mena, A DeMarco, C Borg, L van der Plas, A Gatt - arXiv preprint arXiv:2111.07793, 2021

Developing speech technologies is a challenge for low-resource languages for which both annotated and raw speech data is sparse. Maltese is one such language. Recent years have seen an increased interest in the computational processing of …


Data augmentation impact on domain-specific text summarization

… Association for Computational Linguistics, Seattle / Virtual, United States (2020) 4. Lin C., Text Summarization Branches Out, 74–81. Association for Computational Linguistics, Barcelona, Spain (2004) 5. Common Crawl Foundation, https://commoncrawl.org/


Page 1. Universidad de la República PEDECIBA Informática Master Thesis Data Efficient Deep Learning Models for Text Classification Author: Raúl Garreta Supervisors …


Data Filtering using Cross-Lingual Word Embeddings

C Herold, J Rosendahl, J Vanvinckenroye, H Ney - … of the 2021 Conference of the …, 2021

… The De→En training data consists of the corpora Commoncrawl, Europarl, Rapid and ParaCrawl from the WMT 2019 news translation task2. We use the czeng 1.7 corpus3 from the WMT 2018 news translation task for En→Cs …


Data science with Vadalog: Knowledge Graphs with machine learning and reasoning in practice

L Bellomarini, RR Fayzrakhmanov, G Gottlob… - Future Generation …, 2021

… Such resources originate from data streams collected from internal systems (eg, Enterprise Resource Planning, Workflow Management, and Supply Chain Management) and external resources (eg, news, social media feeds and the Common Crawl 1 ). …


Dataset for Identification of Homophobia and Transophobia in Multilingual YouTube Comments

BR Chakravarthi, R Priyadharshini, R Ponnusamy… - arXiv preprint arXiv …, 2021

Page 1. Abstract The increased proliferation of abusive content on social media platforms has a negative impact on online users. The dread, dislike, discomfort, or mistrust of lesbian, gay, transgender or bisexual persons is de- fined as homophobia/transphobia …


DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing

P He, J Gao, W Chen - arXiv preprint arXiv:2111.09543, 2021

This paper presents a new pre-trained language model, DeBERTaV3, which improves the original DeBERTa model by replacing mask language modeling (MLM) with replaced token detection (RTD), a more sample-efficient pre-training task. Our …


Debiasing Multilingual Word Embeddings: A Case Study of Three Indian Languages

S Bansal, V Garimella, A Suhane, A Mukherjee - arXiv preprint arXiv:2107.10181, 2021

… We do not plan to release non-anonymized bios-data since it is already available for download from the commoncrawl portal. Page 4 … One of the main challenges here is the absence of appropriate datasets (bios) in Indic languages on Common Crawl dumps …


Decentralised Privacy: A Distributed Ledger Approach

P Papadopoulos, N Pitropakis, WJ Buchanan - … , and Devices; Hussain, CM, Di Sia, P …, 2021

Our world due to the technological progress became fast-paced and is constantly evolving, thus changing every single day. Consequently, the most valuable asset on earth is not gold or oil anymore but data. Big data companies try to take advantage …


Deciphering Speech: a Zero-Resource Approach to Cross-Lingual Transfer in ASR

O Klejch, E Wallington, P Bell - arXiv preprint arXiv:2111.06799, 2021

… To prevent this issue we always used a language model trained on CommonCrawl data when deciphering and later re-decoding the training data. The CommonCrawl language models were trained on approximately 100M tokens and …


Deduplicating Training Data Makes Language Models Better

K Lee, D Ippolito, A Nystrom, C Zhang, D Eck… - arXiv preprint arXiv …, 2021

… models trained on CommonCrawl in- clude GPT-3 (Brown et al., 2020) with the addition of book datasets, GROVER (Zellers et al., 2019) on a restricted subset filtered to news domains called RealNews, and T5 (Raffel et al., 2020) …


Deep artificial neural networks reveal a distributed cortical network encoding propositional sentence-level meaning

AJ Anderson, D Kiela, JR Binder, L Fernandino… - Journal of Neuroscience, 2021

… represents individual words as 300-dimensional floating point vectors derived by factorizing a word co- 286 occurrence matrix (vocabulary size is 2.2 million words and co-occurrences were measured across 840 287 billion tokens …


Deep contextualized text representation and learning for fake news detection

M Samadi, M Mousavian, S Momtazi - Information Processing & Management, 2021

JavaScript is disabled on your browser. Please enable JavaScript to use all the features on this page. Skip to main content Skip to article …


Deep Learning and Computational Chemistry

T James, D Hristozov - Artificial Intelligence in Drug Design, 2022

… For example, ImageNet contains approximately 1.2 million labeled images [99] and Common Crawl, which contributes the bulk of the training data for GPT-3, nearly one trillion words [100]. The initial training phase for AlphaGo likewise utilized a set …


Deep Learning for Blocking in Entity Matching: A Design Space Exploration

H Li, N Tang, M Ouzzani, Y Govind, D Paulsen, G Fung…

… Popular word-level embeddings such as word2vec [52] and GloVe [67] and characterlevel embeddings such as fastText [9] often come with pre-trained embeddings that are trained on a large generic external corpus such as Wikipedia, Common Crawl or PubMed …


Deep learning for sentence clustering in essay grading support

LH Chang, I Rastas, S Pyysalo, F Ginter - arXiv preprint arXiv:2104.11556, 2021

… [16] and refer readers to this paper for further details of the embeddings. These embedding were induced using the implementation of the skip-gram algorithm [22] in the word2vec software package on Finnish Common Crawl data …


Deep Learning Transformer Architecture for Named Entity Recognition on Low Resourced Languages: State of the art results

R Hanslo - arXiv preprint arXiv:2111.00830, 2021

… model trained on 100 languages uses 2.5 TB of CommonCrawl (CC) data [2]. From the 100 languages used by the XLM-R multilingual masked language model, it is noted that Afrikaans (af) and isiXhosa (xh) are included in the pre-training. …


Deep Learning With Anaphora Resolution for the Detection of Tweeters With Depression: Algorithm Development and Validation Study

A Wongkoblap, MA Vadillo, V Curcin - JMIR Mental Health, 2021

Background: Mental health problems are widely recognized as a major public health challenge worldwide. This concern highlights the need to develop effective tools for detecting mental health disorders in the population. Social …


Deep Learning-based Sentiment Analysis of Facebook Data: The Case of Turkish Users

Ö Çoban, SA Özel, A İnan - The Computer Journal, 2021

Abstract. Sentiment analysis (SA) is an essential task for many domains where it is crucial to know users' public opinion about events, products, brands, politi.


DeepEva: A deep neural network architecture for assessing sentence complexity in Italian and English languages

GL Bosco, G Pilato, D Schicchi - Array, 2021

Abstract Automatic Text Complexity Evaluation (ATE) is a research field that aims at creating new methodologies to make autonomous the process of the text complexity evaluation, that is the study of the text-linguistic features (eg, lexical, syntactical …


Delving into Deep Imbalanced Regression

Y Yang, K Zha, YC Chen, H Wang, D Katabi - arXiv preprint arXiv:2102.09554, 2021

Page 1. Delving into Deep Imbalanced Regression Yuzhe Yang 1 Kaiwen Zha 1 Ying-Cong Chen 1 Hao Wang 2 Dina Katabi 1 Abstract Real-world data often exhibit imbalanced distributions, where certain target values …


Democratic Backsliding and Media Responses to Government Repression: Machine Learning Evidence from Tanzania

… Using the list of international, regional and national domains, we first check GDELT and the Internet Archive for available links, pull the available web pages from the Common Crawl and from the websites directly. We then initialize Scrapy …


Dense Events Grounding in Video

… 2015) as previous methods to extract C3D video features on both datasets. And we use Glove (Jeffrey Pennington and Manning 2014) word embeddings pretrained on Common Crawl to represent each word in the sentences …


Depression Detection of Twitter Posters using Deep Learning with Anaphora Resolution

… occurrent words as vectors [26]. GloVe was trained on several textual datasets eg, Wikipedia and common crawl (a copy of web content), and supported 50, 100, 200, and 300 dimensional vectors. Anaphora resolution is another …


DESCGEN: A Distantly Supervised Datasetfor Generating Abstractive Entity Descriptions

W Shi, M Joshi, L Zettlemoyer - arXiv preprint arXiv:2106.05365, 2021

… We further parse the HTML texts of the web pages and extract contents as source documents. Real News To expand the collection of source documents, we extract entity mentions in RealNews (Zellers et al., 2019) …


Design, analysis and implementation of advanced methodologies to measure the socio-economic impact of personal data in large online services

The web ecosystem is enormous, and overall it is sustained by an intangible attribute that mainly supports the majority of free services: the exploitation of personal information. Over the years, concerns on how services use personal data …