Project Debater Datasets

Annotated datasets developed as part of IBM Project Debater, covering argument mining, stance classification, debate speech analysis, key point analysis and core NLP tasks.

About Project Debater

Project Debater was an IBM Research initiative aimed at advancing artificial intelligence systems capable of understanding complex topics, analyzing large collections of information, and constructing coherent arguments. The project gained international attention in February 2019 when it participated in a live public debate against world champion debater Harish Natarajan in San Francisco. During the event, the AI system delivered a structured opening speech, rebutted its opponent's arguments, and presented a closing summary, demonstrating significant advances in natural language understanding, evidence-based argumentation, and persuasive speech generation. The debate marked a milestone in AI research and showcased how AI can assist people in exploring complex issues and making more informed decisions. A recording of the debate is available here.

Beyond its public demonstrations, Project Debater drove significant research in argument mining, evidence detection, speech generation, topic understanding, and decision support. The datasets listed on this page were created and curated as part of this effort to support research in natural language processing, argumentation, and AI reasoning and were released to the public.

Requesting access

To obtain access to any of the datasets listed below, please contact Yoav Katz (katz@il.ibm.com) or Roy Bar-Haim (roybar@il.ibm.com).

Project Debater Datasets

The development of an automatic debating system naturally involves advancing research in a range of artificial intelligence fields. This page presents several annotated data sets developed as part of Project Debater to facilitate this research. It is organized by research sub-fields explained below.

Argument Mining is a prominent research frontier. Within this field, we distinguish between Argument Detection - the detection and segmentation of argument components such as claims and evidence; and Argument Stance Classification – determining the polarity of an argument component with respect to a given topic.

Beyond argument mining, a debating system should face the challenge of interactivity i.e., the ability to understand and rebut the text of the opponent’s speech. Debate Speech Analysis is a new research field that focuses on this challenge.

Another important aspect of a debating system is the ability to interact with its surroundings in a human-like manner. Namely, it should be able to articulate arguments and listen to arguments made by others. Regarding the former, the Text to Speech system must demonstrate human-like expressiveness to keep human listeners engaged. The latter may call for Speech-to-text systems that are especially designed for a debating scenario.

Finally, a debating system should naturally rely on more fundamental NLP capabilities. One example is the ability to assess the semantic relatedness of various pieces of texts and glue these into a coherent narrative. The system should also have the ability to identify the basic concepts mentioned in the text. The corresponding benchmark data we released thus far in this context are described in the section on Basic NLP.

Licensing and copyright

The datasets are released under the following licensing and copyright terms, unless specified otherwise in their release notes:

Each copy or modified version that you distribute must include a licensing notice stating that the work is released under CC-BY-SA and either (a) a hyperlink or URL to the text of the license or (b) a copy of the license. For this purpose, a suitable URL is: https://creativecommons.org/licenses/by-sa/3.0/.

Argument Detection

The various argument detection datasets differ in size (e.g., number of topics), type of element detected (claims, claim sentences, or evidence), and method used for detection (pre-selected articles vs. automatic retrieval). The table below lists the different datasets and provides information on their characteristics:

DatasetReferenceTopicsElementMethod
IBM Debater® - Argumentative Sentences in Recorded DebatesFindings of EMNLP 202020Argumentative SentenceExtracted from ASR (automatic speech recognition) output of debate speeches over controversial topics
IBM Debater® - Evidence SentencesAAAI 2020222EvidenceAutomatically retrieved Wikipedia sentences with manual crowd-sourcing scoring
IBM Debater® - Claim Sentences SearchCOLING 20181501Claim SentenceAutomatically retrieved Wikipedia sentences
IBM Debater® - Evidence SentencesACL 20181182EvidenceAutomatically retrieved Wikipedia sentences
IBM Debater® - Claims and EvidenceEMNLP 2015583Claim/EvidencePre-selected Wikipedia articles
IBM Debater® - Claims and EvidenceArgMining 2014333Claim/EvidencePre-selected Wikipedia articles

1 70 train, 30 held-out, 50 test
2 83 train, 35 test
3 Leave one topic out

  • New annotation layer for 700 sentences that were extracted from ASR (automatic speech recognition) output of debate speeches over controversial topics. A sentence is annotated as positive if it contains an argument for the given topic. The sentences were sampled from the releases of IBM Debater® - Recorded Debating Datasets.

  • 29,429 pairs of a topic and a sentence with a score between 0 to 1, indicating the extent to which the sentence can serve as evidence that supports or contests the topic.

    The sentences were extracted from Wikipedia, and the score was obtained by manual annotation using crowd-sourcing via the Figure Eight platform.

  • Claim sentence search results of the q_mc query containing 1.49M sentences, and claim sentence test set containing 2.5k top predicted sentences along with their labels. The sentences were retrieved from Wikipedia 2017.

    The dataset includes:

    • readme_mc_queries.txt - Readme of the claim sentence search results
    • readme_test_set.txt - Readme of the test set
    • q_mc_train.csv - Sentences retrieved by the q_mc query on 70 train topics
    • q_mc_heldout.csv - Sentences retrieved by the q_mc query on 30 heldout topics
    • q_mc_test.csv - Sentences retrieved by the q_mc query on 50 test topics
    • test_set.csv - Top predictions of our system along with their labels
  • 5,785 pairs of a topic and a sentence with a binary annotation indicating whether the sentence is valid evidence relevant for the topic. The sentences were extracted from Wikipedia, and the prior for a positive instance is 41%.

    The data set includes 118 diverse topics, from domains such as politics, science and education. Each topic generally deals with one clearly identifiable concept.

    The data set is split into two sets: 83 topics for training (4,066 sentences), and 35 topics for testing (1,719 sentences).

  • 2294 labeled claims and 4690 labeled evidence for 58 different topics. Labeled data published by Rinott et al. EMNLP-2015. This data is an extension of the CE-ACL-2014 data.

    The dataset includes:

    • Two CSV files containing, for each topic, the claims and evidence that were identified for it in relevant Wikipedia articles.
    • The original Wikipedia articles - from Wikipedia April 2012 dump - in the form of text files, cleaned from any Wikisyntax or HTML markup.
  • 1,392 labeled claims for 33 different topics, and 1,291 labeled evidence for 350 distinct claims in 12 different topics. These data were published by Aharoni et al. in the First Workshop on Argumentation Mining at ACL-2014.

    The dataset includes:

    • Two CSV files containing, for each topic, the claims and evidence that were identified for it in relevant Wikipedia articles.
    • The original Wikipedia articles - from Wikipedia April 2012 dump - in the form of text files, cleaned from any Wikisyntax or HTML markup.

Argument Quality

After arguments are detected, one faces the task of identifying those of a higher quality. To that end, we created the following data sets, with the focus on identifying the better argument.

DatasetReferenceTopicsElementMethod
IBM Debater® - Evidence QualityACL 201969EvidenceAutomatically retrieved Wikipedia sentences
IBM Debater® - IBM-ArgQ-6.3kArgs
IBM Debater® - IBM-ArgQ-5.3kArgs
IBM Debater® - IBM-ArgQ-14kPairs
IBM Debater® - IBM-ArgQ-9.1kPairs
EMNLP 201922ArgumentActively collected arguments from crowds
IBM Debater® - IBM-ArgQ-Rank-30kArgsarXiv71ArgumentActively collected arguments from crowds
  • 5,697 pairs of evidence with annotations for the question of which evidence, in each pair, is more convincing in the context of a given topic.

    The evidences were taken from the data set IBM Debater® - Evidence Sentences and cover 69 different topics.

    The data set is split into two sets: 4,319 pairs for train and 1,378 for test.

  • 6.3K arguments collected actively from crowds about 22 debatable motions. All arguments were labeled for point-wise quality and a set of 14K argument pairs were labeled for pair-wise quality. The dataset includes the arguments before and after cleansing.

    • IBM Debater® - IBM-ArgQ-6.3kArgs - 6.3K arguments with point-wise quality label (before cleansing)
    • IBM Debater® - IBM-ArgQ-5.3kArgs - 5.3K arguments with point-wise quality label (after cleansing)
    • IBM Debater® - IBM-ArgQ-14kPairs - 14K argument pairs with pair-wise quality label (before cleansing)
    • IBM Debater® - IBM-ArgQ-9.1kPairs - 9.1K argument pairs with pair-wise quality label (after cleansing)
  • 30,497 arguments actively collected from the crowd on 71 debatable topics, labeled for point-wise quality and stance.

    Each argument is presented with two types of quality scores, as presented in the paper, and a stance score.

    Arguments are split into train, dev and test sets.

Argument Stance Classification and Sentiment Analysis

A debating system must distinguish between arguments that support its side in the debate and those supporting the opponent’s side. The following datasets were developed as part of the work on Project Debater’s stance classification engine.

Claim Stance

The claim stance dataset includes stance annotations for claims, as well as auxiliary annotations for intermediate stance classification subtasks.

DatasetReferenceTopicsNumber of ClaimsMethod
IBM Debater® - Claim Stance DatasetEACL 2017552,394Manually identified and annotated claims from Wikipedia
  • 2,394 labeled claims for 55 topics. The dataset includes the stance (Pro/Con) of each claim towards the topic, as well as fine-grained annotations, based on the semantic model of Bar-Haim et al. [EACL 2017] (topic target, topic sentiment towards its target, claim target, claim sentiment towards its target, and the relation between the targets).

    The dataset includes:

    • A utf-8 JSON file containing the topics and the claims found for these topics in Wikipedia articles. Topics and claims are annotated as described above.
    • A utf-8 CSV file containing the same information as the JSON file.
    • The original Wikipedia articles - from Wikipedia April 2012 dump - in the form of text files. For each article, we provide both the original (raw) version, and a clean version, in which any Wikisyntax and HTML markup is removed.
    • A CSV index file, containing the article title and Wikipedia URL for each article.

    The dataset is divided into a training set (25 topics, 1,039 claims) and a test set (30 topics, 1,355 claims).

Sentiment Analysis

Sentiment analysis is an important sub-component of our stance classification engine. The following two resources address sentiment analysis of complex expressions, which goes beyond simple aggregation of word-level sentiments. The first resource is a sentiment lexicon of idiomatic expressions, like “on cloud nine” and “under fire”. The second resource addresses sentiment composition – predicting the sentiment of a phrase from the interaction between its constituents. For example, in the phrases “reduced bureaucracy” and “fresh injury”, both “reduced” and “fresh” are followed by a negative word. However, “reduced” flips the negative polarity, resulting in a positive phrase, while “fresh” propagates the negative polarity to the phrase level, resulting in a negative phrase. Accordingly, “reduced” is part of our “reversers” lexicon, and “fresh” is part of the “propagators” lexicon.

DatasetReferenceContentSource
IBM Debater® - Sentiment Lexicon of IDiomatic Expressions (SLIDE)LREC 20185,000 frequently occurring idioms with sentiment annotationManually annotated idioms from Wiktionary
IBM Debater® - Sentiment Composition LexiconsCOLING 2018Sentiment composition lexicons containing 2,783 words and sentiment lexicons containing 66K unigrams and 262K bigrams.Automatically learned from a large proprietary English corpus
  • 5000 frequently occurring idioms with sentiment annotation. The idioms were selected from Wiktionary, and over 40% of them were found to be sentiment-bearing via crowdsource labeling.

    Dataset includes idioms, sentiment labels, and distribution of sentiment annotation from crowdsourced labels.

  • Sentiment composition lexicons containing 2,783 words and sentiment lexicons containing 66K unigrams and 262K bigrams.

    The lexicons were learned from a large proprietary English corpus.

    The dataset includes:

    • ReleaseNotes.txt - release notes
    • SEMANTIC_CLASSES.xlsx - composition lexicons for reversers, propagators, and dominators
    • ADJECTIVES.xlsx - composition lexicons for two pairs of gradable adjectives
    • LEXICON_UG.txt - unigrams sentiment lexicon
    • LEXICON_BG.txt - bigrams sentiment lexicon

Expert Stance

Expert evidence (premise) is a commonly used type of argumentation scheme. Prior knowledge about the expert’s stance towards the debate topic can help predict the polarity of such arguments. For example, an argument made by Richard Dawkins about atheism is likely to have a PRO stance, since Dawkins is a well-known atheist. Such information can be extracted from Wikipedia categories: Dawkins, for instance, is listed under “Antitheists”, ”Atheism activists”, “Atheist feminists” and “Critics of religions”. The Wikipedia Category Stance dataset contains stance annotations of Wikipedia categories towards Wikipedia concepts representing controversial topics.

DatasetReferenceTopicsNumber of CategoriesMethod
IBM Debater® - Wikipedia Category StanceArgMining 20161324603Manually annotated Wikipedia Categories
  • 4603 Wikipedia categories and lists annotated for stance (Pro/Con) towards a concept, for a set of 132 concepts. The data were published by Toledo-Ronen et al. at the ACL-2016 Workshop on Argument Mining.

    The dataset includes:

    • ReleaseNotes.txt - release notes file describing the data
    • WikipediaCategoriesResults.csv - the dataset
    • WikipediaCategoriesLabeling.docx - the guidelines used for labeling the data

Debate Speech Analysis

In order to respond to an opponent’s speech, the system must process the opponent’s voice and ‘understand’ its content. The provided datasets focus on the Automatic Speech Recognition (ASR) output and on the upstream tasks related to understanding opponents' speeches.

DatasetReferenceSpeechesTopics
IBM Debater® - Recorded Debating Dataset - Release #5 + Counter speech annotationsACL 20203,562440
IBM Debater® - Recorded Debating Dataset - Release #4 + Annotated general-purpose claim-rebuttal pairsEMNLP 2019
EMNLP 2018
20050
IBM Debater® - Recorded Debating Dataset - Release #3 + Annotated mined claimsArgMining 2019 @ACL
LREC 2018
400200
IBM Debater® - Recorded Debating Dataset - Release #2 + Annotated argumentsEMNLP 2018
LREC 2018
20050
IBM Debater® - Recorded Debating Dataset - Release #1LREC 20186016

Each release is available in three versions: the full version, the version with compressed (down-sampled, mono & compressed with flac) audio files and the light version without audio files.

  • 3,562 speeches recorded by professional debaters discussing 440 controversial topics (with their automatic and manually-corrected transcript texts), and an annotation specifying the response speeches recorded for each speech.

    Contents:

    • Recordings of expert debaters
    • Automatic and manually-corrected transcripts of the speeches, in both raw and cleaned (processed) versions
    • An annotation specifying the response speeches recorded for each speech, and the type of the response (explicit/implicit)
    • Metadata describing the speeches, such as the topic discussed in each speech

    The dataset includes:

    • Audio files of all debate speeches (not included in the light version)
    • Automatic and manually-corrected transcripts of the speeches, in both raw and cleaned (processed) versions
    • An annotation specifying the response speeches recorded for each speech, and the type of the response (explicit/implicit)
    • Metadata describing the speeches, such as the topic discussed in each speech

    Available versions:

    • Full version - 2 parts: 30 + 21.7 GB
    • Downsampled audio files: 21.2 GB
    • Light version - no audio files: 12.1 MB
  • 200 speeches recorded by professional debaters discussing 50 controversial topics (with their manual and automatic transcriptions), and 55 general-purpose claim-rebuttal pairs, along with the results of several annotation experiments performed on these data.

    Contents:

    • Recordings of expert debaters
    • 55 general-purpose claim and rebuttal pairs written by an expert human debater
    • An annotation specifying for each of the 50 controversial topics, which of the 55 general-purpose claims is relevant to the topic
    • An annotation of general-purpose claims relevant to a topic, specifying whether a relevant claim was mentioned in speeches discussing the topic
    • An annotation of general-purpose claims and sentences from speeches in which they were mentioned, specifying whether the claim was mentioned in the sentence
    • An annotation of general-purpose rebuttals, specifying whether they are a plausible response to general-purpose claims mentioned in speeches

    The dataset includes:

    • Audio files of 200 debating speeches (not included in the light version). [first released in IBM Debater® - Recorded Debating Dataset - Release #2]
    • Manual and automatic transcripts of the speeches, in both raw and cleaned (processed) versions. [first released in IBM Debater® - Recorded Debating Dataset - Release #2]
    • 55 general-purpose claim-rebuttal pairs written by an expert human debater
    • The results of several annotation experiments performed using the general-purpose claim-rebuttal pairs and the speeches

    Available versions:

    • Full version: 3.2 GB
    • Compressed audio files: 1.2 GB
    • Light version - no audio files: 3.3 MB
  • 400 speeches recorded by professional debaters about 200 controversial topics (with their manual and automatic transcripts) and 4,876 mined claims annotated as mentioned explicitly, implicitly, or not at all, in those speeches.

    Contents:

    Recordings of expert debaters + mined claims annotated in a listening comprehension task

    The dataset includes:

    • Audio files of 400 debating speeches (not included in the light version)
    • Manual and automatic transcripts of the speeches, in both raw and cleaned (processed) versions
    • 4,876 annotated mined claims

    Available versions:

    • Full version: 5.6GB
    • Compressed audio files: 2.4GB
    • Light version - no audio files: 5.16MB
  • 200 speeches recorded by professional debaters about controversial topics (with their manual and automatic transcripts) and 756 arguments annotated as mentioned/not mentioned in these speeches.

    Contents:

    Recordings of expert debaters + arguments annotated in a listening comprehension task

    Note: 60 speeches from release#1 are not included in this dataset.

    The dataset includes:

    • Audio files of 200 debating speeches (not included in the light version)
    • Manual and automatic transcripts of the speeches, in both raw and cleaned (processed) versions
    • 756 annotated arguments

    Available versions:

    • Full version: 5GB
    • Compressed audio files: 1.19GB
    • Light version - no audio files: 2.87MB
  • 60 speeches recorded by professional debaters about controversial topics, and their manual and automatic transcripts, in both raw and cleaned (processed) versions.

    Contents:

    Recordings of 10 expert debaters

    The dataset includes:

    • Audio files of 60 debating speeches (not included in the light version)
    • Manual and automatic transcripts of the speeches, raw and cleaned versions

    Available versions:

    • Full version: 1.62GB
    • Compressed audio files: 326MB
    • Light version - no audio files: 1MB

Debate Topic Expansion

When debating a controversial topic, it is often desirable to expand the boundaries of the discussion, and bring up arguments about related topics. For example, when discussing the pros and cons of the presidential system, it is natural to contrast it with those of the parliamentary system. When debating alternative medicine, we may discuss specific examples, such as homeopathy and naturopathy. Conversely, when discussing bitcoins, we can speak more broadly on cryptocurrency. Debate Topic Expansion is the task of automatically finding related topics that can enrich our arguments and strengthen our case when debating a given topic.

We distinguish between two types of expansions: "consistent" and "contrastive". Arguing in favor or against a consistent expansion may support the same stance towards the original topic, whereas for contrastive expansions the stance is reversed. For example, "cryptocurrency" is a consistent expansion of "bitcoin", while "parliamentary system" is a contrastive expansion of "presidential system", as we may support the presidential system by criticizing the parliamentary system.

The Debate Topic Expansion dataset includes pairs (DC,EC) where DC is a debate topic and EC is an automatically-extracted candidate expansion for DC. Both DC and EC are Wikipedia concepts. Each such pair was manually labeled as good/bad expansion.

DatasetReferenceTopicsAnnotated ExpansionsMethod
IBM Debater® - Debate Topic ExpansionACL 2019632Consistent: 1,741
Contrastive: 1,326
Candidate expansions were generated automatically, and then were manually labeled as good/bad expansions
  • The Debate Topic Expansion dataset includes Wikipedia concept pairs (DC, EC), where DC is a debate topic and EC is a candidate expansion for DC.

    Each pair is manually annotated as good/bad expansion.

    The dataset includes:

    1,741 annotated consistent expansions and 1,326 annotated contrastive expansions for 632 topics.

    Size: 682 KB (101KB zipped)

Expressive Text to Speech

The emphasized words dataset was created to train and evaluate a system that receives a written argumentative speech and predicts which words should be emphasized by the Text-to-Speech component.

DatasetReferenceNumber of ParagraphsNumber of SentencesSource
IBM Debater® - Labeled Emphasized Words in SpeechInterspeech 201824854002The speeches were created based on claims/evidence automatically detected from Wikipedia
  • The dataset contains 2485 paragraphs, which comprise of 4002 sentences from audience-addressed speeches that were annotated for emphasized words.

    The dataset can be used to train models to predict words that should be emphasized in expressive TTS.

    It is a subset of the benchmark data that was used in the InterSpeech'18 paper "Word Emphasis Prediction for Expressive Text to Speech"

Basic NLP Tasks

The following datasets relate to basic NLP tasks, addressed as part of Project Debater.

Semantic Relatedness

Predicting semantic relatedness between texts is a basic NLP problem with a wide variety of applications. Relatedness can be measured between several types of texts, ranging from words to documents. The relatedness datasets listed below differ in the type of elements considered (words, multi-word-terms, and concepts), number of topics from which the pairs were extracted, and number of annotated pairs.

DatasetReferenceNumber of TopicsType of elementsNumber of pairs
IBM Debater® - Wikipedia Oriented Relatedness Dataset (WORD)LREC 2018123 (82 train, 41 test)Wikipedia Entities19,276 (12,969 train, 6307 test)
IBM Debater® - Multi-word Term Relatedness Benchmark (TR9856)ACL 201547Words and Multi-word Terms9,856
  • A large semantic relatedness dataset, composed of 19,276 pairs of Wikipedia concepts with manual scores for their level of relatedness.

  • Term-relatedness values for 9,856 pairs of terms. These data were published by Levy et al. at ACL-2015.

    The dataset includes:

    • Release Notes.txt - release notes describing the data
    • TermRelatednessResults.csv - the dataset
    • TermRelatednessLabeling.doc - the guidelines used for labeling the data

Mention Detection

The goal of Mention Detection is to map entities/concepts mentioned in text to the correct concept in a knowledge base. This process involves segmenting the text (as some concepts span multiple words) and the disambiguation of terms with more than one meaning.

DatasetReferenceNumber of SentencesNumber of TopicsSource
IBM Debater® - Mention Detection BenchmarkarXiv 20183000 (500 train and 500 test for each of the three text sources)88Mix of Wikipedia articles and ASR/manual transcripts of speeches by expert debaters
  • 3000 sentences annotated with Mentions. The sentences are taken from Wikipedia and from professional speakers discussing several topics.

    There are about 20,000 annotated Mentions on those sentences.

    The benchmark includes:

    • The topics of discussion
    • The sentences related to those topics
    • The Mentions annotated on those sentences

Text Clustering

Text clustering is a widely-studied NLP problem. Clustering can be applied to texts at different levels, from single words to full documents, and can vary with respect to the clustering goal. In thematic clustering, the aim is to cluster texts based on thematic similarity between them, namely grouping together texts that discuss the same theme.

Thematic clustering of sentences is important for various use cases. For example, in multi-document summarization, one often extracts sentences from multiple documents that should be organized into meaningful sections and paragraphs. Similarly, within the emerging field of computational argumentation, arguments may be found in a widespread set of articles, which further require thematic organization to generate a compelling argumentative narrative.

Evaluation of thematic clustering methods requires a ground truth dataset of sentence clustering. Unfortunately, sentence clustering is considered a very difficult task for humans. As a result, there is no standard human annotated sentence clustering dataset.

In the dataset “Thematic Clustering of Sentences” sentences are annotated for their thematic clusters. This annotation enables to evaluate thematic clustering methods. The dataset was generated automatically by leveraging the partition of Wikipedia articles into sections. The underlying assumption of its creation was that the section structure of a Wikipedia article can serve as ground truth for the thematic clustering of its sentences. Details about the way this dataset was generated can be found in the article.

DatasetReferenceNumber of TopicsNumber of Clusters per TopicNumber of Sentences per Topic
IBM Debater® - Thematic Clustering of Sentences DatasetACL 20186925-1217-1614
  • A benchmark of sentence-clustering created automatically based on the partition of Wikipedia articles into sections. The dataset contains 692 Wikipedia articles; each one corresponds to a different clustering task. The dataset includes the article name, the sentence text, and the name of the cluster to which the sentence belongs, which is the title of the section from which the sentence was extracted. A detailed description of the generation process of this dataset can be found in the article.

Concept Abstractness

During the last decades, the influence of psycholinguistic properties of words on cognitive processes has become a major topic of scientific inquiry. Among the most studied psycholinguistic attributes are concreteness, familiarity, imagery, and average age of acquisition. Abstractness quantifies the degree to which an expression denotes an entity that can be directly perceived by human senses. As an example, the word "feminism" is usually perceived as abstract, but the word "screwdriver" is associated with a concrete meaning.

We introduce a weakly supervised approach for inferring the property of abstractness of words and expressions in the complete absence of labeled data. Exploiting only minimal linguistic clues and the contextual usage of a concept as manifested in textual data, we train sufficiently powerful classifiers, obtaining high correlation with human labels. The released dataset contains 300K Wikipedia concepts automatically rated for their degree of abstractness.

DatasetReferenceNumber of Topics
IBM Debater® - Concept AbstractnessEMNLP 2018300,000
  • 100,000 * 3 Wikipedia titles (unigrams, bi-grams and tri-grams) rated for their degree of abstractness.

    A detailed description of the generation process can be found in the article, and description of the dataset format can be found in the attached README file

Concept Controversiality

With the growing interest in social applications of Natural Language Processing and Computational Argumentation, a natural question is how controversial a given concept is. Prior works relied on Wikipedia’s metadata and on content analysis of the articles pertaining to a concept in question. Here we show that the immediate textual context of a concept is strongly indicative of this property, and, using simple and language-independent machine-learning tools, we leverage this observation to achieve state-of-the-art results in controversiality prediction. In addition, we analyze and make available two new datasets of concepts labeled for controversiality. One is an extension of an already published dataset, and the other is significantly larger than existing datasets, and grades concepts on a 0-1 scale, rather than treating controversiality as a binary label.

DatasetReferenceNumber of Topics
IBM Debater® - Concept ControversialityarXiv1213 + 3560
  • 1213 plus 3560 Wikipedia titles rated for their degree of controversiality.

    Details about the generation thereof can be found in the paper, and description of the dataset format can be found in the attached README file.

Automatic Claim Negation

The main goal of argumentation mining is to analyze argumentative structures within an argument-rich document, and reason about their composition. Recently, there is also interest in the task of simply detecting claims (sometimes called conclusion) in general documents. In this work we ask how this set of detected claims can be augmented further, by adding to it the negation of each detected claim. This presents two NLP problems: how to automatically negate a claim, and when such a negated claim can plausibly be used. We present first steps into solving both these problems, using a rule-based approach for the former and a statistical one towards the latter.

DatasetReferenceNumber of Sentences
IBM Debater® - NegatedClaims DatasetACL 20151240
  • The dataset contains 1,240 examples while each example contains a claim, an automatically generated negated claim, and a label that indicates whether the negated claim is “usable” according to 5 annotators (0 - no , 1 - yes, ? - undecided).

Classes of Principled Arguments

Principled common-place arguments are one of the fundamental building blocks of effective debating. This data set describes a modelling of such arguments by organizing them into Classes of Principled Arguments (CoPAs), and associating these classes with a large set of motions.

DatasetReferenceContent
IBM Debater® - CoPA-Motion LabelingACL 2019Association of CoPAs to 689 motions
IBM Debater® - CoPA-Speech LabelingACL 2019Statistics for the number of claims mentioned in recorded speeches from each CoPA
  • Matching of motions to CoPAs

  • Fraction of claims mentioned in speeches, by CoPA

Key Point Analysis

Key Point Analysis (KPA) is a novel summarization technology that extracts the main points and their salience from a collection of opinions. It has been successfully applied to diverse types of data, including arguments, user reviews and survey responses. KPA maps the input texts to a set of automatically-extracted short sentences and phrases, termed Key Points (KPs), which provide a concise plain-text summary of the data. The prevalence of each KP is quantified as the number of its matching sentences. We release the following benchmarks for the task of mapping input texts to key points.

Argumentation Datasets

The ArgKP dataset contains 24,093 (argument, key point) pairs labeled as matching/non-matching, for 28 controversial topics. For each pair, the topic and stance are also indicated. The ArgKP-2021 dataset extends ArgKP with three additional topics, amounting to 31 topics and 27,519 (argument, key point) pairs overall. The ArgKP-2023 dataset includes 7,584 pairs for 10 additional topics (9,281 including “undecided” pairs).

Municipal Survey Dataset

The SurveyKP dataset contains 15,075 (sentence, key point) pairs labeled as matching/non-matching (15,189 including “undecided” pairs), for sentences that were sampled from responses to the 2016-2017 Austin Community Survey.

DatasetReferenceTopicsPairs
IBM Debater® - SurveyKP - 15K labeled (sentence, key point) pairsEMNLP 2023 Industry Track115,189
IBM Debater® - ArgKP - 2023 9.2K labeled (argument, key point) pairsEMNLP 2023 Industry Track109,281
IBM Debater® - ArgKP - 2021 27.5K labeled (argument, key point) pairsArgument Mining workshop @ EMNLP 20213127,519
IBM Debater® - ArgKP - 24K labeled (argument, key point) pairsACL 20202824,093
  • 15,189 (argument, key point) pairs labeled as matching/non-matching, for 1 controversial topic.

    For each pair, the topic and stance are also indicated.

    Source:

    The sentences were sampled from open-ended responses to the 2016-2017 Austin Community Survey. The key points were automatically extracted by our KPA system from the entire survey, with minor manual edits. The stance of each pair (whether the feedback is positive or negative) is also indicated. The labeling of each (sentence, key point) pair as matching/non-matching was performed manually, by an in-house team of annotators.

  • 9,281 (argument, key point) pairs labeled as matching/non-matching, for 10 controversial topics.

    For each pair, the topic and stance are also indicated.

    Source:

    Similar to the ArgKP dataset (see below), ArgKP-2023 includes pro and con arguments from the IBM-ArgQ-Rank-30kArgs dataset, for 10 additional topics. Key points were extracted automatically by our system. The labeling of each (argument, key point) pair as matching/non-matching was performed manually, via crowd-sourcing.

  • 27,519 (argument, key point) pairs labeled as matching/non-matching, for 31 controversial topics.

    For each pair, the topic and stance are also indicated.

    Source:

    ArgKP dataset extended with 3 additional topics. The extension arguments were actively collected from the crowd and labeled for stance. As in ArgKP, the key points for each new topic were manually composed by an expert debater. The labeling of each new (argument, key point) pair as matching/non-matching was performed manually, via crowd-sourcing.

  • 24,093 (argument, key point) pairs labeled as matching/non-matching, for 28 controversial topics. For each pair, the topic and stance are also indicated.

    Source:

    The arguments, along with their topic and stance were taken from the IBM-ArgQ-Rank-30kArgs dataset (see above). The key points for each topic were manually composed by an expert debater. The labeling of each (argument, key point) pair as matching/non-matching was performed manually, via crowd-sourcing.

Claim Generation

Claim generation is a challenging task whose research is timely considering its potential impact on social media and the dissemination of information. The following dataset contains texts that were generated by various fine-tuned GPT-2 models for a large set of topics, labeled for being plausible and stance-bearing, determining the "claimness" of the generated texts.

DatasetReferenceGenerated TextsTopics
IBM Debater® - GPT-2 Finetuned Claim Generation DatasetFindings of EMNLP 20202839136
    • 2,739 texts generated by various GPT-2 fine-tuned models, on 131 topics from the dev and test sets, labeled for plausibility and stance.
    • 400 generated texts from the test set labeled for whether they are factual or opinion.
    • 100 additional generated texts on 5 other topics related to the covid-19 pandemic.

    Details regarding the GPT-2 fine-tuned models can be found in the paper, and additional details about the format of the dataset can be found in the README file.

Multilingual Argument Mining

The growing interest in argument mining and computational argumentation brings with it a plethora of Natural Language Understanding (NLU) tasks and corresponding datasets. However, as with many other NLU tasks, the dominant language is English, with resources in other languages being few and far between. This multilingual dataset contains data for three major computational argumentation tasks -- stance classification, evidence detection, and argument quality prediction -- in 5 languages: Italian, French, Spanish, German and Dutch.

DatasetReferenceTopics
IBM Debater® - XArgMining DatasetFindings of EMNLP 2020Evidence: 321
Arguments: 71
  • Contents:

    • 6,752 Human-authored arguments in 5 languages, annotated for their stance and quality
    • 30,497 English arguments annotated for their stance and quality, along with their machine-translation to 5 languages
    • 35,211 English Wikipedia sentences annotated for whether they are valid Evidence and their stance towards the discussed topic, along with their machine-translation to 5 languages
    • 210 English arguments and 200 English Wikipedia sentences, manually and automatically translated to German and Italian, with their corresponding labels collected for English and for the translated texts

    Please see the paper for further details on the creation of the dataset, and the accompanying readme file for technical details on the data format.

Targeted Sentiment Analysis

Understanding sentiment has been a major research area in NLP for decades, from the classification of review documents as expressing a positive or negative sentiment towards their topic, through a similar analysis of single sentences, and, in recent years, going deeper into identifying the sentiment expressed towards single words or phrases. The targeted sentiment analysis dataset provides sentence level sentiment target annotations of data sampled from five sources of product and user reviews.

DatasetReferenceData sourcesContents
IBM Debater® - TSA DatasetArxivSemEval'14, Stanford Sentiment Treebank, Opinosis, Yelp, AmazonA sample of sentences selected from each data source was annotated for targets and their sentiment:
- SemEval'14: 100 sentences from each domain (laptops and restaurants)
- SST: 500 sentences
- Opinosis: 512 sentences
- Yelp: 501 sentences
- Amazon: 502 sentences

Intent Classification

Intent classification or detection is the task of interpreting the user intent when conversing in a task-oriented dialogue setting.

The following dataset contains diverse queries mapped to intents concerning COVID-19 vaccines, in the context of VIRA, a chatbot that aims to address COVID-19 vaccine hesitancy (vaxchat.org).

DatasetReferenceIntentsExpressions
IBM Debater® - COVID-19 Vaccine Intent ExpressionsArxiv1817,990

Trust Classification

The following dataset contains annotations of High trust, Low institutional trust and Low agent trust in user inputs to VIRA, a chatbot that aims to address COVID-19 vaccine hesitancy (vaxchat.org).

DatasetReferenceUser InputsSource
IBM Debater® - COVID-19 Vaccine Trust AnnotationsArxiv3,022VIRADialogs