Analyzing the Persuasive Effect of Style in News Editorial Argumentation

News editorials argue about political issues in order to challenge or reinforce the stance of readers with different ideologies. Previous research has investigated such persuasive effects for argumentative content. In contrast, this paper studies how important the style of news editorials is to achieve persuasion. To this end, we first compare content- and style-oriented classifiers on editorials from the liberal NYTimes with ideology-specific effect annotations. We find that conservative readers are resistant to NYTimes style, but on liberals, style even has more impact than content. Focusing on liberals, we then cluster the leads, bodies, and endings of editorials, in order to learn about writing style patterns of effective argumentation.


Introduction
The interaction between the author and the intended reader of an argumentative text is encoded in the linguistic choices of the author and their persuasive effect on the reader (Halmari and Virtanen, 2005). News editorials, in particular, aim to challenge or to reinforce the stance of readers towards controversial political issues, depending on the readers' ideology . To affect readers, they often start with an enticing lead paragraph and end their argument with a "punch" (Rich, 2015).
Existing research has studied the persuasive effect of argumentative content and structure (Zhang et al., 2016; or combinations of content and style (Wang et al., 2017;Persing and Ng, 2017). In addition, some works indicate that different types of content affect readers with different personalities (Lukin et al., 2017) and beliefs (Durmus and Cardie, 2018). However, it remains unexplored so far what stylistic choices in argumentation actually affect which readers. We expect such choices to be key to generating effective argumentation . This paper analyzes the persuasive effect of style in news editorial argumentation on readers with different political ideologies (conservative vs. liberal). We model style with widely-used features capturing argumentativeness (Somasundaran et al., 2007), psychological meaning (Tausczik and Pennebaker, 2010), and similar (Section 3). Based on the NY-Times editorial corpus of El  with ideology-specific effect annotations (Section 4), we compare style-oriented with content-oriented classifiers for persuasive effect (Section 5). 1 While the general performance of effect prediction seems somewhat limited on the corpus, our experiments yield important results: Conservative readers seem largely unaffected by the style of the (liberal) NYTimes, matching the intuition that content is what dominates opposing ideologies. On the other hand, the style features predict the persuasive effect on liberal readers even better than the content features -while being complementary. That is, style matters as soon as ideology matches.
Knowing about the specific structure of news editorials, we finally obtain common stylistic choices in their leads, bodies, and endings through clustering. From these, we derive writing style patterns that challenge or reinforce the stance of (liberal) readers of (liberal) news editorials, giving insights into what makes argumentation effective.
(2016) modeled the structure underlying editorial argumentation, we use the corpus of El  meant to study the persuasive effects of editorials depending on the readers' political ideology. Halmari and Virtanen (2005) state that four aspects affect persuasion in editorials: linguistic choices, prior beliefs of readers, prior beliefs and behaviors of authors, and the effect of the text. Persuasive effectiveness reflects the rhetorical quality of argumentation . To assess effectiveness, Zhang et al. (2016) modeled the flow of content in debates,  the argumentative structure of student essays. Others combined different features for these genres (Persing and Ng, 2015). The impact of content selection relates to the notion of framing (Ajjour et al., 2019) and is well-studied in theory (van Eemeren, 2015). As Wang et al. (2017), however, we hypothesize that content and style achieve persuasion jointly. We target argumentative style here primarily, and we analyze its impact on liberal and conservative readers.
In related work, Lukin et al. (2017) found that emotional and rational arguments affect people with different personalities, and Durmus and Cardie (2018) take into account the religious and political ideology of debate portal participants. In followup work, Longpre et al. (2019) observed that style is more important for decided listeners. Unlike them, we focus on the stylistic choices made in well-planned argumentative texts.
The lead paragraphs and the ending of an editorial have special importance (Rich, 2015). Hynds (1990) analyzes how leads and endings changed over time, whereas Moznette and Rarick (1968) examined the readability of an editorial based on them. To our knowledge, however, no one investigated their importance computationally so far. In this paper, we close this gap by analyzing what style of leads and endings is particularly effective compared to the editorial's body.

Style Features
To model style, we need to abstract from the content of a news editorial. This section outlines the feature types that we employ for this purpose. Most of them have been widely used in the literature. Table 1 summarizes all features.
LIWC Psychological word usage is reflected in the Linguistic Inquiry and Word Count (Tausczik and Pennebaker, 2010). LIWC is a lexicon-based text analysis that assigns words to psychologically meaningful categories (Tausczik and Pennebaker, 2010). We use the LIWC version of Pennebaker et al. (2015), which contains 15 dimensions listed in the following with examples.
(2) Function words: pronouns, auxiliaries. NRC Emotion&Sentiment To represent the mood of editorials, we use the NRC lexicon of Mohammad and Turney (2013). NRC contains a set of English words and their associations with (1) emotions such as anger, disgust, and fear as well as (2) negative and positive sentiment polarities. These features are represented as the count of words associated with each category.
Webis ADUs To identify argumentative units in editorials that present evidence, we use the pre-trained evidence classifier of Al-Khatib et al. (2017). For each editorial, we identify the number of sentences that manifest anecdotal, statistical, and testimonial evidence respectively. Somasundaran et al. (2007) constructed a lexicon that includes various patterns of arguing such as assessments, doubt, authority, emphasis. For each lexicon, we have one feature that represents the count of the respective pattern in an editorial.

MPQA Subjectivity
We apply the subjectivity classifier provided in OpinionFinder 2.0 (Riloff and Wiebe, 2003;Wiebe and Riloff, 2005) on the editorials, in order to count the number of subjective and objective sentences there.

Data
As the basis of our analysis, we use the Webis-Editorial-Quality-18 corpus (El . The corpus includes persuasive effect annotations of 1000 English news editorials from the liberal New York Times (NYTimes). 2 The annotations capture whether a given editorial challenges the prior stance of readers (i.e., making them rethink it, but not necessarily change it), reinforces their stance (i.e., helping them argue better about the discussed topic), or is ineffective for them. Each editorial has been annotated by six annotators: three with liberal and three with conservative ideology.
To evaluate an editorial's persuasive effect on liberals, we computed the majority vote of their annotations for the editorial (and, similarly, for conservatives). We ended up with 979 editorials with effect labels for liberals and conservatives, because we found 21 duplicate editorials with the same content but different IDs (for these, we use the majority vote across all duplicates).
The corpus does not have predefined evaluation datasets. To mimic real-life scenarios, we chronologically split it into a training set (oldest 80%) and a test set (newest 20%). Table 2 shows the distribution of ideology-specific effects in the datasets.

Class
Training Test Liberal Conserv. Liberal Conserv.
Challenging 126  128  22  41  Ineffective  118  292  32  71  Reinforcing  539  363  142  84   Overall  783  783  196  196   Table 2: Distribution of the majority persuasive effect of the news editorials in the given training and test set for liberal and conservative ideology respectively.

Prediction of Persuasive Effects
To assess the impact of news editorial style on readers, we employ our style-based features on the task of predicting an editorial's persuasive effect: Given either of the two ideologies (liberal or conservative), predict for each editorial whether it is challenging, reinforcing, or ineffective.
We developed separate prediction models for the effect on liberals and conservatives, respectively. For each style feature type and for their combinations, we trained one SVM model with a linear kernel on the training set using scikit-learn (Pedregosa et al., 2011).
Given the dataset split mentioned above (training set 80%, test set 20%), we tuned the SVM's cost hyperparameter using grid search with 5-fold cross-validation on the training set. Since the distribution of effect labels is highly skewed, we set the hyperparameter class_weight to "balanced". We then trained the best model on the whole training set and evaluated it on the test set. For comparison, we also built models for standard content features (lemma 1-to 3-grams), and we consider the random baseline that picks an effect class by chance.
For both ideologies, Table 3 reports the macroand micro F 1 -scores for the style features, their best-performing combination, 3 the content features, and the best combination of content and style. 4 We computed significance using Wilcoxon's test to reveal differences between each two approaches among best style, content, best content+style, and baseline. 5 We obtained the means of F 1 -scores used in the significance tests by conducting fivefold cross-validation on the test set, using the same SVM hyperparameters as above.  In general, the results indicate that the persuasive effect seems hard to predict on the given corpus. Still, we observe that the style features play a notable role in predicting the effect of editorials on liberals. They achieve a significantly better macro F 1 -score of 0.43 when combined with content compared to 0.36 when using content alone, at p < 0.05. On the other hand, the F 1 -scores of content (macro 0.37, micro 0.38) and style (both 0.36) in predicting the effect on conservatives, are insignificantly different even from the baseline (0.33, 0.34).
These results suggest that style is important as soon as the ideology of a reader matches the one of the news portal (at least, this holds for liberal ideology), but not if it mismatches (here, conservative).

Identification of Style Patterns
Observing that the style of NYTimes editorials affects liberal readers, we seek to learn what patterns of writing style makes their argumentation effective. To this end, we (1) abstract each discourse part of an editorial (lead, body, ending) into a style label using cluster analysis and (2) identify sequential patterns of style labels that are specific to challenging, ineffective, and reinforcing editorials.
Clustering Styles of Discourse Parts Given the importance of specific discourse parts of editorials (Rich, 2015), we split each editorial into lead, body, and ending. For each part, we separately perform three steps on the training set of the given corpus: 6 6 The corpus of Sandhaus (2008) contains lead and paragraph annotations. The lead spans either the first two paragraphs (994 editorials), the first three (5), or the first only (1). We consider the last paragraph as the ending in all cases.
1. Extract the style features from Section 3.
2. Perform a cluster analysis on the style features using cosine k-means. k is determined with the elbow method on the inertia of the clusters.
3. Derive cluster labels from the most discriminating features across clusters: For each cluster, we determine those 2-3 values (e.g., "high tone, low authenticity") whose combination suffices to significantly distinguish a cluster from others. With high to very low, we mean here a feature has significantly higher or lower scores compared to other clusters. 7 Table 4 shows the distribution of lead, body, and ending clusters over challenging, ineffective, and reinforcing editorials.
For each discourse part, the most discriminating feature is tone, followed by authenticity. The former combines positive (higher scores) and neg-  Figure 1: Sequences of lead, body, and ending styles most specific to challenging, ineffective, and reinforcing news editorials. The triangles denote whether the given style attribute is high, medium, or (very) low. The ordering of attributes reflects their importance.
ative (lower scores) emotional tones (Cohn et al., 2004). The latter indicates the degree to which people authentically reveal themselves; the higher the score, the more personal, humble, or vulnerable the writer is (Newman et al., 2003). In Table 4, we observe, for example, that the lead of challenging editorials over-proportionally often shows low authenticity, or that bodies with positive tone but low authenticity tend to be ineffective. Table 4, we determine the (maximum) two labels for each discourse part that are most specific to each of the three persuasive effect classes. From these, we build all possible lead-body-ending sequences, as visualized in Figure 1. According to a χ-square test, the distributions of these sequences differ significantly at p < 0.05. They reveal the following patterns of NYTimes editorials for liberal readers:

Identification of Style Patterns From
• Challenging editorials often begin with a polar emotional tone, followed by a negative tone. They tend to have low authenticity (i.e., not humble/personal) in the whole discourse (see Figure 2 for an example).
• Ineffective editorials over-proportionally often start with authenticity and dull tone. They then tend to diffuse in different directions and to have a short ending paragraph.
• Reinforcing editorials tend to start and end with a negative tone. They often avoid relativ-  ity in the actual arguments (i.e., in the body).
While these insights are naturally still vague to some extent and require more analysis in follow-up research, they show a first way of capturing the style of editorial argumentation.

Conclusion
This paper analyzes the importance of news editorials style in achieving persuasive effects on readers with different political ideologies. We find evidence that style has a significant influence on how a (liberal) editorial affects a (liberal) reader. Inspired by the theory of the high importance of the lead and ending in writing editorials (Rich, 2015), we also reveal common effective and ineffective style sequences (lead-body-ending) statistically. Our findings help to understand how effective argumentation works in the political sphere of editorial argumentation -and how to generate such argumentation. In related work, El Baff et al. (2019) revealed the impact of style features on generating pathos-and logos-oriented short argumentative texts based on the rhetorical strategies discussed by . With the findings of this paper, we go beyond, defining the basis of a styledependent generation model for more sophisticated argumentation, as found in news editorials.