Automatic Identification of Narrative Diegesis and Point of View

,


Introduction
Interpreting a text's veridicality, correctly identifying the implications of its events, and properly de-limiting the scope of its references are all challenging and important problems that are critical to achieving complete automatic understanding of news stories and, indeed, text generally. There has been significant progress on some of these problems for certain sorts of texts, for example, recognizing implications on short, impersonal, factual text in the long-running Recognizing Textual Entailment challenge (RTE 1 ). On the other hand, narrative text (including much news writing) presents additional complications, in that to accomplish the tasks above one must take into account the narrator's point of view (i.e., first person or third person), as well as the narrator's personal involvement in the story (a feature that narratologists call diegesis).
In news stories specifically writers are encouraged to use the third person point of view when they wish to emphasize their objectivity regarding the news they are reporting (Davison, 1983). In opinion pieces or blog posts, on the other hand, first person is more common and implies a more personal (and perhaps more subjective) view (Aufderheide, 1997). News writers are also often in the position of reporting on events which they themselves have not directly observed, and in these cases can use an uninvolved style (known as hetereodiegetic narration) to communicate their relative remove from the action. When writers observe or participate in events directly, however, or are reporting on their own lives (such as in blog posts), they can use an involved narrative style (i.e., homodiegetic narration) to emphasize their personal knowledge and subjective, perhaps biased, orientation.
Before we can integrate knowledge of point of view (POV) or diegesis into text understanding, we must be able to identify them, but there are no systems which enable automatic classification of these features. In this paper we develop reliable classifiers for both POV and diegesis, apply the classifiers to texts drawn from five different news genres, demonstrate the accuracy of the classifiers on these news texts, and show that the POV and diegesis correlates much as expected with the genre. We release the classifiers and the training data so the field may build on our work and integrate these features into other text processing systems.
Regarding the point of view of the narrator, narratologist Mieke Bal claimed "The different relationships of the narrative 'I' to the objects of narration are constant within each narrative text. This means that one can immediately, already on the first page, see which is the [point of view]." (Bal, 2009, p. 29) This assertion inspired the development of the classifiers presented here: we had annotators mark narrative POV and diegesis from the first 60 lines of each of 270 English novels, which is a generous simulation of "the first page". This observation allowed us to transform the collection of data for supervised machine learning from an unmanageable burden (i.e., having annotators read every novel from start to finish) into a tractable task (reading only the first page). We chose novels for training, instead of news texts themselves, because of the novels' greater diversity of language and style.
Once we developed reliable classifiers trained and tested with this annotated data, we applied the classifiers to 39,653 news-related texts across five news genres, including: the Reuter's corpus containing standard newswire reporting; a corpus of scientific press releases scraped from EurekAlerts; the CSC Islamist Extremist corpus containing ideological story telling, propaganda, and wartime press releases; a selection of opinion and editorial articles scraped from LexisNexis, the Spinn3r web blog corpus, and . We checked a sample of the results, confirming that the classifiers performed highly accurately over these genres. The classifiers allowed us to quickly assess the POV and diegesis of the texts and show how expectations of objectivity or involvement differ across genres.
The paper proceeds as follows. In §2 we define point of view and diegesis, and discuss their different attributes. In §3 we describe the annotation of the training and testing corpus, and then in §4 describe the development of the classifiers. In §5 we detail the results of applying the classifiers to the news texts. In §6 we outline related work, and in §7 we discuss how shortcomings of the work and how it might be improved. We summarize the contributions in §8. In short, this paper asks the qeustion: can point of view and deigesis be automatically classified? The experimental results in this paper show that it can be done.

Point of View
The point of view (POV) of a narrative is whether the narrator describes events in a personal or impersonal manner. There are, in theory, three possible points of view, corresponding to grammatical person: first, second, and third person. First person point of view involves a narrator referring to themself, and implies a direct, personal observation of events. In a third person narrative, by contrast, the narrator is outside the storys course of action, looking in. The narrator tells the reader what happens to the characters of the story without ever referring to the narrator's own thoughts or feelings. In theory second person POV is also possible, although exceedingly rare. In a second person narrative, the narrator tells the reader what he or she is feeling or doing, giving the impression that the narrator is speaking specifically to the reader themselves and perhaps even controlling their actions. This is a relatively rare point of view (in our training corpus of English novels it occurred only once), and because of this we exclude it from consideration.
Knowing the point of view (first or third person) is important for understanding the implied veridicality as well as the scope of references within the text. Consider the following example: (1) John made everyone feel bad. He is a jerk.
With regard to reference, if this is part of a first person narrative, the narrator is included in the scope of the pronoun everyone, implying that the narrator himself has been made to feel bad. In this case we might discount the objectivity of the second sen-tence if we know that the narrator himself feels bad on account of John. A third person narrator, by contrast, is excluded from the reference set, one can make no inference about his internal state and, thus, it does not affect our judgment of the implications of the accuracy or objectivity of later statements.
With regard to veridicality, if the narration is third person, statements of fact can be taken at face value with a higher default assumption of truthfulness. A first person narrator, in contrast, is experiencing the events not from an external, objective point of view but from a personal point of view, and so assessment of the truth or accuracy of their statements is subject to the same questions as a second-hand report.

Diegesis
Diegesis is whether the narrator is involved (homodiegetic) or not involved (heterodiegetic) in the story. In a homodiegetic narrative, the narrator is not just the narrator but a character as well, performing actions that drive the plot forward. In a heterodiegetic narrative, the narrator is observing the action but not influencing its course. As reflected in Table 1, third person narrators are almost exclusively heterodiegetic, but first person narrators can be either. Like point of view, diegesis provides information to the reader on how to discount statements of fact, and so to judge the veridicality of the text.

Corpus
To train and test our classifiers we chose a corpus of diverse texts and had it annotated for point of view and diegesis. We used the Corpus of English Novels (De Smet, 2008), which contains 292 English novels published between 1881 and 1922, and was assembled to represent approximately a generation of writers from turn-of-the-century English literature. Novels were included in the corpus if they were available freely from Project Gutenberg (Hart, 1971) when the corpus was assembled in 2007. There are twenty-five authors represented in the corpus, including, for example, Arthur Conan Doyle, Edith Wharton, and Robert Louis Stevenson. Genres represented span a wide range including drama, fantasy, adventure, historical fiction, and romance.
To simulate "the first page" of each novel, we manually trimmed each text file so that they started with the beginning of the first chapter. This was done by hand since automating this process was not a trivial task. Then, we automatically trimmed each file down to the first 60 lines, as defined by line breaks in the original files (which reflect the Gutenberg project's typesetting). These shortened texts were used by our annotators, and were the data on which the classifiers were trained and tested.
We wrote an annotation guide for point of view and diegesis, and trained two undergraduate students to perform the annotations. The first 20 books from the corpus were used to train the annotators, and the remaining 272 texts were annotated by both annotators. After annotation was complete we realized that two of the files erroneously contained text from the preface instead of the first chapter, so we removed them from our study. Minus the training and removed texts, we produced a gold-standard corpus of 270 novels annotated for point of view and diegesis.

Inter-annotator Agreement
We evaluated the inter-annotator agreement using Cohen's kappa coefficient (κ). For point of view κ was 0.635, which is considered substantial (Landis and Koch, 1977). The κ for diegesis is 0.592, almost substantial. Out of 270 markings, there were 36 and 33 conflicts between the annotators for POV and diegesis respectively. The first author resolved the conflicts in the POV and diegesis annotations by reading the text and determined the correct characteristic according to the annotation guide. We release this gold-standard corpus, including the annotation guide, for use by the community. 2 As we expected, there are no third person homdiegetic texts in the training corpus. Although in principle this is possible, it is narratively awkward, requires the narrator to be involved in the action of the story (homodiegetic), but report the events from a dispassionate, third-person point of view, never referring to themselves directly. Our data imply that this type of narrator is, at the very least, rare in turn of the century English literature. More generally, from our own incidental experience of narrative, we would expect this be quite rare across narrative in general.

Preprocessing
The preprocessing was the same for both classifiers. The full text of the first 60 lines of the first chapter was loaded into a string, then all text within quotes was deleted using a regular expression. For both POV and diegesis it is important to focus on language that is uttered by the narrator, whereas quoted text represents words uttered by the characters of the narrative. The benefits of removing the quoted text is shown in Tables 3 and 4. After we removed the quoted text, we used the Stanford CoreNLP suite to tokenize and detect sentence boundaries (Manning et al., 2014). Finally, we removed all punctuation 4 ). This produced an array of tokenized sentences, ready for feature extraction.

Experimental Procedure
To determine the best sets of features for classification, we conducted two experiments, one each for POV and diegesis. In each case, texts were preprocessed as described above ( §4.1), and various features were extracted as described below. Then we partitioned the corpus training and testing sets using ten-fold cross-validation. Precisely, this was done as follows: for POV, the texts annotated as first person were divided into ten sets containing nearly equal numbers of texts, and we did the same for the third person texts. Then the first set of both the first person and third person texts were designated as the test sets and the classifier was trained on the remaining nine sets from each class. This was repeated with each set (second, third, fourth, etc. . . . ), designating each set in order as the test set, with the remaining sets used for training. There are more third person narrators in the corpus; hence, each training fold has more examples of third person narrators than first person narrators. We performed cross-validation for diegesis in exactly the same manner. We then trained an SVM classifier on the training fold using specific features as described below (Chang and Lin, 2011). To evaluate performance of the classifiers we report macro-averaged precision, recall, and F 1 measure. This is done by averaging, without any weighting, the precision, recall, and F 1 from each fold. We also report the average of F 1 for overall performance (weighted by number of texts).

Determining the Best POV Feature Set
The best set of features for point of view should be straightforward: narrators either refers to themselves (first person) or they don't (third person). Naturally, a first-person narrators will refer to themselves with first person pronouns, and so the presence of first person pronouns in non-quoted text should be a clear indicator of a first person point of view. Importantly, as soon as a narrator uses a first person pronoun they become a first person narrator, regardless of how long they were impersonally narrating. A list of the sets of first, second, and third person pronouns that we used as features can be found in Table 2.
We investigated eight different features sets for POV classification. The classifier with the best per-1st I, me, my, mine, myself, we, us, our, ours 2nd you, your, yours 3rd he, him, his, she, her hers, they, them, theirs formance uses counts of the first, second, and third person pronouns as the feature set. Six of the remaining experiments use different subsets of the pronouns: we test the performance of on each individual set of pronoun as well as each combination of two pronouns sets. Features sets that did not consider first person pronouns were unable to classify first person narrations, but, importantly, first person pronouns alone were not the best for classifying first person narratives. The classifier that considers all three types of pronouns has an F 1 almost six percentage points higher than the classifier that only considers first person pronouns.
Previously we discussed that it is important to remove quoted text before the features are extracted. To test this we ran an experiment where we did not remove quoted text in preprocessing, and then used all pronouns as in the best best performing classifier. This negatively impacted F 1 for first person narrators by 13 percentage points and the F 1 for third person narrators by about 3 percentage points. This shows that it is important to remove quoted text before extracting features for POV classification. The only feature sets that did worse than the feature set with quoted text removed were those feature sets that did not include first person pronouns.

Determining the Best Diegesis Feature Set
Pronouns are also a prominent feature of diegesis, but it is not as simple as counting which pronouns are used: diegesis captures the relationship of the narrator to the story. On the one hand, if the narrator never refers to themself (i.e., a third person narrator), then it is extremely unlikely that they are participating in the story they are telling, and so they are, by default, a heterodiegetic narrator. On the other hand, first person narrators may be either homo-or heterodiegetic. In this case one cannot merely count the number and type of pronouns that occur, but must pay attention to when first person pronouns, which represent the narrator, are used as arguments of verbs that represent events in the story. Event de-tection is a difficult task (Verhagen et al., 2007), so we focus on finding when first person pronouns are used as arguments of any verb. While in reality not all verbs represent events, a large fraction do, and as the performance of the classifier shows this feature correlates well with the category. To find the arguments of verbs, we use our in-house semantic role labeler (SRL) that is integrated into the Story Workbench (Finlayson, 2008; Finlayson, 2011).
We tested four different sets of features for diegesis classification. The simplest counts how many times each first person pronoun appears in an argument of a verb. Although this classifier is somewhat successful, it is somewhat weak identifying homodiegetic narrators.
The best performing diegesis classifier uses occurrences of the first, second, and third person pronouns in addition to the features from the simple diegesis classifier as features. We hypothesized that we could further improve the performance of this classifier by including a feature that counted the occurrences of second and third person pronouns as arguments of verbs that also have a first person pronoun as an argument (this is listed as the "cooccurrence" feature in Table 4). Our reasoning was that this feature would encode where the narrator and another character were connected by the same event, which is indicative of homodiegesis. Contrary to our expectations, however, this feature undermined homodiegetic classification: this classifier could not train an SVM model that could recognize homodiegetic narrators. This was the weakest of all of the diegesis classifiers.
Above we claimed that removal of quoted text is useful for diegesis classification. To show this, we took the feature set from our best diegesis classifier (with first person pronouns as arguments to a verb, and the occurrences of all pronouns), and took out the quoted text removal from the pipeline. This caused the F 1 measure to drop over 13 percentage points for homodiegetic and approximately 2 percentage points for heterodiegetic. These drops in performance indicate that the classifier performs better when quoted text is removed.

Application of the Classifiers to News
To reveal the relationship of POV and diegesis to news story genres, we applied both classifiers a diverse set of news corpora. The classifiers for these experiments were trained on all 269 first and third person texts from the CEN, 5 using the best performing sets of features. We applied the classifiers to texts drawn from five corpora: the Reuters-21578 newswire corpus, 6 a corpus of scientific press releases scraped from EurekAlerts, a selection of opinion and editorial articles scraped from LexisNexis, the Spinn3r web blog corpus ( . (2012). These five corpora are used for testing the POV and diegesis classifiers; these corpora are not used for training the classifiers. For each experiment in this section, the best set of POV and diegesis features from S4.3 and §4.4, were used to train a classifier, these classifiers were trained on the first page of each novel from the CEN. For each corpora, after running the classifiers we randomly sampled texts and checked their classification to produce an estimate of the true accuracy of the classifiers. Sample sizes were determined by calculating the number of samples required to achieve a 99% confidence for a point estimate of proportion, using the proportion estimated by the classifier (Devore, 2011). In all cases the ratio of first person to third person texts (and homo-to hetero-diegetic texts) was chosen to be equal to the ratio in the classification.

Reuters-21578 Newswire
This corpus contains 19,043 texts, and all but one were marked by the classifiers as third person and heterodiegetic. We expected this, as journalists typically use the third person POV and heterodiegetic narration to communicate objectivity. The erroneous classification of one text as first person was the result of a type of language we did not anticipate. The article in question uses direct speech to quote a letter written by Paul Volcker, Federal Reserve Board chair, to President Ronald Reagan. The majority of the article is the text of the letter, where Volcker repeatedly refers to himself, using the pronoun "I". The POV classifier interpreted this document at 1st person because the text of Volcker's letter was not removed in the quotation removal phase. The letter is quoted using direct speech, which our simple, regular-expression-based quotation detection system cannot recognize.
To estimate the true accuracy of the POV classifier over the Reuters corpus we randomly sampled and checked the POV of 200 texts (including the single first person text). All of the classifications were correct except the single first person text, resulting in an accuracy estimate of 99.5% over the newswire text for the POV classifier (1.3% margin of error at 99% confidence).
To estimate the true accuracy of the diegesis classifier over this corpus we randomly sampled and checked the diegesis of 200 texts (including the single homodiegetic text). Of the 199 heterodiegetic texts, all were correct, while the single homodiegetic text was incorrect, resulting in an accuracy estimate of 99% for the diegesis classifier over the newswire text (1.81% margin of error at 99% confidence).

EurekAlert Press Releases
This corpus contains 12,135 texts scraped from Eu-rekAlert 7 , dated between June 1st and December 31st, 2009. The distribution of this corpus is similar to the Reuters corpus, and over 99% of the texts were classified as third person and heterodiegetic narrations. Press offices write press releases to entice journalists to write newswire articles, and so it makes sense that they will attempt to mimic the desired narrative distance in the press release, seeking to present themselves as unbiased narrators.
To estimate the true accuracy of the POV classifier over the press releases we randomly sampled and checked the diegesis of 120 texts, including two first person and 118 third person. Of the two first person texts, one was correct, and of the 118 third person texts, 115 were correct, resulting in an accuracy estimate for the POV classifier of 97% over the press release text (4.03% margin of error at 99% confidence).
To estimate the true accuracy of the diegesis classifier over this corpus we randomly sampled and checked the diegesis of 120 texts, including 2 homodiegetic and 118 heterodiegetic. Of the two homodiegetic texts, neither were correct, and of the 118 heterodiegetic texts, 111 were correct, resulting in an accuracy estimate for the diegesis classifier of 94% over the press release text (5.6% margin of error at 99% confidence).

LexisNexis Opinions and Editorials
This corpus comprises 4,974 texts labeled opinion or editorial scraped from the LexisNexis website 8 , dated between January 2012 and August 2016. Texts were included if they contained more than 100 words and appeared in one of a set of major world publications including, for example, the New York Times, the Washington Post, and the Wall Street Journal. About one-quarter of these texts are first person, and more than half of the first person narrators were homodiegetic. We expected this increased abundance of first person and homodiegetic texts, as the purpose of these types of articles is often to express individual opinions or the writer's personal experience of events.
To estimate the true accuracy of the POV classifier over the LexisNexis articles, we randomly sampled and checked the POV of 200 texts, 50 from those classified as first person and 150 from those classified as third person. Of the 50 texts classified as first person all were confirmed correct, while of the 150 texts classified as third person only 90 were confirmed correct. This suggests that our classifier is not properly identifying all of the first person narrators in the LexisNexis corpus, and results in a accuracy estimate of 70% for the POV classifier over the LexisNexis texts (2.7% margin of error at 99% confidence).
To estimate the true accuracy of the diegesis classifier over this corpus we randomly sampled and checked the diegesis of 200 texts, including 24 homodiegetic and 126 heterodiegetic texts. Of the 24 homodiegetic texts, all were correct, and of the 126 heterodiegetic texts, 51 were correct, allowing us to estimate that the diegesis classifier has an accuracy of 40% over the press release text (11% margin of error at 99% confidence).

Spinn3r Web Blogs
This corpus comprises 201 stories extracted by Gordon and Swanson (2009) from the Spinn3r 2009 Web Blog corpus (Burton et al., 2009). These texts come from web blogs, where people often tell personal stories from their perspective, or use the blog as a public journal of their daily life. In contrast with newswire text, there is no expectation that a blog will report the truth in an unbiased manner. The distribution of the POV on this corpus reflects this tendency, with 66% of the texts being first person.
The diegesis distribution for the web blog stories was not unexpected: slightly more than half of the blog stories with first person narrators are homodiegetic. These are the most personal stories of the web blog story corpus, in which the narrator is involved in the story's action.
To estimate the true accuracy of the POV classifier on the Spinn3r corpus, we randomly sampled 20 texts, 13 from those classified as first person and 7 classified as third person. Of the 13 first person texts 9 were confirmed correct, while of the 7 third person texts only 3 were confirmed correct. Overall, our classifier has trouble classifying the web blog texts. This might be due to syntactic irregularities of blog posts, which vary in their degree of adherence to proper English grammar. With respect to third person narrators we estimate that the POV classifier has an accuracy of 42% over the web blog text (34% margin of error at 99% confidence).
To estimate the true accuracy of the diegesis classifier over this corpus we randomly sampled and checked the diegesis of 20 texts, including six homodiegetic and 14 heterodiegetic texts. Of the six homodiegetic texts, all were correct, and of the 14 heterodiegetic texts, three were correct. With re-spect to the heterodiegetic narrators we estimate that the diegesis classifier has an accuracy of 21% over the press release text (27% margin of error at 99% confidence).

Islamic Extremist Texts
The CSC Islamist Extremist corpus contained 3,300 story texts, as identified by Corman et al. (2012). These texts were originally posted on Islamist Extremist websites or forums. Our POV classifier found that 99.7% of the extremist stories were written in the third person. For the most part, the extremist stories were second hand accounts of events, often to share news about the outcome of battles or recount the deeds of Jihadists.
To estimate the true accuracy of the POV classifier on this corpus, we randomly sampled 150 texts, 2 from those classified as first person, and 148 classified as third person. Both of the texts classified as first person were verified to be first person narrators. Of the 148 texts classified as third person, 139 were verified correct. With respect to third person narrators, we can estimate the classifier has an accuracy of 93.9% over the extremist texts (4.92% margin of error at 99% confidence).
To estimate the true accuracy of the diegesis classifier over this corpus we randomly sampled and checked the diegesis of 150 texts, including 2 homodiegetic and 148 heterodiegetic texts. Of the 2 homodiegetic texts, 1 was correct, and of the 148 heterodiegetic texts, 137. With respect to heterodiegtic narrators, we can estimate the classifier has an accuracy of 92% over the press release text (5.6% margin of error at 99% confidence).

Related Work
As far as we know this is the first study on the automatic classification of point of view and diegesis at the level of the text. In his book "Computational Modeling of Narrative", Mani framed the problem of computational classification of narrative characteristics, including point of view and diegesis, defining with reference to narratology (Mani, 2012). He gives a framework for representing features and characteristics of narrative in his markup language NarrativeML. However, he does not actually implement a classifier for these characteristics. Person", "3rd Person", "Homo.", and "Heterodigetic" indicate the number of texts placed in each class by the classifiers trained on the CEN corpus. Percentages in parentheses indicate the fraction of that corpus falling into the specified category. The last column reports the measured accuracy of the classifiers as determined by randomly sampling and checking the results: the first percentage refers to the POV classifier and the second percentage to the diegesis classifier.
Wiebe proposed an algorithm for classifying psychological point of view in third person fictional narratives (Wiebe, 1994). The algorithm is a complex rule-based classifier which tracks broadening and narrowing of POV, and reasons whether each sentence is objective or subjective. She discusses a study where people used the algorithm to classify sentences, but the accuracy of people in that task was not given. Thus, while intriguing, it is not clear how well this algorithm performs since its correctness was not verified with a human annotated corpus.
In more recent work, Sagae et al. employed a data-driven approach for classifying spans of objective and subjective narrations (Sagae et al., 2013). Their experiments were performed on a corpus of 40 web blog posts from the Spinn3r 2009 web blog corpus (Burton et al., 2009). Their features included lexical, part of speech, and word/part of speech tag n-grams. The granularity of their classifier is fine grained, in that the system tags spans of text within a document, as opposed to our classifiers which classify the whole document.

Discussion
Our best classifier for POV uses the occurrence of all pronouns as features, with an F 1 of 0.857 for first person POV, and 0.954 for third person POV. The weighted average over the two classes is a 0.928 F 1 . Table 3 contains the results for the POV classification experiments. This is a great start for the automatic classification of POV, and comes close to human performance. It is reasonable and expected from narratological discussion that the best set of features is the number of first, second, and third per-son pronouns in non-quoted text.
The best diegesis classifier in our study, the one that counts the first person pronouns as verb arguments as well as the occurrence of each pronoun, has an F 1 of 0.721 for homodiegetic, and 0.947 for heterodiegetic. The weighted average over the two classes is a 0.898 F 1 . Table 4 contains the results for the diegesis classification experiments. This is a good first start for diegesis classification, but the performance for homodiegetic narrators falls short. The features for this classifier are also reasonable: first person pronouns in verb arguments shows that the narrator is either causing action to happen or being affected by actions, and so should naturally correlate with homodiegesis. The inclusion of all pronouns as a feature for diegesis also makes sense, as point of view and diegesis are closely correlated. As noted previously, third person narrators cannot refer to themselves, so they cannot be related to the story.
The best performing POV and diegesis classifiers performed signifcantly than their respective baseline classifiers. In Table 3, the majority class baseline classifier has 0.607 F 1 , while the best POV classifier has 0.928 F 1 . Table 4 shows that the majority baseline classifier for diegesis has 0.706 F 1 , while the best diegesis classifier has 0.898 F 1 .
Diegesis classification might be improved by restricting pronoun argument detection only to those verbs that actually indicate events in the story. This focuses the classifier on places where the narrator is involved in driving the story forward, which is more closely aligned with the definition of diegesis.
To do this, we would need to incorporate an automatic event detector (Verhagen et al., 2007, e.g.). On the other hand, event detection currently is not espe-cially accurate, and incorporating such a feature may very well depress our classification performance.
Another approach of interest would be to adapt our classifiers to detect if a narrative characteristic changes over the course of a text. Our study focused on short spans of traditional, formal, edited novels where the point of view and diegesis remained constant. In longer texts it is possible that these characteristics could change, for example, in a stream of text comprised of multiple narratives, or in a text which explicitly is trying to defy convention (e.g., in highly literary texts such as James Joyce's Ulysses).
Finally, our classifier assumed that the classified texts were all approximately the same length (i.e., the first page, or approximately 60 lines). A modification that would be important to explore is using densities or ratios for the occurrences of the pronouns, instead of raw counts, for classifying texts that are less than 60 lines long.

Contributions
In this paper, we described and made significant progress against the problem of automatic classification of narrative point of view and diegesis. We demonstrated a high performing classifier for point of view with 0.928 F 1 , and a good classifier for diegesis with 0.898 F 1 . To evaluate our classifiers we created a doubly annotated corpus with goldstandard annotations for point of view and diegesisbased on the first 60 lines-of 270 English novels. We applied these classifiers to almost 40,000 news story texts drawn from five different corpora, and show that the classifiers remain highly accurate and that the proportions of POV and diegesis they identify correlates in an expected way with the genre of the news texts. We provide the annotation guide, annotated corpus, and the software as resources for the community.