Selecting NLP Techniques to Evaluate Learning Design Objectives in Collaborative Multi-perspective Elaboration Activities

PerspectivesX is a multi-perspective elaboration tool designed to encourage learner submission and curation across a range of collaborative learning activities. In this paper, it is shown that the learning design objectives of collaborative learning activities can be evaluated using NLP techniques, but that careful analysis of learner impact and pedagogical intent are required in order to select appropriate techniques. In particular, this paper focuses on the NLP techniques required to deliver an instructor dashboard, personalized learner feedback and content recommendation within multi-perspective elaboration activities. Key NLP techniques considered for inclusion include summarization, topic modeling, paraphrase detection and diversified content recommendation.


Introduction
PerspectivesX is a multi-perspective elaboration tool that allows instructors to create grid activities where students are able to both submit their own ideas and curate diverse ideas from other learners (i.e., add ideas submitted by other learners to their own list) using a declarative user interface. PerspectivesX allows instructors to either select a multi-perspective elaboration template such as Strengths Weaknesses Threats and Opportunities (SWOT) analysis, Six Thinking Hats (De Bono and Pandolfo, 1999) or define a custom template. PerspectivesX incorporates ideas from Computer Supported Collaborative Learning (CSCL) and the Knowledge Community of Inquiry (KCI) model (Slotta and Najafi, 2013). The tool adheres to the key principles of KCI by providing a knowledge base of student perspective submissions (Principle 1), including curation mechanics (Principle 2&3) and facilitating instructor moderation (Principle 4) (Slotta and Najafi, 2013). The tool development has been directed by clear design guidelines (Bakharia and Lindley, 2018).
NLP techniques have the potential to play an important role in collaborative learning activities, particularly at scale when a large number of learners are participating (i.e., in MOOCs or within large on-campus courses with enrollments exceeding a thousand students). In this paper, the domain of multi-perspective elaboration is used to illustrate that while Natural Language Processing (NLP) techniques are able to aid in the evaluation and implementation of key tool learning design objectives, that principled and critical analysis of learner impact is required in order to select appropriate techniques.

PerspectivesX Functionality
The PerspectivesX learner interface is shown in Figure 1. Each perspective is displayed as a grid element. Learners are able to submit new items and specify whether the item is shared with other learners, submitted as an anonymous submission or not shared. Learners are also able to view a full list of submissions for a perspective from all other learners and are able to curate items for inclusion in their own grid (i.e., perspective). On each perspective grid, the items that a learner has submitted are clearly distinguished from their list of curated items. PerspectivesX encourages active participation, idea sharing and learner knowledge growth. In particular curation, should lead to learner knowledge diversification.
The PerspectivesX tool has been implemented using React and the Django web application framework. PerspectivesX is open source and integrates with edX and other learning management

Learning Design Objectives
Learning design objectives are targeted statements about the expected student performance when participating in a learning activity. In the case of multi-perspective elaboration, the learning design objectives define the pedagogical intent of learner interaction within the PerspectivesX tool. The aim of the PerspectivesX tool is to support the following learning design objectives: 1. Encourage students to submit ideas across all perspectives 2. Encourage students to curate a list of diverse ideas within and across perspectives 3. Trigger discussion among learners in a postactivity forum 4. Encourage students to start sharing ideas (even if they initially submit ideas as anonymous or choose not to share ideas) At scale when hundreds of learners are participating in a learning activity, it becomes difficult for instructors and learning designers to evaluate whether more complex learning design objectives are being met. NLP techniques may therefore be required to serve as computational aids. Learning Design Objectives 1, 3 and 4 are clearly able to be evaluated using simple statistical clickstream analysis. The evaluation of Learning Design Objective 2, however, requires the use of NLP techniques. In the sections that follow, NLP techniques will be discussed and selected to assist both instructors in evaluating learning design objectives and learners in meeting the desired learning design objectives.

Learner Item Recommendation
The curation of diverse, original and innovative items is an intrinsic requirement of learner participation in a multi-perspective elaboration activity. NLP techniques are required to help the learner navigate the knowledge base consisting of ideas submitted by other learners.
Multiple learners may submit the same ideas but use slightly different phrases. Within the MOOC context where a cohort may reach sizes exceeding 100,000, the submission of similar items presents an information retrieval issue. The learner may need to read multiple pages of similar learner submissions. Either paraphrase detection (Socher et al., 2011) or clustering algorithms can be used in PerspectivesX. Similar student responses can be grouped together, with only the centroid submission shown to learners.
While the activity is progressing, if a learner is unable to curate a diverse list of items within and across perspectives, one of the techniques considered was diversified item recommendation. Numerous techniques and algorithms exist to find a diversified and novelty list of items in a corpus. Simply suggesting items for inclusion in the learner's curation list, however, would make the activity too easy and not require any effort from the learner. The activity only becomes beneficial to learners if they are able to view others submissions and actively decide on the items that need to be curated. Curation in itself is a key 21st-century literacy that depends on critical inquiry and exploration (Mihailidis and Cohen, 2013).
The use of a new algorithm that uses the output of a topic modeling algorithm provides a good solution to suggest a topic that a learner needs to curate items on, without suggesting the exact item. Topic modeling algorithms such as Latent Dirichlet Allocation (Blei et al., 2003) and Non Negative Matrix Factorization (Xu et al., 2003) are able to find the topics within a document collection. An item to word matrix needs to be created from student submissions and passed to the topic modeling algorithm. The output of the topic modeling algorithm is a topic defined by the top words and top documents that belong to the topic.
The algorithm implemented in PerspectivesX, matches a learners submissions to topics and suggests topics that the learner has not submitted or curated items on. As only a few top words in a topic are shown to the learner, the learner is still required to explore the knowledge base. For a future release, a topic labeling technique (Mei et al., 2007) will be included as some topics can be hard to interpret from only the top words.

The Instructor Dashboard
Submission count distributions are included for each perspective to support the evaluation of Learning Design Objective 1. The instructor dashboard currently includes a timeline chart with a series for each sharing option (i.e., shared with other learners, shared anonymously or not shared). The timeline gives the instructor an indication of learner sharing behavior over time to support the evaluation of Learning Design Objective 4. NLP techniques are however required to aid in the evaluation of Learning Design Objective 2 (i.e., "Encourage students to curate a list of diverse ideas within and across perspectives"). In order to evaluate Learning Design Objective 2, instructors require the ability to gain a high-level overview of student submissions.
Summarization, either abstractive (Luo et al., 2016) or subtractive (Rush et al., 2015), was initially considered to provide a high-level overview of learner contributions and curated items. Sum-marization techniques were however found to be inappropriate because usually only a top sentence is returned. Instructors need an indication of how learners as a group are contributing and the topics that are being covered. Topic modeling was selected as the high-level overview of topics was found to be more conducive to the aims of instructors.
Topic modeling algorithms, by returning the key topics in a collection of documents, provide a high-level overview of the topics encapsulating the items submitted by learners to a perspective. As topic modeling algorithms return both the top words and top documents (i.e., items) in a topic, the instructor is given an indication of the number of learner submissions per topic. As the top words in a topic may be hard for the instructor to interpret, the top n items are displayed where n can be specified.
While providing a high-level overview of the topics covered by learner submissions, the topic modeling algorithms (both NMF and LDA) are not able to directly return the number of students that have contributed to a topic. This is, however, determined by linking the top documents in a topic to the author (i.e., learner) of the item. Once the topics that a learner has contributed to has been determined, a distribution of the number of topics learners have contributed to, can be calculated. The distribution of learner submissions to topics provides useful insight to the instructor on whether a diverse range of topics is being addressed by learners for each perspective.
Topic modeling can also be applied to curated items as well. In particular, the topics of submitted items can be compared to the topics that are being curated. The comparison of topics gives a good indication of the level of originality in learner submissions and how ideas have diversified after curation.

Instructor Provided Feedback
The high-level overview of topics, generated by a topic modeling algorithm provides a good foundation for providing personalized feedback for learners. As learner submissions can be mapped to a topic, the instructor is able to identify common topics that learners have not been addressing. An interface for instructors to filter learners based upon submissions to a topic and send personalised feedback to learners is provided. Feedback can ei-ther be displayed on the learner's submission grid or sent via email.
Instructor feedback also needs to address misconceptions. Topic modeling algorithms, however, will include both the correct and incorrect items in the same topic if they use similar words. Providing keyword-in-context functionality will help the instructor to inspect word usage. The inclusion of keyword-in-context functionality has been shown to aid in the interpretation of topic models and allow the user to gather supporting evidence (Bakharia, 2018). Instructors are able to view and click on top words in a topic with the top words highlighted within the text of student submitted items. Paraphrase detection and clustering can also be used to group similar learner submitted items within a topic, making it easier for instructors to gain an overview of the range of items that are placed in the topic. Once misconceptions are identified, they can be linked back to contributing or curating learners, with appropriate feedback provided.
Instructor feedback can either direct learners to submit additional items for a perspective or provide guidance on the types of items that must be curated.

Discussion
While NLP techniques are able to support key learning design objectives, critical analysis of pedagogical intent must be conducted. The examples presented in this paper have shown that content (i.e., item) recommendation distracts from the purpose of curation (i.e., critical inquiry and exploration) and that a hybrid topic modeling and recommendation algorithm is instead able to meet pedagogical intent.
Both abstractive and subjective summarization were also considered as NLP options to provide instructors with a high-level overview of learner submissions per perspective. Summarization algorithms, however, produce a single sentence which would not provide the instructor with an overview of all topics being discussed and a count of learner contributions across topics. Topic modeling algorithms were selected for inclusion on the instructor dashboard but required enhancements. The output of topic modeling algorithms in particular, need to link items (i.e., documents) back to their authors (i.e., learners) to enable learner contribution and curation distribution counts per perspective to be calculated.
Topic modeling was also found to be useful in helping the instructor identify student misconceptions but only if the keyword-in-context functionality was included.

Conclusion
In this paper, the rationale underpinning the selection of NLP techniques for PerspectivesX, a collaborative multi-perspective elaboration and curation tool were discussed. NLP techniques were required to support and aid in the evaluation of a key learning design objective ("Encourage students to curate a list of diverse ideas within and across perspectives"). NLP techniques were considered for inclusion in an instructor dashboard, to help the instructor provide feedback and to recommend content items for learners to curate. Paraphrase detection and clustering were able to be used to help group together similar student submissions. A hybrid topic modeling algorithm was found to provide a viable solution for providing a high level overview of the topics learners were contributing to within a perspective for instructors. Diversified content recommendation algorithms were found to detract from the pedagogical intent of curation by making the selection of items too simplistic for learners. A hybrid topic modeling and recommendation algorithm was selected instead to recommend topics that the learner had to curate item on rather than the actual items to curate. Summarization algorithms were not selected as their single sentence output did not provide instructors with an appropriate high-level overview for analysing student contributions.