Summary Cloze: A New Task for Content Selection in Topic-Focused Summarization

A key challenge in topic-focused summarization is determining what information should be included in the summary, a problem known as content selection. In this work, we propose a new method for studying content selection in topic-focused summarization called the summary cloze task. The goal of the summary cloze task is to generate the next sentence of a summary conditioned on the beginning of the summary, a topic, and a reference document(s). The main challenge is deciding what information in the references is relevant to the topic and partial summary and should be included in the summary. Although the cloze task does not address all aspects of the traditional summarization problem, the more narrow scope of the task allows us to collect a large-scale datset of nearly 500k summary cloze instances from Wikipedia. We report experimental results on this new dataset using various extractive models and a two-step abstractive model that first extractively selects a small number of sentences and then abstractively summarizes them. Our results show that the topic and partial summary help the models identify relevant content, but the task remains a significant challenge.

• Task: -Input: Document(s) -Output: Summary 2 A man in suburban Boston is selling snow online to customers in warmer states. For $89, he will ship 6 pounds of snow in an insulated Styrofoam box.
(CNN) The only thing crazier than a guy in snowbound Massachusetts boxing up the powdery white stuff and offering it for sale online? People are actually buying it. For $89, self-styled entrepreneur Kyle Waring will ship you 6 pounds of Boston-area snow in an insulated Styrofoam box -enough for 10 to 15 snowballs, he says. But not if you live in New England or surrounding states. "We will not ship snow to any states in the northeast !" says Waring 's website… reference document summary Traditional Summarization • Task: -Input: Document(s) -Output: Summary • Popular datasets -CNN/DailyMail (Nallapati et al., 2016) -New York Times (Sandhaus, 2008) - Newsroom (Grusky et al., 2018) 3 A man in suburban Boston is selling snow online to customers in warmer states. For $89, he will ship 6 pounds of snow in an insulated Styrofoam box.

Roger Federer, Philanthropy and outreach
The current world number two tennis player, who was born in Basel, received the title from the medical faculty, which praised him for his role in increasing the international reputation of Basel and Switzerland…In a press release it also highlighted the 36-yearold sportsman's engagement for children in Africa through his charitable foundation… topic reference document context cloze input output • Propose a cloze-style task -Topic-focused summarization

This Work
• Propose a cloze-style task -Topic-focused summarization • Collect large-scale cloze dataset from Wikipedia 9 … Federer received an honorary doctorate awarded to him by his home university, the University of Basel. He received the title in recognition for his role in increasing the international reputation of Basel and Switzerland, and also his engagement for children in Africa through his charitable foundation.

Roger Federer, Philanthropy and outreach
The current world number two tennis player, who was born in Basel, received the title from the medical faculty, which praised him for his role in increasing the international reputation of Basel and Switzerland…In a press release it also highlighted the 36-yearold sportsman's engagement for children in Africa through his charitable foundation…

Nate Morris, Philanthropy Topic
The Lewis Sexton Scholarship Fund was established in 2016 by the Morris Industries Foundation in honor of Lewis Sexton , former President of UAW Local 862 . The Scholarship Fund will provide three awards of $ 5,000 non -renewable scholarships to children ( or legal wards ) of UAW 862 members or retirees who plan to attend college in the Commonwealth of Kentucky… In 1971 , Lewis became the first Committeeman to be elected President and Building Chairman of UAW Local 862 and served in that capacity for ten years .

Extractive
Step in 1971 , lewis became the first committeeman to be elected president and building chairman of uaw local 862 .

Ground-Truth
Programs funded by the foundation include the Lewis Sexton Scholarship Fund, which provides college scholarships in Kentucky to children of UAW Local 862 members or retirees.

Topic
The Lewis Sexton Scholarship Fund The Scholarship Fund will provide three awards of $ 5,000 non -renewable scholarships to children ( or legal wards ) of UAW 862 members or retirees Extractive Step

Ground-Truth
Programs funded by the foundation include the Lewis Sexton Scholarship Fund, which provides college scholarships in Kentucky to children of UAW Local 862 members or retirees.
Extractive step contains ground-truth information Step in 1971 , lewis became the first committeeman to be elected president and building chairman of uaw local 862 .

Ground-Truth
Abstractive step selects wrong information, not about philanthropy (not even about the correct person)

Future Research Directions
• Evaluation -Evaluation is a challenge across summarization, even worse with one sentence -Many plausible endings, so how do we choose one over another?