Pictogrammar: an AAC device based on a semantic grammar

As many as two-thirds of individuals with an Autism Spectrum Disorder (ASD) also have language impairments, which can range from mild limitations to complete non-verbal behavior. For such cases, there are several Augmentative and Alternative Communication (AAC) devices available. These are computer-designed tools in order to help people with ASD to palliate or overcome such limitations, at least partially. Some of the most popular AAC devices are based on pic-tograms, so that a pictogram is the graphical representation of a simple concept and sentences are composed by concatenating a number of such pictograms. Usually, these tools have to manage a vocabulary made up of hundreds of pictograms/concepts, with no or very poor knowledge of the language at semantic and pragmatic level. In this paper we present Pictogrammar, an AAC system which takes advantage of SUpO and PictOntology. SUpO (Simple Upper Ontology) is a formal semantic ontology which is made up of detailed knowledge of facts of everyday life such as simple words, with special interest in linguistic issues, allowing automated grammatical super-vision. PictOntology is an ontology developed to manage sets of pictograms, linked to SUpO. Both ontologies make possible the development of tools which are able to take advantage of a formal semantics.


Introduction
Language acquisition and comprehension is difficult for people with certain language impairments. Communications based on modern technologies could play a relevant role in helping with such processes. This papers introduces one of those potential technologies with important novelties.
Autism and autism spectrum disorders (ASD), such as Asperger syndrome, are neurodevelopmental conditions diagnosed on the basis of a triad of behavioral impairments: impaired social interaction, impaired communication and restricted and repetitive interests and activities (American Psychiatric Association, 2004). Thus, communication is severely impaired in persons with acute autism. What the individual understands (receptive language) as well as what is actually spoken by the individual (expressive language) are significantly delayed or nonexistent. Deficits in language comprehension include the inability to understand simple directions, questions or commands (Lim, 2011). Furthermore, the absence of verbal communication is common or, when present, it is often very immature: "want water" instead of "I want some water, please". Some of the most popular tools used to palliate, at least partially, severe communication impairments are the so-called Systems of Augmentative and Alternative Communication (SAAC). Augmentative and Alternative Communication (AAC) involves the study and proposal of alternative communication mechanisms and, when needed, to compensate for temporary or permanent impairments, activity limitations, and participation restrictions of individuals with severe disorders of speech-language production and/or comprehension, including spoken and written modes of communication (Beukelman and Mirenda, 2005).
Some of the most popular SAACs are based on   (Andy and Lori, 1994;Andy and Lori, 2001) (Figures 1 and 2). Pictograms are images which are used to support text, making the meaning clearer and easier to understand. PECS is not only a SAAC, but a method for teaching young children or any individual with communication impairment a way to communicate within a social context. PECS has a very remarkable influence on development of most of SAACs based on pictograms.
In the most advanced phases, individuals are taught to answer questions and to comment. Additionally, descriptive language concepts such as size, shape, color, number, etc. are also taught so the student can make her message more specific by combining picture symbols. For example, "I want a big yellow ball" (Figure 3). There are several communicators inspired by PECS, such as Speak4Yourself 1 , ARaSuite 2 , SC@UT 3 , CPA 4 , e-Mintza 5 or Mind-Express 6 . The pictograms are usually categorized under families of words. The way such families are defined is a bit vague and it varies across applications. For example, the pictograms in e-Mintza are sometimes categorized according to the most usual part-ofspeech of the word such as verbs or adjectives, while other words are categorized based on meaning, such as food or friends. With this application, the user chooses every pictogram from the full set of pictograms available which is made up of several hundred or even thousands of pictograms in some cases. The main objective of this paper is to describe Pictogrammar, a complete AAC system within a framework under the following hypothesis: an upper ontology which includes a formal representation of a controlled natural language is an adequate framework for developing therapeutic and palliative tools for severe language impairments in general, and beginning communicators in particular, which refers to those people (mainly children) learning how to communicate. This idea is depicted in a very simplified way in Figure 4.
Pictogrammar is described in section 2. Sections 3 and 4 describe both SUpO and PictOntology ontologies. Then, an overview of an ontology to manage collections of pictograms, called PictOntology, is given, along with how it works with SUpO, an ontology that models the piece of language which is supported by Pictogrammar. Section 5 provides a more detailed view of the issues regarding the integration of these ontologies into Pictogrammar. We finish with some conclusions and future work.

Pictogrammar
Pictogrammar is an AAC device that follows the scheme outlined in figure 4. It is therefore necessary to define the three components: a controlled language, the ontology to model, at a conceptual level, this language, and an authoring tool to obtain access to the ontology. In the end, this authoring tool makes communication possible. More specifically, the elements involved in the platform are described as follows: • The controlled natural language is the piece of symbolic communication to be learned/mastered.
• The users are the student and a number of people such as language pathologists, family and caregivers. The student is the person to be introduced into symbolic communication, i.e. a beginning communicator.
• Simple Upper Ontology, SUpO, is the upper ontology, a semantic grammar with regard to the world in which it is suitable to communicate within a controlled language. This ontology is feasible because the size of the core vocabulary of the controlled language is relatively small and it is used to make straightforward assertions (Martínez-Santiago et al., 2015). Pict-Ontology is an ontology linked to SUpO (described in section 4).
• The authoring tool is Pictogrammar, a system of Augmentative and Alternative Communication (SAAC) based on pictograms. Pictogrammar uses as input a vocabulary made up of a set of pictograms contained in PictOntology.
Since Pictogrammar is linked to an ontology with linguistic knowledge, it provides several benefits and novel issues in message generation (when the user "writes the message by using the pictograms) and in message communication (when the system "reads" what the user wrote to transmit this message to a counterpart). These are the benefits expected: 1. In message generation: (a) Expandable. By using ontology authoring tools, it is possible to increase the knowledge of the world as the user requires. (b) Predictive semantic grammar: given the construction of a sentence, the SAAC filters pictograms based on the context of the sentence. (c) Only syntactically correct sentences are allowed, by means of a controlled grammar where words are related to syntactic categories (part of speech). For instance, the pictogram related to dog could not be used as a verb. (d) Only meaningful sentences are allowed, due to syntactic correctness and topicbased relations. (e) Adaptive. The syntax and systems of meanings are adapted to the user's linguistic skills. For example, a given user could be ready to use articles and prepositions whereas other users could use just a few nouns.
2. In message communication: (a) Sentences generated using the SAAC are grammatically correct, so they are more natural. This is achieved by means of: i. Correction at the morphosyntactic level: genre and number concordance, or correct conjugation of verbal tenses. The more inflective the term is, the more impact this aspect produces.
ii. Compensating for syntactic deficiencies in the input, as much as possible. For example, the user omits pictograms about articles or prepositions, but in some cases the grammar could assign "default" values. (b) It is easy to translate to other languages and means of communication.
3. A common ontology makes it possible to share knowledge about the language model that users are able to understand and produce. In this way, a teacher (usually a language therapist) uses a computer-aid system in order to improve the verbal behavior of a given student (therapeutic software).
In addition, the same student could use an augmentative and adaptive communicator (palliative software). If the language of the student is formally modeled then it is possible that both the therapeutic and palliative software share the same knowledge, creating a homogeneous learning ecosystem. For example, if the teacher is teaching the expression I want (also know as a mand according to Skinner (Chomsky, 1959)), then the student will have access to specific exercises for this expression and, in addition, she could use the expression in her own communicator. This is the case of Pictogrammar as a system rather than as a standalone communicator: the language therapist defines every detail of the vocabulary for every student by using a cloud-based application (an application program that functions in the cloud) that, in the end, is an authoring tool used to produce new knowledge. Meanwhile, the student learns and uses such a language by using their own Android device (smartphone or tablet) with our communication app, an authoring tool to make use of the same piece of knowledge (see Figure 5).
Some of these components are detailed in section 5 when explaining the integration of Pictogrammar, SUpO and PictOntology.

SUpO, a simple upper Ontology
SUpO (Martínez-Santiago et al., 2015) models generic and basic knowledge about the world such as the main properties and uses of everyday concepts: food, toys, places, persons, etc. On the other hand, there are several upper ontologies available such as DOLCE (Masolo et al., 2003) (Descriptive Ontology for Linguistic and Cognitive Engineering), SUMO (Pease, 2006) (Suggested Upper Merged Ontology) or OpenCyC 7 . All of them formally define concepts of the world and are filled with axioms and rules for tasks which concern reasoning and planning, but they have no semantic detail, especially predicateargument structure. Since one of the highlights of SUpO is the provision of support tools to improve communicative skills, this ontology has special interest in modeling the semantic level of the language. Thus, SUpO is oriented to a more suitable form for language modeling: FrameNet (Baker et al., 1998) and the Resource Grammar Library of Grammatical Framework (GF) (Ranta, 2011).
The semantic model of SUpO is an adaptation and specialization of a small piece of FrameNet, which provides the taxonomy of concepts on which SUpO is designed.
The syntactic and morphosyntactic modules are implemented with Grammatical Frameworks and Grammatical Framework Resource Grammar Library, which is available for more than 20 languages, including English and Spanish.
An important property of Grammatical Frameworks is that they are designed to support multilingual grammars, and the translation between grammars which share the same representation of meanings is automatic. From this point of view, it is possible to define PictOntology as the SUpO version whose vocabulary is concreted by means of pictograms.

PictOntology
This section is a brief description of PictOntology, a straightforward ontology to manage a collection of pictograms linked to SUpO. PictOntology is part of Pictogrammar, but it could be reused in future applications where pictograms are involved.
Pictograms provide a visual representation of a concept. We have chosen pictograms as the visual representation of every concept included in SUpO. Thus, our ontology is made up of 621 pictograms belonging to SymbolStix 8 collection. We have chosen SymbolStix because it is a comprehensive collection of symbols (about 12,000 symbols) which covers a wide range of categories such as Geography, Sports, Logos, People, Health, Technology, Food and Drink, and many others.
The construction of PictOntology is mainly an integration process with the ontology for media resources, which is a recommendation of the W3C (the Media Resource Ontology (Champin et al., 2012)). The intent of the ontology for media resources is to bridge the different descriptions of media resources, and to provide a core set of descriptive properties. It defines a core set of metadata properties for media resources, along with their mappings to elements from a set of existing metadata formats. PictOntology reuses several concepts and attributes concerning Exchangeable Image File Format (EXIF), authoring and creator. Thus, PictOntology could be defined as a specialization of the ontology for media resources where we have established additional properties and a taxonomy of the pictograms (see Table 1).
The definition of each category in PictOntology is similar to the SUMO (Niles and Pease, 2001) idea of category: a category is made up of words which are intended to be used as constituents of the same semantic role. In any case, the PictOntology definition of category is more restrictive: a PictOntology category groups together pictograms which naturally evoke multiple ideas about the same concept (the category) and, moreover, share the same part of speech. In this way, we have defined 35 different categories. Some examples are shown in Table 2. The goals behind it are: • To simplify and make intuitive the location of every pictogram as much as possible.
• The integration of PictOntology whitin SUpO. It is possible to define PictOntology as the concretion of SUpO by using a vocabulary based on pictograms so that every PictOntology category has a one-to-one relation with a SUpO category.
• To group together similar categories, if needed. Categories are pictograms themselves, thus a category could be part of a more general category. This is useful, for example, for adapting the rendering of categories to different screen resolution (the bigger the screen, the more categories) 9 .
Since PictOntology is based on Media Resource Ontology, we have formalized PictOntology in the same way that Media Resource Ontology is formalized, i.e. by using the Web Ontology Language (OWL) (Horrocks et al., 2012).

Improving communication strategies using Pictogrammar
In this section we outline some available communicative strategies by using Pictogrammar which are possible by means of both SUpO and PictOntology. 9 The current version of the AAC device of Pictogrammar is optimized to be used on 7 inches displays. It makes possible to include 50 categories x 50 concepts per category approximately. It provides access up to 2500 concepts by implementing a twoclick category + concept pattern The textual translation of the pictogram in the group of supported languages pt:level {"transparent" "learned" "abstract"} • "transparent" symbols are very obvious depictions of the concepts that they illustrate.
• "learned": the meaning needs to be learned. The consistent nature of learned symbols means that the concepts they represent become obvious when they are shown together.
• "abstracts": symbols that have no obvious meaning when viewed on their own, and typically represent determiners or adpositions.
pt:learned group String If a pictogram is learned, then it is consistent with other pictograms relative to the same concept. This label is shared for the learned pictograms relative to the same concept. For example, in, on, under and behind are all about "relative position" pt:SUpO concepts { identifier:URI, type:List of Strings } The ID of SUpO concepts depicted by the pictogram  The American Speech Language Hearing Association recommends that an AAC should be thought of as a system comprising four components: symbols, aids, strategies, and techniques (American Psychiatric Association, 2004). Strategy is defined to be the way in which symbols can be conveyed most effectively and efficiently. During the writing of a sentence, Pictogrammar improves the strategy by reducing the number of pictograms that the user has available at a given moment. This makes possible the implementation of three complementary strategies: predictive grammar, Language Acquisition Motor Planning (LAMP) and Motor Planning overload.

Semantic predictive parser
Because of the use of a semantic grammar, the construction of phrases using Pictogrammar is supported by a predictive parser (Angelov, 2009). Note that this predictive parser is different to usual predictive text tools which are applied to the prediction of possible words when typing the first letters of the word, or the word taking into account just the grammatical role, not the semantic role. Since Pictogrammar predicts by taking into account the semantics of the phrase, the predictions are much more accurate. Moreover, the number of pictograms available while the phrase is being constructed is reduced because the user only has access to the pictograms which make sense in the sequence. For example, if a user writes "I want to eat" then Pictogrammar only shows the most popular foods, other categories of foods and properties of food. Pictogrammar knows this from the patterns and categories found in SUpO. In addition, since semantic predictive grammar only allows her to write meaningful sentences, the user receives feedback of the correct use of language at the pragmatic level, which could be relevant from a therapeutic point of view, although we have no evidence for this at the moment.

Language acquisition motor planning
A usual criticism of AAC systems based on pictograms is the high number of pictograms required as the communicative user's skills grows. Since every word has its own pictogram, the user has to spend a lot of time at first learning every pictogram, and then finding out where the pictogram is located in the SAAC. Language Acquisition Motor Planning (LAMP) (Baker, 1982), (Halloran and Emerson, 2006) is a suitable strategy for dealing with a growing vocabulary. In short, praxis or motor planning is the planning and execution of a series of movements. Language Acquisition through Motor Planning (LAMP) is a therapeutic approach based on motor planning principles that can be reinterpreted according to our purposes as follows: • The motor patterns used to "speak" with the SAAC must be consistent and unique, that is, the user should not have variants in the way she clicks and dives into categories in order to select the same pictogram in different situations.
• Each consistent pattern of one, two or three "clicks" on the SAAC must always result in the same term.
• These motor patterns are meant to reflect the consistent and unique motor patterns that result in the production of speech.
• The vocabulary sets in LAMP are organized to maintain consistent and unique motor patterns.
Consistent motor patterns for word selection allow the development of automaticity in communication but they are difficult to implement in a low-tech SAAC because of the number of pictograms. In such a case, consistently searching for the location of desired symbols and the placement of those individual symbols on a strip requires more motor planning and cognitive attention to the communication process.
Thus, the LAMP strategy is implemented in Pictogram as follows: 1. The predictive parser allows the user to hold a reduced number of icons on the main screen.
2. The only way to communicate with Pictogrammar are pictograms (even categories are pictograms) 3. Every PictOntology concept is associated with a unique motor pattern.
Furthermore, motor planning is usually implemented in systems which are not based on pictograms but on icons with several meanings depending on every motor pattern which the icon forms. But using pictograms which represent a concept or idea is a valuable resource if you are interested in teaching literacy (Flewitt, Nind, A Payler, 2009) (Lacey, Layton, Miller, Goldbart, A Lawson, 2007.

Motor planning overload
Another advantage of using motor planning by means of a predictive parser is that we are able to design more compact boards by overloading motor patterns, and this is not a minor issue because makes it possible to implement Pictogrammar for smaller screens which would make it easier to carry a device. Thanks to SUpO, it is possible to infer words which are mutually exclusive, for example apple and happy. Thus, two pictograms share the same motor pattern only if such pictograms cannot be simultaneously available. For example, two words which both represent fruits have to have necessarily different motor patterns, but if one word is about a fruit and another word is about feelings, they could share the same motor pattern since there is no way to choose between both words in a given state of the construction of a phrase. This issue is what we call motor planning overload.

Conclusion and future work
In this paper we present Pictogrammar, a SAAC based on PictOntology which has some remarkable properties from a therapeutic and palliative point of view, such as: (a) sharing of a common ontology among students, language pathologists, family and caregivers, (b) implementation of an effective predictive parser, (c) motor planning overload and (d) generation of natural language which is grammatically correct even if the input is not. We also describe PictOntology, an ontology developed to manage sets of pictograms. PictOntology is linked with SUpO, a formal semantic which is made up of detailed knowledge of facts of everyday life as simple words, with special interest in linguistic roles. In addition, it is a model of correct use of the language at the pragmatic level, since Pictogrammar only allows the user to write meaningful sentences.
As future work, we want to measure the effectiveness of this system in terms of speed of learning and size of the vocabulary acquired, evaluating the system with real users. The application this approach to other disorders regarding literacy development and oral acquisition of language is also under study. Regarding oral acquisition of language, we are encouraged in following this line since (Kasari et al., 2014) showed that speech-generating-devices improve the acquisition of oral, spontaneous, and communicative utterances in school-aged, minimally verbal children with autism.