Multimedia in Concept Maps:
A Design Rationale and Web-Based Application

Sherman R. Alpert and Keith Grueneberg
IBM T.J. Watson Research Center
PO Box 218
Yorktown Heights, NY 10598 USA
{salpert, kgruen} @

In Proceedings of ED-MEDIA 2001, World Conference on Educational

     Multimedia, Hypermedia and Telecommunications.
Copyright © 2001, Association for the Advancement of Computing in Education (AACE).
Distributed on the Web by permission of AACE.
Abstract: A concept map is a graphical representation of a person’s (student’s) knowledge of a domain. Concept maps have been used in educational settings for some time and many computer-based implementations of interactive concept map building tools exist. These concept mapping tools often provide for a solely propositional, primarily textual, knowledge representation scheme and do not fully capitalize on the functionality offered by the computational medium. In this paper, we motivate the enhancement of "traditional" computer-based concept mapping tools with dynamic multimedia—sound, video, and animated images—which offer representational capabilities that are analogous to human knowledge representation mechanisms.
Concept maps
A concept map is a graphical representation of knowledge of a domain. A concept map consists of nodes representing concepts, objects, or actions, connected by directional links defining the relationships between nodes. A node is represented by a simple geometric object, such as an oval, containing a textual concept name. Inter-node relationship links are represented by textually labeled lines with an arrowhead at one or both ends. Together, nodes and links define propositions, assertions about the domain. For example, the nodes and link in Figure 1 represent the proposition "horses can gallop" and might be a portion of a concept map about horses.
Figure 1. Nodes and relationship link representing the proposition "horses can gallop."

Concept maps have been used in the educational community for decades, in virtually every subject area. In educational settings, concept maps have filled many roles: they allow students to reflect on and demonstrate their knowledge of a domain, may be used for teacher assessment of students’ knowledge of a topic covered in class, act as tools to aid study and comprehension of a domain or story, support idea generation and organization in preparation for prose composition, and are used as instructional materials for learning new concepts and their interrelationships.

This paper offers psychological and pedagogical design rationales for the inclusion of multimedia in computer-based concept maps. It also describes a concept mapping application named Webster1 whose goals include more comprehensively representing students’ knowledge of a domain, providing facilities that make concept maps more pedagogically effective for students using them to learn new concepts, and in doing both, capitalizing more fully on the capabilities of the computational medium. There are a number of characteristics of Webster that attempt to achieve these desiderata but in this paper we focus in particular on Webster’s use of multimedia to realize these goals.

Why Multimedia Concept Maps?
The purpose of concept maps is to visually represent knowledge of a subject or domain. Indeed, Jonassen (1992) claims that concept maps are accurate reflections of their authors’ cognitive structures. But this argument is based on concept maps that portray text-based propositions only. Jonassen goes on to pose the question "What constraints does the software impose on the product?" (p. 20). Older concept map tools are indeed very good at representing text-based propositional statements—wherein concepts are described by verbal means alone—but not necessarily other forms of knowledge. However, if concept maps are to be used to externally represent one’s (internal) knowledge, they should allow for nodes to be something other than of a verbal or textual nature. "It is important to note that not all nodes in a semantic memory system have names corresponding to words in natural language" (Rumelhart & Norman, 1985, p. 24). This notion is further supported by numerous researchers who have offered evidence or arguments for the notion that one’s knowledge contains more than simply verbal propositional knowledge. Kosslyn (1980), for example, proposes a cognitive model that represents information about objects with both propositional properties and images. Johnson-Laird (1983) similarly asserts that mental models of a domain include both propositions and imagery. Baddeley adds there is "abundant evidence for separate visual and verbal coding" (1985, p. 212). Shephard (1967) has, furthermore, provided evidence that memory for visual imagery is more robust than that for purely textual information. And Paivio (1986) asserts information encoded both visually and verbally is more memorable. So, to begin with, we see a need for image-based nodes in concept maps if we wish to more comprehensively represent one’s knowledge of a domain or use maps to convey information new to learners.

Intuitively, in addition to static imagery, temporally dynamic visual and auditory memories are also part of one's knowledge of an object or domain. Here we’re speaking of images in motion and sounds, both of which occur over time. Indeed, this notion has been acknowledged by cognitive science researchers and others. Johnson-Laird (1983) asserts that some mental representations are of temporally dynamic nature. Numerous studies involving music cognition provide strong empirical evidence for long-term auditory memory (see, e.g., Dowling & Harwood, 1986). Even popular culture reflects these notions; for example, as Lennon and McCartney wrote and the Beatles sang, "Penny Lane is in my ears and in my eyes," highlighting the reality that recollections involve long-term auditory and dynamic visual memories. Hence, one’s knowledge of horses might include not only what a horse looks like, but what a horse looks like when galloping and when trotting, the image of a jumping horse gliding over an obstacle, the sight and thundering sound of a group of wild horses stampeding across a sandy plain, the moving images of a wild bronco bucking in its attempt to throw a rodeo cowboy off its back, the sound of a horse "neighing." Certainly these are parts of our long-term memories for horses, and we ought to be able to represent these memories in concept maps to demonstrate our own knowledge or to use concept maps as instructional resources.

Imagery in general also offers the important capability of reifying concepts described by text. The ability to represent concrete instances and link them to text-based concepts supports the learning and memorability of those concepts. For example, Moore and Skinner (1985) demonstrated that abstract textual information is better understood and learned when presented along with reifying illustrations. The ability to portray concrete instances of concepts adds significantly to the representational and instructional potential of maps. The ability to use dynamic imagery and audio significantly enhances this representational capability. For example, here is a definition of the trotting gait of a horse: "In this most diagonal gait, the diagonal hooves lift off from and hit the ground at the same moment. [Right hind, left front, alternate with left hind, right front.] To see this, focus on a front foot, then include the diagonal hind in your field of vision. You can clearly see them lift off and hit together, in a hard trot. The sound will be a 1-2 beat" (Ziegler, 1988). Compare a concept map that represents these ideas via text-only nodes to another map that accompanies this information with a slow-motion video of a horse actually trotting. Clearly, the latter not only makes the conceptual information concrete but also provides an additional dimension to the map, namely sense-based information the text alone cannot.

The incorporation of temporally dynamic visual and aural elements in a concept map knowledge representation tool enhances its "flexibility of expressiveness" (Heeren & Kommers, 1992). More semantic and aesthetic options are open for the user to decide upon. This may help to accommodate individual differences, that is, to better support students with different expressive or learning preferences or needs when using concept maps to demonstrate their own knowledge or acquire new knowledge. And, clearly, temporally dynamic content in concept maps will better support those using maps for learning new domains and concepts because these users will be presented with: what objects look like when in motion, what they sound like, how things move or sound in particular contexts or situations, instructional visualizations of how to perform procedural tasks, videos showing dynamic interactions between objects or people.

Lastly, the use of multimedia in an interactive concept map builder and viewer may provide for a more engaging user experience. Of course, engaged and motivated learners are a desirable effect of any educational environment or tool (Vroom, 1994).

If we are using the computer to represent students’ knowledge, we should more fully benefit from what the computational medium offers over more static media, such as pencil and paper. This includes the ability to incorporate dynamic content in concept maps, thereby more comprehensively representing the knowledge a person possesses. If we are asking students to represent their knowledge, we need to be cognizant of the fact that some aspects of a student’s mental model may be expressible only through visual or auditory media. And for students using concept maps to learn about new concepts, the enhancement of those maps with dynamic media has the potential to improve the maps’ pedagogical effectiveness. Of course, one’s knowledge of a domain may also contain tactile, olfactory, or taste memories. However, standard PCs are not currently capable of representing and "playing back" such sensory information, notwithstanding ongoing research in these areas. But they are capable of manipulating video, audio, and animation.

To recap, concept map tools should be able to represent multiple types or forms of knowledge, not merely text-based propositions. Specifically, incorporating multimedia in concept mapping software should (a) provide for greater cognitive fidelity in student-constructed concept maps, allowing students to more comprehensively represent their knowledge in ways similar to their own cognitive representations; (b) offer the illustrative advantages of dynamic visual imagery and audio to students learning new concepts and domains; (c) provide the capability of reifying concepts with concrete instances that can be seen and heard; (d) offer richer expressive power for concept map authors; (e) provide for a more engaging student experience; and (f) better capitalize on functionality available in modern personal computers.

Figure 2. A Webster concept map, in a Web browser window, with a static image (the horseshoes), an animated image node (the galloping horse), an audio element (represented by the radio node), three video nodes (the television nodes), and a hyperlink to an external Web site (

Multimedia in Webster
On the basis of the preceding cognitive and pedagogical design rationale, Webster allows for the representation of dynamic media components as integral constructs of concept maps. That is, Webster provides tools for the facile incorporation of animated images, video, and audio in concept maps. In Webster, multimedia elements are first-class elements of concept maps with the same status and ease-of-use as textual concept nodes.

Figure 2 shows Webster running in a Web browser. Webster is implemented in Java™ and thereby runs as an applet in standard Web navigators, providing student access from anywhere they have access to the Internet. Along the left side of Webster’s user interface are tools for creating elements that may reside in a Webster concept map: concept nodes (of various shapes; the user may assign different semantics to different shapes as appropriate for each map), relationship links, image, audio, and video nodes, and other types of map elements. At the top of this toolkit are two mutually exclusive toggle buttons ("iconic radio buttons") that determine whether the user is in author mode (the pencil icon) or viewer mode (the eye). A user can create or modify maps while in author mode; in viewer mode, the map’s knowledge elements may not be edited and it is in this mode that multimedia elements may be viewed. When a teacher reviews a student’s concept map, he or she has access to viewer mode only for that map.

  • image
  • Figure 3. When a user clicks on a television node while in viewer mode, Webster directs its Web browser to download and play the node’s associated video file. Radio/audio nodes are handled in the same fashion.

    Continuing with our example domain, suppose a student is building a concept map to demonstrate her knowledge of horses. In addition to propositional assertions such as "a horse is an animal," "a horse can gallop," and "domesticated horses wear horseshoes" the student wishes to represent what a typical horse sounds like, what it looks like when galloping, and so on. She therefore requires the ability to incorporate visual imagery and audio in her map. To incorporate sound, she includes an audio node in the map; similarly for video and image data and their respective nodes. To add an audio node to a concept map in Webster, the user clicks on the "radio" tool button in the toolkit on the left and drops a radio (that is, audio) element onto the map. This is the same process used to add textual concept nodes (and other node types) to a map; multimedia nodes are not handled in any sort of special case fashion. Webster responds to the addition of the new audio node by popping up a list of available audio files (only audio files are shown for a radio node), allowing the user to select one to associate with the node. Image and video nodes are handled in the identical manner using the "portrait" tool and "television" tools, respectively. The multimedia file associated with an existing media node may be modified by double-clicking on the particular video, audio, or image node; Webster again displays a list of files of the appropriate media type for the user to select from. This is another example of how multimedia knowledge elements do not involve special case handling; double-clicking to edit a node’s content is precisely how textual concept and Web-link nodes are modified.

    To listen to an audio, the user (say, a teacher viewing a completed concept map) simply clicks on a radio node in the map (while in viewer mode). In response, Webster plays the associated audio file. Video playback is similarly invoked via TV nodes in a map (see Figure 3).

    With regard to images, there are other concept map tools that allow the use of static images as nodes in a map (see Related Work). Webster goes a step further in that image-based nodes in a concept map may be animated-GIF images. Thus we may have animations directly within a concept map. For example, the animated horse image in Figure 2 appears to be galloping within the map. Image nodes may be sized (stretched) directly within the map by the map author to show or hide details or otherwise accommodate aesthetic and spatial considerations. If a user wishes to view an image in its originally-created size, she may click on the image node (while in viewer mode) and the full-sized image is displayed in the browser frame that exists below the Webster applet (not shown in Figure 2).

    Webster also automates the conversion of concept maps to a different representation form, namely outlines. A common activity associated with concept map usage in the classroom is generating and organizing ideas in preparation for writing a report, composition, story, or, a more modern form of "composition," a Web page. Of course, a common representation for idea organization is an outline. Further, alternative representations support developing deeper understanding of a domain. Hence, as in the Inspiration® concept mapping product (more on Inspiration below), Webster can translate a map into an outline containing the map’s semantic content.

    In Webster, multimedia concept maps become multimedia outlines. The same static and animated images, videos, and audios that are part of a concept map appear in the associated outline representation, as shown in Figure 4. Webster converts maps into outlines as HTML documents that appear in a separate Web browser window. As in the concept map itself, the HTML generated by Webster embeds radio images for audio nodes and a TV for each video. Outlines incorporate the text labels that appear on relationship links in the map, so an outline item (a single line in the outline) might appear with the text "looks like:" followed by a TV image. In the HTML document, the radio and TV images are coded as hyperlinks that reference specific audio or video files. Hence, users simply click on embedded radio and television images to "play" them. With regard to static and animated images, they are handled much as they are in concept maps: images are embedded and visible directly within outline items, so, for example, once again the horse seems to be galloping within the outline. All images are displayed in the outline with the same default thumbnail dimensions. These thumbnail images are also encoded in the HTML as hyperlinks to the full-size image file, so to view the image in its original size, a user simply clicks on the thumbnail in the outline.

    Figure 4. The concept map shown in Figure 2 translated to a multimedia outline displayed as a Web/HTML page. The animated horse appears to be galloping, the radio and TV images are hyperlinks to audio and video files.

    Related Work
    There are many existing computer-based concept map tools, far too many for exhaustive description here. But, some bear mention. With regard to images, Inspiration and Axon®, two commercial concept mapping products, and SemNet™, a shareware tool (Fisher, 1992), do allow the use of static images as nodes in a map. Another mapping tool permits the association of static imagery with a concept node: Learning Tool™ (Kozma, 1992) has the provision for a "notecard" to be associated with each concept node, and this notecard may contain text and images. These tools certainly offer a step in the right direction beyond text-only representations. But we should be able to incorporate temporally dynamic content in concept maps as well. Further, there are significant differences in the outlines generated by Inspiration and Webster (we’ll focus only on the multimedia aspect). Even though Inspiration incorporates only static image and textual nodes in concept maps, the images are absent in its outline translation of a map, whereas Webster’s outlines incorporate all the media elements that appear in the corresponding concept map. None of these mapping tools is accessible from a Web browser.

    HandLeR is a handheld computer running proprietary software which includes a conceptual mapping tool that incorporates multimedia elements (Sharples, 2000a). In HandLeR’s "topic map" tool, topic nodes may reference the same types of media files as supported in Webster (Sharples, 2000b). There are a number of differences between HandLeR’s map tool and Webster. Topic nodes in HandLeR are all identical ovals (with embedded text), connected by unlabeled links. Hence, static and animated images are not visible within the map itself. The further implication is that nodes are not "typed"—this is, different node types are not visually distinguished—whereas in Webster concept maps, nodes provide visual cues regarding their content, including the clear indication of the existence of media nodes (video nodes appear as "televisions" and audio nodes as "radios"). We refer to this as visually typing, that is, making each node’s content type evident by its appearance. Unlike HandLeR, Webster provides an alternative outline representation that incorporates multimedia as well. Lastly, Webster is Web-enabled, providing broader student access to the tool in terms of location and time as well as eliminating the logistical problems of distributing software to individuals, while HandLeR operates on a standalone basis.

    In the IHMC CMap toolkit (IHMC, 2000) users may associate resources with a concept node. Resources may include images, sounds, or video files. Again, images do not appear directly within the visible concept map per se—users must follow a hyperlink to view an image, and this tool does not incorporate outline translations (multimedia-based or not) of concept maps. There are no labeled relationship links between verbal concepts and media nodes; media resources are, in a sense, part of the concept nodes. We feel that labeled links showing the relationships between media nodes and other nodes represent additional knowledge thereby enhancing the semantics of a map. (Link labels can be omitted in Webster if desired; making labels optional and the possible enhanced semantics when links are included increase Webster’s flexibility of expressiveness for knowledge representation—more options and decisions are left to the user’s preferences and representational requirements.) Indeed, based on studies with learning disabled students, Griffin, Simmons, and Kameenui (1992) strongly recommend that concept maps and other graphical organizers explicitly depict (i.e., label) the relationships among conceptual elements to better support the learning and memorability of the represented domain. Further, Webster attempts to make the representation of knowledge elements straightforward and simple, both conceptually and in terms of user interaction, in part by treating different types of knowledge elements in a uniform fashion. Multimedia knowledge is represented as distinct nodes which are manipulated in the same manner as are other nodes in a map. Plus, in our design rationale we stated that there are apparently distinct cognitive systems for the representation of verbal and imaginal elements; if we wish to reflect one’s knowledge in a concept map, it seems to follow that knowledge elements of these types be represented by distinct nodes rather than bundled into a single node.

    A concept map is a tool for representing knowledge. In asking students to construct concept maps, we are asking them to demonstrate and communicate to others their knowledge of a domain. We ought not limit those students by restricting the types of knowledge they can they can represent. Webster offers greater expressiveness and broader representation capabilities by incorporating multimedia elements in concept maps, thereby attempting to offer a closer match to cognitive knowledge representations. The incorporation of dynamic media also provides concomitant benefits to students using pre-constructed concept maps as tools for new concept learning, because they are exposed to a broader range of knowledge media. Lastly, a concept mapping tool should be able to translate maps into alternative representations, such as outlines which many students find useful or necessary for composing narratives. Webster outlines contain all of a map’s content including its multimedia knowledge elements.

    Baddeley, A. (1982). Domains of recollection, Psychological Review, 89, 708-729.

    Dowling, W.L. & Harwood, D.L. (1986). Music Cognition. Orlando, FL: Academic Press.

    Dunston, P.J. (1992). A critique of graphic organizer research. Reading Research and Instruction, 31(2), 57-65.

    Fisher, K.M. (1992). SemNet: A tool for personal knowledge construction. In P.A.M. Kommers, D.H. Jonassen, and J.T. Mayes (Eds.), Cognitive Tools for Learning (pp. 63-75). Berlin: Springer-Verlag.

    Griffin, C.C., Simmons, D.C., & Kameenui, E.J. (1992). Investigating the effectiveness of graphic organizer instruction on the comprehension and recall of science content by students with learning disabilities. Reading, Writing, and Learning Disabilities, 7(4), 355-376.

    Heeren, E., & Kommers, P.A.M. (1992). Flexibility of expressiveness: A critical factor in the design of concept mapping tools for learning. In P.A.M. Kommers, D.H. Jonassen, and J.T. Mayes (Eds.), Cognitive Tools for Learning (pp. 85-101). Berlin: Springer-Verlag.

    IHMC (2000). The Institute for Human & Machine Cognition (IHMC) Concept Map Software: A Knowledge Construction Toolkit.

    Johnson-Laird, P.N. (1983). Mental Models. Cambridge, UK: Cambridge University Press.

    Jonassen, D.H. (1992). Semantic networking as cognitive tools. In P.A.M. Kommers, D.H. Jonassen, and J.T. Mayes (Eds.), Cognitive Tools for Learning (pp. 19-21). Berlin: Springer-Verlag.

    Kosslyn, S.M. (1980). Image and Mind. Cambridge, MA: Harvard University Press.

    Kozma, R.B. (1992). Constructing knowledge with Learning Tool. In P.A.M. Kommers, D.H. Jonassen, and J.T. Mayes (Eds.), Cognitive Tools for Learning (pp. 23-32). Berlin: Springer-Verlag.

    Moore, P.J. & Skinner, M.J. (1985). The effects of illustrations on children’s comprehension of abstract and concrete passages. Journalof Research in Reading, 8, 45-56.

    Paivio, A. (1986). Mental Representations: A Dual Coding Approach. New York: Oxford University Press.

    Rumelhart, D.E. & Norman, D.A. (1985). Representation of knowledge, In A.M. Aitkenhead & J.M. Slack (Eds.), Issues in Cognitive Modeling. London: Lawrence Erlbaum Associates.

    Sharples, M. (2000a). Disruptive Devices: Personal Technologies and Education. Inaugural Lecture of Kodak/Royal Academy of Engineering Research Chair in Educational Technology, University of Birmingham. documents/inaugural.pdf.

    Sharples, M. (2000b). Personal email communication.

    Shephard, RN. (1967). Recognition memory for words, sentences, and pictures. Journal of Verbal Learning and Verbal Behaviour, 6, 156-163.

    Vroom, V.H. (1964). Work and Motivation. New York: Wiley.

    Ziegler, L. (1988). Working with Fox Trotters: How to tell one gait from another, Or, What is that horse doing?

    1 The name Webster is derived from the facts that our concept map tool is accessible from the Web and concerns “definitions” of knowledge, as did the lexicographer Noah Webster.

    Acknowledgments. Webster’s implementation began with JHotDraw, a set of Java classes comprising a graphical editor framework. Thanks to Erich Gamma for the JHotDraw source. Thanks to Dick Lam, Lei Kuang, and Cyndi Conway for helping integrate the Webster applet into the Wired for Learning community-based educational environment. Thanks to Dick Lam, Peter Fairweather, and Mike Sharples and his students for suggestions regarding Webster. Inspiration® is a registered trademark of Inspiration Software, Inc., Axon®, registered trademark Axon Research, SemNetTM, trademark SemNet Research Group, see JavaTM, trademark Sun Microsystems, Inc. "Penny Lane" by John Lennon and Paul McCartney: ©1967 Northern Songs Limited.