Short Talk Panel DML: Games, Learning and the Future of Assessment

Agile Development meets Evidence-Centered Design: GlassLab and the Design of Game-based Assessments

Presenter: Katie Salen

This presentation will describe the development of game-based assessments to support the learning of domain-based knowledge and skills. The design work to be discussed is being done at GlassLab, a collaboration between Electronic Arts, ESA, Institute of Play, Educational Testing Service and the Pearson Center for Digital Data, Analytics, & Adaptive Learning.

Labeled serious games, gamification, or game-like engagement, the idea of games and play as central to human experience, development, and exploration is beginning to take firm hold of the learning and assessment worlds. Whether it is Csíkszentmihályi’s idea that games induce flow, a state of heightened engagement and concentration during which learning productivity and assessment validity increase, or are used to create safe, adaptive, and engaging learning and assessment environments to manipulate otherwise time-, space-, or cost-prohibitive objects, games have matured considerably as paradigm for defining, designing, and interacting with systems. Needless to say, advances in the learning and cognitive sciences and the spreading availability of technology have opened up many opportunities for connecting the realm of (digital) games and their immersive environments, challenge-and-reward systems, and multiplayer experiences with the realm of learning and assessment.

There is still a gap to bridge between how games define the operable space for interaction and how learning and assessments (particularly formative assessments) define that. Rubin, Fein and Vandenberg lamented three decades ago that play assessment procedures lacked attention to psychometric concepts such as validity, reliability, and stability. Getting to a state of what they call “psychometric respectability” means building complete construct representation and developing a paradigm of reasoning that explicates what does and does not constitute evidence about what we aim to teach or make claims about. It also means preserving, if not establishing, the critical aspects of play that turned us toward games in the first place. The notion that games and assessments are based on the same learning principles provides the basis for bridging that gap.

On the surface, designing a game and designing an assessment are very different. Typically, they have very different purposes, design criteria, and constraints. Games are designed to create an immersive, engaging experience for the player—and above all, to be fun. Typically, assessments are designed to extract evidence from student performances to make inferences about student learning.

Beneath the surface though, there are some surprising similarities. Both involve creating carefully designed sequences of activities with allowable interactions. Both make inferences from collected responses. Both strongly connect to learning albeit in different ways.

Addressing design criteria from games and assessment simultaneously may lead to improvements in both. This talk will first outline the potential value of including criteria and constraints from games, such as engagement, choice, and feedback, in the design of assessments. It will then explore a process for the development of game-based assessments that combines aspects of evidence-centered design (ECD) with an agile game development framework. Finally, we discuss an application of this process to a game-based assessment currently under development.

Interest-Driven Learning in Game Design Environments

Presenters: Gabriella Anton, Amanda Ochsner, Kurt Squire

Online games and affinity spaces offer a vast array of literacy practices and reciprocal apprenticeship (Gee, 2003; Steinkuehler, 2007; Black & Steinkuehler, 2009; Black, 2008). Interest-driven learning in affinity spaces and games can be harnessed to facilitate the understanding of concepts in core classes (Squire & Jenkins, 2003). The learning benefits of having youth design and develop their own games is not as thoroughly researched. Previous work has suggested that video games and design practices promote a wealth of literacy and critical thinking practices, (Peppler, Warschauer, & Diazgranados, 2010; Peppler & Kafai, 2007) as well as facilitating the development of computational thinking skills like logic, debugging, and algorithm design (Berland & Lee, 2011). We created Studio K as a curriculum and online learning environment for youth that supports asynchronous, collaborative, and interest-driven learning around game design and computational thinking. Within the curriculum, students design and develop games using Kodu, a 3D visual programming language, and collaborate with other students in the site’s community. We hypothesize that interest-driven learning will be indicated in the quality and quantity of interactions a participant has within the website. We expect to find that participants who design more games and revised iterations will produce higher quality games and will show more advanced use of video game language, culture of critique, and academic literacy in their interactions online compared to participants with low numbers of games and revised iterations. The existence of these practices within informal game design curriculum suggests the benefits of such practices for the development of literacy and computational thinking.