Open eBook Tests

03 August 2013

AssessmentExaminationsStudent Workload

A question from a colleague sparked this posting. What happens when the traditional paper books used in such exams or tests become eBooks?

This is not a trivial question.

New technologies provide useful opportunities to revisit models of work that we think we understand and test whether our approaches and assumptions need to be reframed. The inevitable growth in the use of eBooks as replacements for paper textbooks raises some challenging questions for academics who use open book tests in their courses, and also provides a hint of a wider need for change in our conceptions of information use, learning and assessment.

I should be upfront in declaring my biases here. Personally I think exams or tests are of marginal use in helping students learn. There are some practical advantages to be sure, exams can help detect issues of cheating or plagiarism, and they are potentially a less time-intensive way of assessing students and assigning grades. These are not particularly strong educational justifications. Exam proponents I am sure will point at the benefits of having important information memorised and able to be used under pressure as being useful, but I think the counter arguments that more important skills are the ability to maintain an awareness of how to find up to date and verifiable information efficiently, use modern information tools relevant to particular disciplines, and the ability to work effectively as part of a team. Frankly, my sense is that far too often exams and tests represent a holdover from the industrial model of education retained out of inertia or convenience rather than reflecting a carefully designed pedagogical strategy (although to be fair I do know of notable exceptions - there are never absolutes in higher education).

Setting that debate aside, the open book test has been used in a number of subjects for many years as well, and is seen as having a number of useful features. Students in many fields, such as law and medicine, need to be familiar with the structure of complex bodies of knowledge and need to be skilled at quickly and efficiently searching out key information. The vast bodies of knowledge and the need to accurately cite details trump any sense that students could or should memorise this information.

When every student had to purchase the same textbooks in the same form, the experience of an open book test could be relatively easily controlled. The biggest issues were making sure that students had not added materials that were not allowed and the noise of the books being used during the exam itself. The rise of eBook versions of textbooks as alternatives, often heavily promoted by publishers keen to destroy second-hand book sales, has seen many, but crucially not all, students adopt eBook readers such as the Kindle or iPad, or access eBooks on laptops.

The affordances of these eBooks and the reader devices are not the same as those of traditional paper. As well as providing the text, eBook software invariably provides search facilities and bookmarking that far exceed traditional indexes, post-it flags and highlighters in their speed, reliability, convenience and comprehensiveness. A small tablet device can contain the equivalent of an entire large university library. Students can search for phrases and combinations of keywords in seconds. The modern eBook is also evolving rapidly beyond the paper version with electronic simulations and other media being integrated into the 'book'. Developers are working hard on sophisticated information management tools that help people extract information from the vast corpus of knowledge and organise it in useful ways.

All of these useful features and we haven't even begun to consider the almost universal availability of networking on eBook readers and the consequent ability to use search engines and other research tools on the internet, including social networks of expertise.

This rich set of modern features, so useful in normal life, becomes problematic for academics who want to construct an artificial bounded space for their exams. A number of responses are possible and are clearly being used based on the online comments and articles appearing from academics.

One option is to abandon the open book test entirely. This can mean switching to an entirely closed exam model, which has the advantage of simplicity, but loses many things that make the process a positive educational experience, even one relevant to student learning. The other direction is to switch to a take home test model where students are given a short period of time, commonly 24 hours, and can use any resources they like to complete the tasks. This requires some additional work on the part of the academic to design assessments that discourage collusion and other forms of cheating but has the real advantage of authenticity and helps support students who work better when given time to reflect on their work.

Another option is to try to control the eBook affordances. A common question from staff is whether eBook readers be prevented from accessing the Internet? This has been plausible in the past, when devices like laptops required an physical ethernet port to access a network. The explosion in wireless networking means that increasingly the answer is no. Institutional wi-fi networks can, in theory, be managed to restrict access but in practical terms this gets more and more complicated as institutions add more access points to improve normal coverage. Ultimately these issues can be solved by the application of money and technical resources but the reality is that wi-fi networking is rapidly being overtaken by cellphone data networks (3G and 4G etc.) and these are almost completely outside of academic influence. There are devices (femtocells) that prevent cellular networks from operating but these are highly illegal in many countries. There is paint and wallpaper that can be applied to walls to prevent wireless signals, but its expensive and most exams are conducted in multi-purpose facilities that need networks and cellphones to operate normally most of the time.

A further complication at the moment is the vexed question of equity of opportunity. Most students have access to portable electronic devices but not all do, and amongst those with the devices there is considerable variation in capability. One solution is to provide a standard model of tablet or laptop for students subsidised as part of their education, but this requires an institutional commitment and as a strategy it is increasingly being abandoned as the cost of devices drops - although with notable exceptions in the United States and Australia. Very low cost tablets are starting to appear, priced well under US$100 and I expect that soon textbook publishers will be bundling the devices 'free' as part of the purchase of a bundle of textbooks.

While we work through this transition, students can be loaned devices or tests can be scheduled in computer labs, either of which seems a more positive solution than the alternative use by some institutions who have banned the electronic devices altogether and provided students who have purchased eBooks as texts with printouts of key chapters. While this is normally legal (New Zealand copyright law has a permitted use for examinations for example), it seems ridiculous and smacks of a desperation to deny the existence of digital technology for just a little longer (perhaps until the academic retires you might ask unkindly...).

There are many parallels with earlier experiences of using calculators in school mathematics classrooms. Initially they were banned, then they could be used in class but they couldn't be used in exams, then they were allowed in exams but everyone had to have the same model, and then the explosion of programmable and feature rich calculators really took off and complicated the issue even more. Teachers tried making students clear the memory of devices but savvy students simply wrote software on their calculators that dsiplayed a fake memory clear message (can we talk about how much those students learnt in the process and how well many of them have done in the modern world?). Finally mathematics teachers started teaching and assessing in ways that used the modern tools effectively to support student learning, making the questions of equity essentially irrelevant and using the tools to generate richer outcomes beyond the memorisation of times-tables.

I think we're in a transition in our understanding of how information tools are a part of normal life; part of the normal human capabilities that we should help students integrate into their lives. A significant challenge in doing so is going to be the widespread absence of such tools from many academic lives. How many of your colleagues use electronic tools in their scholarship? I don't mean just the odd database search and the use of styles in Word (but even those are not universal by any measure), I mean using tools that change the fundamental qualities of academic work and productivity, frankly obvious things like rss to discover new material online and maintain collaborations and connections, Zotero to manage source materials, NVivo to analyse qualitative evidence, and wikis to build personal knowledgebases. How many academics at your institution are actively building and maintaining an online professional presence? How many think that having a personal website and domain is a major expense and technical challenge?

Augmented reality tools and technologies like Google Glass are starting to demonstrate some basic ways by which new tools will start to augment cognition, bringing contextual information to our lives to supplement the information from our biological senses. Artificial intelligence has had a bad couple of decades following early disappointments as the initial models of cognition and understanding failed to deliver meaningful results, but increasingly we're seing ways by which computers can analyse information collected as text, sound or imagery and extract useful information. Faster computers, improved power utilisation and pervasive high speed networks are providing an enabling infrastructure that will see a proliferation of personal cognitive technologies over the next decade.

We need to be recognising the value that these tools provide for ourselves as working professionals and reflecting that in our approaches to learning. Assessment is one of the most influential tools educators have in motivating and supporting learners and we need to be using assessment techniques and settings that are authentic and support meaningful strategies, not creating artificial and unrealistic constraints that reflect nostalgia for a simpler world.

My response to the question about eBooks in open book test? Let them use anything they want, a laptop, a full internet connection, social networks, everything. By all means provide a facility to support students without their own tools, but don't cripple those of us who have started really using modern tools because of paranoia about cheating. Yes, its harder to write good test questions under these conditions, but its not that hard. Some ideas that occurred to me while writing this:

Get the students to extend an earlier piece of assessed work, such as an essay or project, by taking it in a new direction. Get them to critique their own prior work by adopting the roles of conflicting stakeholders, reanalyse it using a different set of criteria or with different tools or theories. Have them interact with a complex body of knowledge to demonstrate that they can extract meaning from it relevant to particular situations, audiences or purposes.

These are not rote exercises that can be marked automatically, they require commitment by the teacher to engage with the students at a deeper level than many tests that I remember taking, and also giving. But I also think they will stimulate far greater learning outcomes than I ever achieved by memorising facts for a test defined as regurgitation so maybe the investment is worth it.