Augmented Reality in Astrid Lindgren’s landscape

10 
 
1 Introduction 
1 Introduction 
 
1.1 Augmented Reality 
Augmented reality (AR) is a term for a live direct or an indirect view of 
a physical, real-world environment whose elements are augmented by 
computer-generated sensory input, such as sound or graphics. 
Augmented reality is related to a more general concept called 
mediated reality, in which a view of reality is modified (possibly even 
diminished rather than augmented) by a computer. As a result, the 
technology functions by enhancing one’s current perception of reality. 
By contrast, virtual reality replaces the real world with a simulated one 
(Augmented Reality). 
For many years, augmented reality has been topic of study for 
researchers from all over the world who tried to enhance the real 
world by wearing sophisticated devices into augmented reality labs. 
After the smartphones revolution, research focused to implement 
mobile applications which allow to enhance and augment the reality 
no more just in the labs, but also “on the move”. 
Last years have been crucial in AR development. Many companies 
invested in developing reality augmenting software for many 
purposes, from industry to medicine, from games to e-learning 
platforms. The shared idea was to make the interaction experience 
between user and software as natural as possible. 
1.2 Prototyping 
Someone claims that prototyping is an “art”. 
“The word “prototype” derives from the Greek πρωτότυπον 
(prototypon), "primitive form", neutral of πρωτότυπος 
(prototypos), "original, primitive", from πρῶτος (protos), "first" 
and τφπος (typos), "impression".” 
(Online etymology dictionary) 
So, essentially, prototyping is the art of producing prototypes, which 
are early models of the product that is going to be built. 
In many fields, there is great uncertainty as to whether a new design 
will actually do what is desired. New designs often have unexpected
11 
 
1 Introduction 
problems. A prototype is often used as part of the product design 
process to allow engineers and designers the ability to explore design 
alternatives, test theories and confirm performance prior to starting 
the production of a new product. (Prototype) 
In general, an iterative series of prototypes will be designed, 
constructed and tested as the final design emerges and is prepared for 
production. With rare exceptions, multiple iterations of prototypes are 
used to progressively refine the design. 
A common strategy is to design, test, evaluate and then modify the 
design based on analysis of the prototype. 
The word "prototype" has sometimes the meaning of the word 
"model" and this can be confusing. Prototypes can be mainly classified 
into four categories: 
 
Proof-of-Principle Prototypes: 
Proof-of-principle prototypes are built to test the design without 
exactly simulate the visual appearance, the right materials or the final 
process. They can be used to try a potential design approach and they 
are often used to check whether a design options will work or not and 
if further development and testing is necessary. (Prototype) 
Form Study Prototypes:  
Form study prototypes are the kind of prototypes which allows 
designers to “explore the basic size, look and feel of a product without 
simulating the actual function or exact visual appearance of the 
product”. (Prototype)  
They are used to explore the product’s ergonomics and the visual 
aspect of the product's final form. This kind of prototypes is often 
“not-durable”. 
Visual Prototypes: 
Visual prototypes’ purpose is to “capture the intended design aesthetic 
and simulate the appearance, colour and surface textures of the 
intended product” (Prototype), but they don’t have the functionalities 
of the final product. They are generally used in market research. 
Functional Prototypes  (also called “ working prototypes”):
12 
 
1 Introduction 
This type of prototypes’ aim is to try to simulate the final design, look 
and feel, materials and functionalities of the product. They can have a 
different size from the final product. Engineers usually adopt 
functional prototypes to check for design flaws and to do last-minute 
improvements industrialize the product. (Prototype) 
In software engineering, prototypes have a key role in the software  
development process, first of all because they allow to get a valuable 
feedback from the users in the early stages of the project. Moreover, 
they allow to check if the design specifications are met. They  also 
allow the software engineers to do some insights into the accuracy of 
initial project estimates and whether the deadlines and milestones 
proposed can be successfully met. 
The original purpose of a prototype is to allow users of the software to 
evaluate developers' proposals for the design of the eventual product 
by actually trying them out, rather than having to interpret and 
evaluate the design based on descriptions. Prototyping can also be 
used by end users to describe and prove requirements that developers 
have not considered, and that can be a key factor in the commercial 
relationship between developers and their clients. Prototyping can also 
avoid the great expense and difficulty of changing a finished software 
product. (Software prototyping) 
The process of prototyping involves the following steps: (Software 
prototyping) 
1. Identify basic requirements : Determine basic requirements 
including the input and output information desired. Details, 
such as security, can typically be ignored. 
 
2. Develop Initial Prototype: The initial prototype is developed 
that includes only user interfaces. 
 
3. Review: The customers, including end-users, examine the 
prototype and provide feedback on additions or changes. 
 
4. Revise and Enhance the Prototype: Using the feedback both 
the specifications and the prototype can be improved. 
Negotiation about what is within the scope of the 
contract/product may be necessary. If changes are introduced 
then a repeat of steps n° 3 and n° 4 may be needed.
13 
 
2 State of the Art 
2 State of the Art 
2.1 Prototyping  general software applications 
In software engineering, prototyping is a crucial stage of software 
development. A software prototype is something comparable to 
prototypes as we know them in other fields, such as mechanical 
prototypes of engine’s components in automotive industry or buildings 
prototypes in manufacturing industry. In these terms a prototype 
typically simulates only some crucial aspects of the definitive final 
solution and may be totally different from it. 
It’s very important to create prototypes in software engineering as it 
gives immediate feedback already in the early stages of the projects. 
This is not a one way benefit on the developers’ hand: both customers 
and contractors get significant value from a prototype as it makes 
possible to check whether software requirements are met or  not. 
It also allows the software engineer some insight into the accuracy of 
initial project estimates and whether the deadlines and milestones 
proposed can be successfully met. The degree of completeness and 
the techniques used in the prototyping have been in development and 
debate since its proposal in the early 1970s. (Software_prototyping) 
2.1.1 Prototyping strategies 
A prototyping strategy is a set of principles, guidelines, plans of action 
aimed to achieve the final goal of building a useful system prototype. 
Different strategies gives different views on the same problem and 
looking at the same situation with different perspectives is a good way 
to discover usability problems. 
According to the scientific community and literature, there are several 
prototyping strategies which can be adopted to prototype software 
(but also other products). In this chapter we’ll take a look at the most 
used and well-known worldwide. 
Further technical and implementation specifications can be found in 
Jakob Nielsen’s book “Usability Engineering”. 
Horizontal prototypes 
A common term for a user interface prototype is the horizontal 
prototype. It provides a broad view of an entire system or subsystem,
14 
 
2 State of the Art 
focusing on user interaction more than low-level system functionality, 
such as database access. Horizontal prototypes are useful for: 
 Confirmation of user interface requirements and system scope 
 Demonstration version of the system to obtain buy-in from the 
business 
 Develop preliminary estimates of development time, cost and 
effort. 
Vertical prototypes 
A vertical prototype is a more complete elaboration of a single 
subsystem or function. It is useful for obtaining detailed requirements 
for a given function, with the following benefits: 
 Refinement database design 
 Obtain information on data volumes and system interface 
needs, for network sizing and performance engineering 
 Clarifies complex requirements by drilling down to actual 
system functionality 
Task-oriented prototypes 
Task-oriented prototypes are focused on the tasks the user is going to 
perform on the system. The prototype, differently from the vertical 
prototypes, is not intended as the elaboration of a single system’s 
subset, rather it represents the bunch of functionalities needed in 
order to perform a specific task.  
Scenario-based prototypes 
Scenario-based prototypes are prototypes based on a scenario. A 
scenario is the representation of the user’s behaviour in performing a 
task. A task differs from a scenario because the scenario is specific for 
a certain kind of design, and in that “it shows how a task would be 
performed if you adopt a particular design, while the task itself is 
design-independent: it's something the user wants to do regardless of 
what design is chosen” (Lewis & Rieman, 1993, p. 20) 
 
2.1.2 Prototyping methods 
Prototyping methods are the physical ways of building prototypes. 
According to the design stage there are methods that better fits. Each 
prototyping method differs from the other in terms of time needed to 
build it, of costs, of realism, of interaction with the user, of tools 
needed to implement it, of technical skills of the designer. It doesn’t
15 
 
2 State of the Art 
exist the “best” method. Each one has pros and cons, and it’s up to the 
designer to balance them and find the best trade-off for the specific 
case. 
Most of the methods’ descriptions and specifications of this chapter 
comes from the referenced book “Effective prototyping for software 
makers”. (Arnowitz, Arent, & Berger, 2006) 
Paper and pencil 
Paper and pencil prototyping is probably one of the fastest, easiest and 
cheapest way of prototyping. This method of design, refine and test 
user interfaces is particularly adapt in the earliest stages of design, 
when many design patterns are still opened and the designer is just 
exploring the many alternatives. 
Carolin Snyder has defined it as a  
“variation of usability testing where representative users 
perform realistic tasks by interacting with a paper version of the 
interface that is manipulate by a person “playing computer,” 
who doesn’t explain how the interface is intended to work”. 
(Snyder, 2003, p. 4) 
The above definition represents quite concisely this method, in which 
all is needed to start prototyping are some sheets of paper, pens, 
pencils and crayons. 
The way paper prototyping works is quite simple but not obvious. First 
of all it is necessary to identify the typical representative interface’s 
users who will run the tests on the paper prototype. Afterwards, the 
typical tasks must be determined and according with them the 
interfaces’ screen-shots and hand-sketches must be drawn. These 
includes windows, menus, buttons, pages, messages and all that is 
necessary to re-create a realistic setting of the application. When all 
the material is ready it is possible to run the usability test with the 
representative users. This is what differentiate paper and pencils 
prototypes from other kinds of prototypes, also made with paper: the 
interaction with the audience. Even though all the prototypes are just 
made of paper, there is already a possibility of interaction with the 
computer (that is actually played by the usability test administrator). 
This is not true with other kinds of prototypes also made with paper, 
like for example Storyboards and Compositions, in which there is no
16 
 
2 State of the Art 
interaction with the user. The main pros of paper prototyping are the 
following (Snyder, 2003): 
 Fast way to mock up an interface — no coding required. 
 Finds a wide variety of problems in an interface, including many 
of the serious ones. 
 Allows an interface to be refined based on user feedback 
before implementation begins. 
 A multidisciplinary team can participate. 
 Encourages creativity from the product team and users alike. 
On the other hand, cons are: 
 Does not produce any code. 
 Does not find all classes of problems with an interface. 
 Can affect the way users interact with the interface. 
 Makes some development teams nervous because they fear 
users will think it unprofessional. 
 Has stronger benefits in some situations than in others. 
Storyboards 
Storyboards are essentially a set of sketches, images or pictures 
displayed in sequence, with the purpose of illustrating the sequence of 
actions and interactions that happen during the utilization of a 
software. Storyboards are settled milestones in movies and animations 
industry, but are very effective in usability and design science too. 
Their main benefit comes from the possibility of “visual thinking” 
instead of “mental thinking”, e.g. the possibility to follow the evolution 
of the interaction of the user with the software in a visual way, like a 
comic. In this way, many people can focus on the same problem at the 
same time and they can physically see the interaction experience 
evolving. Storyboards are mainly used in the early stages of the project 
because they make brainstorming easier, as it gets easier to develop 
new ideas and immediately place them in the story. Just like in paper 
prototyping, they just need paper and pen/pencils to be created (but 
sometimes also digital illustration software are used) but they differ 
from them for the lack of interactivity between the user and the 
prototype. 
It’s quite easy to create a storyboard and, it’s important to highlight, 
that creating storyboards it’s not an artistic task. Sophisticated 
graphical elements are not required, rather they can be elements of
17 
 
2 State of the Art 
distraction. In fact, one important rule to bear in mind in the 
storyboards’ creation process is to “keep them as simple as possible”. 
It is not relevant to create a “nice-looking” or a “realistic” storyboard, 
instead it should be effective, with all the significant elements 
considered, so that the interaction sequence can be clear and easily 
understandable. 
Storyboards’ audience is both internal than external to the 
development team. Storyboards are used essentially in the first and 
mid stages of the project and their longevity is medium, because they 
are used along all the project as guidelines. They look narrative, 
medium fidelity and with conceptual expression. They are presented 
both on physical and digital media. 
Storyboards pros: 
 Helps to think “linear” 
 Useful brainstorming tool 
 Fast 
 Cheap 
Storyboards cons: 
 Not interactive 
 Prototyping technique only for the early stages of the project 
 Cannot show many usability problems 
Mock ups 
In manufacturing and design, a mock up, or mock-up, “is a scale or full-
size model of a design or device, used for teaching, demonstration, 
design evaluation, promotion, and other purposes. A mock up is called 
a prototype if it provides at least part of the functionality of a system 
and enables testing of a design.” (Mockup) 
In software engineering Mock ups are used to create user interfaces, 
to show to the final users the aspect of the product without the needs 
of creating software and routines. Mock ups can be very simple hand 
sketches, but also very high definition images, and partially working 
user interfaces. 
Mock ups are often used to create Unit tests, to be able to test one 
part of a software system (a unit) without having to use dependent 
modules. The function of these dependencies is then "faked" using 
mock objects. (Mockup)
18 
 
2 State of the Art 
Wizard of Oz 
One important tool for developing complex interactive applications is 
the “Wizard of Oz” simulation. Wizard of Oz simulation allows design 
concepts, content and partially completed applications to be tested on 
users without the need to first create a completely working system. 
(Steven, Jaemin, Christopher, Blair, Jay, & Maribeth, 2005). 
Created in the 80’s by J.Kelley, Wizard of Oz is a testing method used 
before having the system fully working. Wizard of Oz works having 
someone behind the scene who controls the actions, and the user who 
is testing the interface does not know that the responses are not 
automatic, but generated by a human (Wizard of Oz). 
How does it work:  
 human simulates system intelligence and interacts with user  
 uses real or mock interface 
o “Pay no attention to the man behind the curtain”  
 user works with system as if it were real  
 the “wizard” (sometimes hidden):  
o interprets input according to an algorithm 
o controls the user’s screen appropriately  
 wizard of oz trials are good for:  
o adding simulated and complex vertical functionalities 
o testing futuristic ideas 
o systems that would be hard to implement or change  
Wizard of Oz technique has an external audience, it is not intended to 
be used within the development team but to demonstrate the 
interface to the user. It is used essentially in the mid-term stage and 
has a quite rapid development speed. Its longevity is medium-long, it 
can be reused in several iterations and has both conceptual and 
experiential expression. The style is interactive and the fidelity 
medium-high.  
Video prototyping 
Video prototyping is defined as “a technique for visualizing the 
interactive behaviour of a system using video animation.” (Video 
prototyping) 
Video Prototyping can be a very powerful tool to communicate a 
design. However, it can also be expensive and time consuming in order 
to be produced. Video prototyping can be a very compelling way in
19 
 
2 State of the Art 
which to show a design. It allows stakeholders, users, developers and 
others to quickly understand the deign experience. Some major 
drawbacks are that it costs (time and resources) a bit more than other 
forms of prototyping. (video-prototyping) 
Video prototyping has an audience both internal than external to the 
development team. It is used in the early and midterm stage and has a 
medium-long longevity. It is expression of both conceptual than 
experimental issues and the style can be either narrative or interactive. 
It is presented on digital media and it is a medium-high fidelity 
prototype. 
Interactive simulations 
Interactive simulation is a computer based kind of prototype used to 
virtually simulate and approximate realistic behaviours of physical 
phenomenon. Interactive simulations are particularly helpful in design 
process because they allow to interact real time with the prototype 
and to edit and modify it just acting on a software, that is why they are 
commonly considered a kind of cheap prototypes even though the 
fidelity can be medium-high. 
Coded prototypes 
Coded prototypes can be created in several forms including a 
programming language, a mark-up language, or in a visual 
programming environment in conjunction with a scripting language. All 
these forms are digital interactive expressions of program code, each 
meant to portray a software product or one of its components as 
similar to the finished coded result as possible. (Arnowitz, Arent, & 
Berger, 2006) 
A coded prototype is usually developed in a programming language or 
scripting code and should be considered as a later iteration of other 
forms of low and medium-fidelity prototyping. A coded prototype has 
the advantage of allowing the software team to productively reuse 
code for an actual finished product. (Arnowitz, Arent, & Berger, 2006) 
Coded prototypes often comes late in the project. They are intended 
both for internal than external inspection and they require more time 
to be built than other prototypes. Their longevity is long, most of the 
time a coded prototype evolves into a working application/interface 
and they have a pretty experiential expression. The style is interactive, 
the fidelity is high and the medium on which they are presented is 
digital.
20 
 
2 State of the Art 
All the prototyping methods explained above are only a limited subset 
out of all the methods available in literature, but they are the most 
common and used by designers all over the world for general purposes 
applications. Of course, the more specific is the application’s domain, 
the more particular and specific will be the prototype itself. Sometimes 
designers prefer to adopt a mix among different methods or to use an 
“ad-hoc” method for the specific purpose of their application. 
 
2.2 Prototyping mobile applications: fidelity 
Mobile devices are pocket-sized computing devices typically having a 
display screen and an input system that can be a reduced keyboard, an 
extended keyboard, a touch-screen, a voice control and more 
peripherals. Mobile devices are nowadays part of our daily life; we call 
them mobile phones, smartphones, handheld, PDAs, tablets, and we 
use them every day for a broad set of activities, from communicating 
(voice calls, text messages, emails) to organize our activities and our 
businesses (agenda, to-do list, synchronizer), from taking pictures and 
videos with the embedded camera to creating and editing documents 
and, why not, to also play videogames. 
Mobile devices look like small computers, and most of them can do 
what traditional computers can. But, compared to modern laptops and 
desktop computers, they all have a limited computing capability, a 
limited amount of memory and different operating systems, with 
different environments and peripherals. These differences constraint 
how the applications running on these mobile devices are built. 
Moreover, the pace of mobile devices’ evolution and changing is 
incredibly fast and often technology, development methodologies, 
environments and tools get old and inadequate in few months. 
There are many references in literature about guidelines and 
methodologies to develop mobile application. As in “not-mobile” 
applications, e.g. desktop and laptop computer applications, an 
important aspect is reserved to usability evaluation. At this stage the 
existence of a good prototype can be essential and perhaps even more 
important than in not-mobile applications development, given the 
heterogeneous audience and the ubiquitous and pervasive 
characteristics of the mobile software itself.  
As in not mobile applications, prototypes can belong to a wide range of 
categories, according to their level of fidelity, which represents how
21 
 
2 State of the Art 
closely they resemble the final product: “low fidelity”, “medium 
fidelity” or “high fidelity”. Moreover they can be mixed together to get 
a further level called “mixed-fidelity”, which mixes some aspects of the 
former three. Let’s step through them, one by one. 
Prototyping mobile applications at low-fidelity 
In the 2006 paper “Low-Fi Prototyping for Mobile Devices” the authors  
claim that still “low-fidelity prototypes are scarcely used for mobile 
devices” (Sá & Carriço, 2006, p. 696), notwithstanding their 
effectiveness especially in the early evaluation stages of the project 
and the proved successes. Instead, still many teams prefer to adopt 
working software prototypes or specific equipment specifically 
developed and consequentially more expensive, even where not 
needed or justified. Two important aspects to keep in mind when 
prototyping mobile applications are  the context and the settings of 
use (Sá & Carriço, 2006). In fact, besides evidencing some design 
problems that could occur, they promote participatory design, 
allowing users to engage actively on the process. In fact, most of the 
innovations appeared when testing the prototypes on real situations 
and scenarios. The contextual evaluation highlight the advantages of 
the mobile devices, but also provide a deeper insight on their 
problems for those particular applications. (Sá & Carriço, 2006) 
Low fidelity prototypes in mobile applications can be as effective and 
efficient as some higher fidelity ones. They allow developers to build 
them fast and in a cheap way, making easy to test some early design 
ideas and preventing other posterior design errors happening. But it is 
also important to deep analyze the trade-off between a quick and 
cheap prototype, its pros and cons, and the same of having a medium 
or high fidelity one. When the lack of precision, the rough and raw 
interface and the fake interaction are balanced with a quick and cheap 
available prototype it can be a successful strategy to adopt a low 
fidelity prototype. 
Some important guidelines when prototyping at low fidelity can be 
summarized in the next four points: (Sá & Carriço, 2006) 
 Using rigid materials on the prototypes allows users to carry 
them, keep them in their pockets, take them home and 
participate on the sketches’ design directly on the device. 
 Context of use is essential for effective usability testing. 
Evaluation should be done in several possible scenarios.
22 
 
2 State of the Art 
 Users should be allowed to participate on the sketching process 
during contextual sessions. 
 Tasks that users will complete should be defined previously, 
but the possibility of creating new features and test them on 
the spot should be provided. 
All in all, the following list of hints should be kept in mind when using 
low fidelity prototypes in mobile applications: (Sá & Carriço, 2006) 
 Low-fidelity prototypes can be used for mobile applications as 
effectively as on desktop settings. 
 A careful distinction between the device prototypes and the 
application prototype has to be made. 
 The device prototype should be created using rigid materials in 
order to be used in real-life settings. It should have 
approximately the same size and weight of an actual device. 
 A slot where cards can be easily and quickly inserted and 
removed needs to be available. 
 The sketches must be drawn with the same size of the device’s 
screen using similar components and fonts to those available 
for a real device. 
 The interaction type of the actual device should also be 
emulated  
Prototyping mobile applications at medium-fidelity 
Medium-Fidelity Prototypes are: 
“a sort of a best-of-both-worlds implementation that allows for 
a combination of both high-level and detailed views; rapid, 
iterative changes and the ability to conduct meaningful user 
tests to evaluate complex functionality and to help determine 
system specifics.” (Prototypes) 
The medium fidelity prototypes key capabilities are:  
 higher level of interactivity than available with paper. 
 little or no coding required. 
 more accurate look & feel than low-fidelity prototypes. 
 sufficiently “unpolished” to solicit user feedback and criticism. 
In the mobile devices context, a medium fidelity prototype can be an 
adequate compromise to give the users a more realistic feedback 
about the look and feel of the application. This strategy can be