Can you imagine that your book can recite poetry for you, or even quiz you on whether you understand the content? This is not in the world of Harry Potter; this is an AR TEXTBOOK.

Description of the application 

As Furness III said, humans are 3D spatial beings and humans work best with 3D motor control. That is the main reason that AR (Augmented reality) technology is so popular today, from gaming, industry, and marketing, to education, we can see the figure of AR applications. the emergence of AR technology allows human beings to have a third "eye": from the perspective of the camera of the smart device, see a new world created by developers and designers, which greatly expands human cognition and has enormous potential for development.

At this stage, the core advantage of AR technology is its ability to expand the amount of information, and one of the most valuable application scenarios is undoubtedly AR e-books. For example, the amount of information stored in books is limited. The amount of text and pictures is affected by the number of pages of the book and the size of the paper, and at the same time, it is not possible to display dynamic material (because this is not the world of Harry Potter).

Fig. 1.1 3D Book

Human beings have been trying to make the stories in books "come alive". Before the arrival of the information age, the " 3D book" was the main way, and it is also a prototype of AR technology. In addition, some publishers and businesses have tried to add more forms of media to books, such as adding sound to the book to realize a kind of primitive "Audio book". After the arrival of the information age, "Digital audio books" and "E-books" have become common, and as long as people have a tablet, a mobile phone or a personal computer, they can easily obtain E-books through the Internet. But some people think that e-books have no soul, they do not have the fragrance of ink, they cannot bring people the feeling of reading, and they also lack a sense of ceremony. The emergence of AR technology allows people to enjoy the excellent reading experience of physical books, and at the same time, they can also combine it with technology, so that the historical characters in the book can be seen on the paper, or to do a small QUIZ test your understanding, or to make the abstract chemical equations to show in the form of 3D ......

To summaries, by combining multiple cognitive modes, AR technology enables readers to have a deeper and more comprehensive understanding of the contents of a book, which means that AR has immense potential for teaching. Meanwhile, compared with VR (Virtual Reality) technology, AR technology is more lightweight, students do not need to carry a heavy VR Headset, they only need a mobile phone or tablet, which will not disrupt the pace of classroom education. In this development assignment, I will present a DEMO of an AR textbook that combines the Vuforia AR engine with Unity's built-in functionality to achieve an AR player, a biology model display, and an accompanying quiz. With further development, this should have the potential to help students or preschoolers consolidate their knowledge of the textbook while helping them better understand the world.

Interaction Design in AR textbook 

This design implements three main core functions:

AR text/poem reader

Fig 1.2 AR Text/poem reader

When you are reading and want to hear how a professional reciter reads the poem, searching the internet is a good option, but if just a mobile phone can instantly bring the reciter to life in your book, this can certainly provide a better experience. Through the application of AR technology, the user opens the mobile phone application and scans a specific target on the book with the camera, a video player will automatically appear, and the player can interact with it with touch control to play or pause, making it easy for the user to follow the reading. When playback is complete, the video will automatically pause and return to the video cover image, and if the user wishes to play it again, they can simply tap the play button again. In addition, I also designed a feature to lose track of the auto-pause function for this feature. If the user does not want to continue watching this video, just move away from the phone, which is more in line with the user's habits.

Compared with searching for article-related videos on the Internet, this function saves the user's search time cost. Publishers can provide books with supporting apps, and users only need to open the AR APP for scanning to get high-quality, professionally produced videos of the books, which greatly improves the experience of using the app.

AR 3D teaching aids

If you have ever taken a biology class, you must have some painful memories: the complex cell structure is a headache, the numerous organelles are difficult to understand intuitively, and the cells you see under the microscope are not as cute as you think they are. For students who enjoy biology classes, they will of course learn more about the mysteries of biology on their own, but for others, poor educational experience may cause them to lose interest further.

Fig 1.3 AR 3D teaching aid

The emergence of AR is a satisfactory solution to these problems, when you use your mobile phone to scan the pattern on the book, a clear cell model will appear, you can clearly see the structural differences between animal cells, plant cells, fungal cells and bacteria, or a close look at the shape of each organelle. At this point, you can use a microscope to observe the cells and have a deeper understanding.

Fig 1.4 AR 3D teaching aid

In implementing this feature, I took advantage of the Vuforia AR camera's ability to recognize multiple targets, allowing the camera to track multiple targets on the book for visual comparison between models. In subsequent releases, it will also be possible to zoom in, move around, and disassemble the model using Unity's built-in functionality to make the experience even more detailed.

AR Quiz

When the learning is done, the next step is to take a test to check what you have learned. Instead of taking out a pen and paper, or tapping on a web page, just scan the QUIZ page with your mobile phone to take some simple and easy but useful tests. For publishers, this design saves paper and allows tests to be taken throughout the reader's reading process (e.g., providing some revision features), which is more conducive to the reader's learning and retention. In addition, this design can also be combined with 3D modeling to make the reader's understanding more intuitive.

Fig 1.5 AR Quiz

In this assignment, I built a Canvas as a carrier for the questions and option buttons, while adding feedback for the user's choices. In addition, the user can also visualize the 3D model (options) through the lens of the phone to help them make better judgments.

Technical Development 

Vuforia configuration

In addition to the regular Vuforia settings, I made some improvements to the recognition target for the characteristics of Vuforia, such as adding some easily captured feature points to improve the recognition efficiency and creating an image database. According to the test, the recognition speed is significantly improved, and the tracking accuracy is also significantly improved.

Fig. 1.6 Enhanced image traceability

AR video player

Based on Vuforia target recognition, I used Unity's own video play plugin and scripted it to play on the click of a button and pause the video on the click of a button. The play button will automatically disappear when playback starts and reappear when playback is complete. Below are screenshots of the code and settings:

Fig.1.8 Video player setting

Fig.1.9 Video player code

In addition, by modifying the script that comes with Vuforia, I also implemented the function of losing tracking target and stopping it automatically, here is a screenshot of the script settings:

This feature still has potential for further expansion, and in subsequent releases I plan to do the following optimizations:

1. Tap on the book statement on the touch screen to automatically locate the corresponding time of the video

2. The video window can be enlarged, reduced, and moved on the touch screen.

AR 3D Teaching Aids

Since there are 4 recognition targets on this page, I changed the base settings of Vuforia so that the camera can recognize 4 IMAGE TARGETS at the same time, which makes it easy for users to visually compare the differences between the four cell models. Below is a screenshot of the settings details:

Fig. 1.10 Vuforia Max Track 

Additionally, I added rotation scripts for each cell model to easily change the speed of rotation, below are screenshots of the settings and code:

Fig.1.11 3D Teaching aid setting

Fig.1.12 Rotation code

There is still potential for further expansion of this feature, and in subsequent releases, I plan to do the following optimizations:

1. Model detail description: Users can click on components in the model (e.g., chloroplasts in plant cells) and can view the component details individually.

2. Model control: users can slide and zoom in on the model freely.

AR QUIZ

Firstly, I built a Quiz UI (User Interface) using UI Canvas, Panel, and 4 buttons and added 4 organelle 3D models as options. These are bound to the Image target and are visible after the Image Target is recognized through the camera. Here is the structure of SCENE:

Fig. 1.13 Structure of Quiz

Functionally, I wrote a QuizManager script to manage the Quiz, including questions, options, correct answers, feedback when answered correctly or typed incorrectly. Below are screenshots of the options and the script:

Fig. 1.14 Quiz manager setting

Fig. 1.15 Quiz manager code

The feature still has potential for further expansion, and in subsequent releases I plan to do the following optimizations:

1. create a question bank and a review system to automatically count the user's wrong questions.

2. richer interaction, users can choose options such as hints, skip, etc. The system will also feed more animations.

Conclusion

This design creates a prototype of an AR textbook, implementing AR interaction through Vuforia and Unity's own functions and code, and realising the textbook's leap from flat to dynamic video to 3D interaction. In addition, this design shows the potential for further improvement, and through the continuous improvement of the application, it can achieve a deeper level of communication between the mobile application ecology and the physical book, relevant to both education and publishing.

Reference

How to make a Quiz Game with Multiple Choices in Unity 2020, viewed 24 September 2023, <

How to create an Augmented Reality App 2018, viewed 24 September 2023, <

Hot Food | POEM | The Hypnotiser | Kids’ Poems and Stories With Michael Rosen 2008, viewed 24 September 2023, <

Assets:

Biology Cells Pack | 3D | Unity Asset Store n.d., viewed 24 September 2023, <https://assetstore.unity.com/packages/3d/biology-cells-pack-97118>.

2D Casual UI HD | 2D 图标 | Unity Asset Store n.d., viewed 24 September 2023, <https://assetstore.unity.com/packages/2d/gui/icons/2d-casual-ui-hd-82080>.