Virtual Scrolls

Beginning from its name, the medium of the handscroll evokes a tactile interaction between the viewer and the art. A literal translation from the Chinese term shoujuan (手卷), the word handscroll identifies the format as a kind of scroll suitable for the hand, for carrying and manipulation by an individual. While the important role handscrolls play as a primary format for Chinese landscape art has long been recognized, the experience of a person viewing one of these long, thin works of art has become increasingly divorced from the sense of direct handling evoked by the term handscroll. The demands of preservation and protection of these great works of art have relegated the modern viewer to experiencing handscrolls at a distance, separated by the glass of a display case or computer screen.

This project presents a novel virtual reality experience allowing viewers to directly manipulate virtual representations of Chinese landscape handscrolls. High resolution images of notable paintings were sourced from public sources like Wikimedia and museum websites. The experience was created in Unreal Engine for use with a room-scale HTC Vive setup. The VR setup was located in the CST Studiolab, and the project was completed as part of the course ART 356: Landscape and the Visual Arts in China (10th Century to 20th Century).

Background

Handscrolls are long, narrow scrolls commonly used in East Asian painting and calligraphy. They represent an important format for landscape paintings, where the exceptionally long format allows for wide, expansive vistas. Typical handscrolls measure between 20-50 cm tall and many meters long. Unlike most other forms of painting, handscrolls are not intended to be viewed all at once. Instead, the viewer would hold the handscroll in their hands and unroll sections of the painting from right to left. This kind of direct handling enforces a special kind of intimacy or connection between the artist and the viewer. Furthermore, the need to roll and unroll the painting creates a sense of rhythm and pacing. Some artists took advantage of this implicit sectioning to convey the passage of time or spatial progression. For instance, the painting "Elegant Gathering in the Apricot Garden"attributed to Xie Huan shows the progression of a gathering of scholar-officials, while "Along the River During the Qingming Festival" by Zhang Zeduan shows a journey from the countryside into a bustling city center.

However, modern viewers rarely get the opportunity to experience handscrolls in this manner. Museum displays or public exhibitions tend to unroll the complete painting for display, which turns the handscroll into a singular composition rather than a sequence of smaller sub-compositions. Viewing scanned images on a computer fails to capture the physicality of the medium, as the viewer arbitrarily zooms in and out, jumping around the digital image. Many meters of intricate landscape become compressed into the tiny viewport of a computer or projection screen.

Virtual reality technology provides a unique opportunity to replicate this unique intimate viewing experience. Specifically, room-scale VR systems that allow the user to walk around and interact within a virtual space provide an excellent foundation for creating a suitable virtual viewing room. From an interaction design perspective, the hand-held controllers could intuitively serve as analogues for the rollers on either side of the handscroll. Nevertheless, there remained many challenges that had to be solved to make this experience compelling.

VR Environment

The virtual viewing room measured approximately four by four meters and presents a simple room complete with a table and a six thumbnail images hanging on the wall. The controllers represent the two rollers of the handscroll, with the image mapped onto a plane stretched between the rollers. The viewer can roll and unroll the painting by holding onto the trigger buttons and moving the roller together and apart, just like rolling or unrolling a real handscroll. To change paintings, the viewer can walk over and touch the scroll to once of the thumbnails on the walls. The table in the virtual world is sized and positioned to match an actual table, so the viewer could pick up and set down the painting on the table as they pleased.

Tech Development

The virtual reality experience was created using the Unreal Engine. Since this was my first time working with Unreal, I mainly used the drag and drop blueprint environment and VR project templates to get started. Overall, development and learning Unreal took about 24 hours of work between first installing Unreal Engine to the version demoed above. Along the way, a number of interesting technical challenges had to be addressed.

Scroll Representation

The scroll representation itself is a simple plane, but making sure the plane actor is positioned correctly and fits well between the handles proved to be quite challenging. The plane is centered at the midpoint between the two controllers. Its rotation is set to be the average of the rotations of the two controllers, ensuring that it retains a visually reasonable orientation for typical viewing without extensive calculation. To fit between the controllers, the plane is scaled according to match the physical distance between the controllers, and the texture UV scaling changed to match.

Rolling and unrolling the image was accomplished using UV coordinate transformations. Internally, there were two modes defined for the scrolling behavior. One acted as if the left roller was held still and the right roller was being unrolled. The other mode did the same but with the roles of stationary and free roller reversed. In the final version, rolling to the left was the default mode, while rolling the right was activated by pulling the right trigger. Originally, a third mode that kept the center of the image in place and simulated both roller being free was used as the default, but playtesting showed that many people would simply try to unroll the image without using the triggers, leading to a lot of confusion.

Image Tiling

Wrapped Image (7995 × 8000 pixels)


Original Image (39974 x 1600 pixels)

Another issue related to the handling of the paintings' image textures. Extreme length is often a defining characteristic of handscrolls, with the longest painting presenting a 24.9:1 aspect ration. This presents interesting technical challenges when trying to use these high resolution scans as textures. The default maximum texture size in Unreal is 8192x8192px, and simply increasing the maximum size is undesirable for hardware compatibility and performance reasons. Scaling down the image is also undesirable, since a lot of detail must be sacrificed even though the absolute number of pixels in an 8K texture is more than enough.

To solve this problem, the long images were cut and reassembled into wrapped versions, similar to how word wrap reformats long lines of text. A Python script performed the computation of the number of rows needed for a given image and produced the wrapped image. In Unreal, a material was created to perform the UV math needed to unwrap the image. The material accepts a parameter to indicate how many rows existed in the image, and performs the necessary UV math in addition to the offset and scaling required for the scrolling effect.

Conclusion

This project was a lot of fun to work on, and presented a great introduction to using Unreal Engine and virtual reality. Bringing Chinese landscape handscrolls into virtual reality presents a unique opportunity to experience these works of art in a much more natural context and to reconsider handscroll paintings as more than flat images on a screen. The physical characteristics of the handscroll format contribute directly to the sense of intimacy and viewer involvement that sets this format apart.

While the first iteration of this virtual reality environment already presents a compelling experience, there remain many opportunities for future development. In particular, introducing different more complex environments such as a garden or palace may prove interesting in considering these paintings in their historical context. Furthermore, the virtual representation and handling of the handscroll could be made more realistic to emphasize the flexibility and textile nature of the medium. Overall, this project presents a demonstration of how technology can be used to increase the modern viewer’s engagement with art and history by turning abstract theories and imagination into tangible, interactive experiences.

Image Sources

Title	Artist	Period	Dimensions	Image Source
Along the River During the Qingming Festival	Zhang Zeduan	Northern Song	25.5 x 525 cm	Wikimedia
A Thousand Li of Rivers and Mountains	Wang Ximeng	Northern Song	51.3 × 1191.5 cm	Wikimedia
Old Trees Level Distance	Guo Xi	Northern Song	35.6 × 104.4 cm	Metropolitan Museum of Art
Pure and Remote View of Streams and Mountains	Xia Gui	Southern Song	46.5 × 889.1 cm	Wikimedia
Autumn Colors on the Qiao and Hua Mountains	Zhao Mengfu	Yuan	28.4 x 93.2 cm	Wikimedia
Dwelling in the Fuchun Mountains	Huang Gongwang	Yuan	33 x 636.9 cm	National Palace Museum Full Image