Building an multitouch enabled music application

Innovations in computing form factors such as All-in-One (AIO) and tablet devices that combine desktop-like performance with multi-touch-enabled, high-resolution screens are giving people fun new ways to experience making music and giving new meaning to the often-said statement that music is a universal language.

Recognizing this trend, TheBestSync, a China-based software company focused on integrated software and hardware solutions, entered the Intel App Innovation Contest 2013 (Intel AIC 2013) in partnership with Intel® Developer Zone with the idea of combining game and music technology with an AIO device to create an exciting new way for people to play and enjoy music.

TheBestSync and its CEO, Alpha Lam, have been on a mission to create innovative, interactive experiences for the entertainment market for the last three years and are no strangers to Intel Developer Zone contests. They recently took the grand prize in the Intel® Perceptual Computing Challenge with their submission, JOY*, one of the first virtual music instruments built using the Intel Perceptual Computing SDK (now the Intel® RealSense™ SDK). JOY not only won the Challenge, but it also became the inspiration for TheBestSync's most recent app submission, Paint Your Music, into the AIC 2013 contest, which won the Entertainment category for AIO devices running Microsoft Windows* 8.1.

Created specifically for the Intel AIC 2013, which challenged contestants to "Dream up an interactive entertainment experience that helps make the all-in-one an endless adventure," Paint Your Music (PYM) combines multi-touch, multi-player-enabled interaction with a musical matrix game board, virtual "paint balls," and a scoring mechanism to create a unique, fun-to-play virtual musical instrument/game.

As Intel AIC 2013 finalists, TheBestSync received a Lenovo IdeaCentre Horizon 27* Table PC AIO to code PYM on. The system consisted of a 27-inch 10-point touch screen, Windows 8.1, an Intel® Core™ i7 processor, 8 GB RAM, and NVIDA GPU.

Lam and TheBestSync team were excited by the Lenovo AIO's gigantic touch screen, which played a key role in PYM. The touch screen has the ability to lay flat, giving players an arcade-style, table-top game experience. "Being able to lay it flat was especially helpful," he said. "Having such a large touch screen lets multiple people engage with it at the same time, so it's very conducive to creating immersive environments. Plus, the Lenovo's 8-cell lithium-polymer battery lets users run the device for up to two hours, adding a dimension of mobility to an otherwise large device."

The AIO's high-fidelity audio capabilities further enhance gameplay, which allows users to create their own music.

In "recreational" single-player mode, PYM functions as a virtual 3D musical instrument. Using touch, the player places "music note balls" on a matrix. Music note balls act as graphic representations of notes and behave like virtual paintballs, triggering notes and "paint dances." The AIO's multi-touch-enabled screen lets the player position notes with one hand and "spin" the matrix in 3D space (Figure 1).

Figure 1: In single-player mode, notes are positioned on a matrix by touching the screen. Users can change the angle of view in 3D space, where notes appear as pillars of color or animated "paint dances."

In "competitive" two-player mode, players launch music note balls toward their opponents' "music wall." As the balls land on the matrix they produce notes. Melodies result as more notes are launched over the course of a game. Points are scored when "props" are stuck by a ball. All the music is created on-the-fly in real time and can be recorded and played back to make music.

PYM was built using the Unity 3D Engine*. Programming was handled with Microsoft Visual Studio* and C#, an object-oriented language that the team preferred over C++ because of its seamless compatibility with Unity. For PYM's audio capabilities, TheBestSync team used the interactive framework they refined while developing JOY. The framework is based on the audio engine that ships with Unity and includes additional digital signal processing audio effects, such as reverb and echo (delay), as well as the ability to record and play back music created during gameplay.

Before founding TheBestSync, Lam spent a decade running a music production house, which afforded the PYM development team ready access to an extensive, proprietary library of instrument sounds that they were able to plug into Unity's step sequencer. The library supports high-fidelity 16-bit, 24 kHz MP3 and WMA sound files as well as 24-bit, 48 kHz WAV files that were created using Avid Pro Tools* and Apple Logic* Pro digital audio workstation software.

Figure 2: Paint Your Music's multi-player UI. (A) The music matrix. (B) Music note balls. (C) Red player "prop" launcher. (D) Blue player music wall. (E) Elapsed time display and a menu for accessing settings, choosing what kind of applause you'll hear, and other items.

PYM's user interface combines 2D and 3D assets. The 3D content was produced using Autodesk Maya*, and to simulate "paint dances," the team used Unity's particle system and physics engine.

Achieving a balance between great-looking visuals and real-time performance proved to be the team's biggest challenge. The particle system was particularly thirsty for CPU horsepower. To achieve higher frame rates, PYM's code took advantage of both the on-chip GPU in the Intel Core i7 processor and the Lenovo AIO's NVIDIA GPU (Figure 3). To further boost performance, the team reduced the number of simultaneous particles from 2,500 to about 1,000 and adjusted the size and texture of the particles.

Figure 3: Paint Your Music achieves 9 fps playback with the NVIDIA GPU engaged (left) and 15 fps when both the NVIDIA GPU and Intel® HD Graphics are used (right).

In addition to fine-tuning the particle system, TheBestSync team used multi-threading to enable PYM to run multiple processes-graphics, sound, physics-across multiple cores and threads. "PYM's real-time 3D graphics and music would not have been possible without the Intel® processor built into the Lenovo AIO platform," said Lam.

Multi-touch support was another hurdle that TheBestSync had to overcome. The Unity 3D engine doesn't support multi-touch natively, so the team used C# to script a plug-in that references the Windows 8 Touch API. The team followed Microsoft's suggested best practices for setting touch-target sizes. "The Lenovo all-in-one's touch screen is huge, so while the size of touch-targets can be problematic on small form-factor devices, it wasn't an issue for us," said Lam.

Multi-touch support was another hurdle that TheBestSync had to overcome. The Unity 3D engine doesn't support multi-touch natively, so the team used C# to script a plug-in that references the Windows 8 Touch API. The team followed Microsoft's suggested best practices for setting touch-target sizes. "The Lenovo all-in-one's touch screen is huge, so while the size of touch-targets can be problematic on small form-factor devices, it wasn't an issue for us," said Lam.

Intel® Graphics Performance Analyzers (Intel® GPA) was used to diagnose and debug PYM. Intel GPA's ability to display CPU and GPU usage visually helped the team quickly identify bottlenecks in the Microsoft DirectX*-based graphics pipeline. The team used Unity 3D's Profiler window to handle the general debugging of code related to memory usage, audio, physics, and rendering.

Formal, commercial-level beta testing has not yet been conducted.

TheBestSync plans to launch PYM by the end of 2014. Its goals include:

In addition, Lam hopes to add support for OpenGL* and OpenGL ES* to enhance the graphics pipeline's performance on mobile platforms.

Another feature the company plans to implement is in-app purchasing using different device-specific payment SDKs. Currently, TheBestSync uses the Microsoft Azure* platform for distributed cloud services, but they're investigating alternatives such as the Amazon Web Services cloud platform.

Distribution will be handled through the Microsoft App Store and other similar app stores.

Lam is considering migrating PYM to other mobile platforms, which will present new challenges, not the least of which is scaling the app's UI and graphics to accommodate small form-factor devices and screens, devices of varying capabilities, and other fragmentation issues.

Alpha Lam studied applied electronics and spent 10 years in music production before founding TheBestSync. The company currently employs 20 people. The PYM team included three software developers, three 3D graphics programmers, and an audio engineer.