Demonstrations such as the Augmented Reality Browser Demo show how browser based technologies can implement simple augmented reality demonstrations. By building on a browser’s ability to access connected camera feeds, we can reuse third party libraries to detect and track registration images contained within the video feed and 3D plotting libraries to render and overlay 3D objects on the tracked image in real time.
But what if we could also capture information from a modified registration image and use that as part of the rendered 3D model?
A research paper by Disney research – Live Texturing of Augmented Reality Characters from Colored Drawings [PDF] – presented at the International Symposium on Mixed and Augmented Reality (ISMAR 2015) describes “an augmented reality coloring book App in which children color characters in a printed coloring book and inspect their work using a mobile device”, since released as the Disney Color and Play app.
But Disney is not the only company exploring augmented reality colouring books…
Another app in a similar vein is produced by QuiverVision (coloring packs) and is available for iOS and Android devices.
And as you might expect, crayon companies are also keen on finding new ways to sell more crayons and have also been looking at augmented reality colouring books, as in the case of Crayola and their ColorALive app.
DO: grab a coffee and some coloured pen or pencils, install an augmented reality colouring book app, print off an AR colouring pack, then colour in your own 3D model. Magic!:-)
Now compare and contrast augmented reality applications in which a registration image, once captured, can be used to trigger a video effect or free running augmented reality animation with augmented reality applications in which a registration image of environmental feature must be detected and tracked continually in terms of the technology required to implement them, the extent to which they transform a visual scene and the uses to which each approach might be put. Try to think of one or two examples where one technique might be appropriate but the other would not when trying to achieve some sort of effect or meet some particular purpose.
In the post Noise Cancellation – An Example of Mediated Audio Reality? we saw how background or intrusive environmental noise could be removed using noise cancelling headphones. In this post, you’ll learn a simple trick for diminishing an audio reality by removing a vocal track from a musical jingle.
Noise cancellation may be thought of adding the complement of everything that is not the desired signal component to an audio feed in order to remove the unwanted noise component. This same idea can be used as the basis of a crude attempt to remove a mono vocal signal from a stereo audio track by creating our own inverse of the vocal track and then subtracting it from the original mix.
SAQ: Describe an algorithm corresponding to the first part of method suggested in the How to Remove Vocals from a Song Using Audacity video for removing a vocal track from stereo music track. How does the algorithm compare to the algorithm you described for the noise cancelling system?
SAQ: The technique described in the video relies on the track having a mono vocal signal and stereo backing track. The simple technique also lost some of the bass when the vocals were removed. How was the algorithm modified to try to preserve the bass component? How does the modification preserve the bass component?
In The Art of Sound – Algorithmic Foley Artists?, we saw how researchers from MIT’s CSAIL Lab were able to train a system to try to recreate the sound of a silently videoed object being hit by a drumstick using a model based on video+sound recordings of lots of different sorts of objects being hit by a drumstick. In this post, we’ll see another way of recovering audio information from a purely visual capture of a visual scene, also developed at CSAIL.
Fans of Hollywood thrillers or surveillance-themed TV series may be familiar with the idea of laser microphones, in which laser light projected onto and reflected from a window can be used to track the vibrations of the window pane and record the audio of people talking behind the window.
Once the preserve of surveillance agencies, such devices can today be cobbled together in your garage using components retrieved from commodity electronics devices.
The technique used by the laser microphone is based on measuring vibrations caused by sound waves relating to the sound you want to record. Which suggests that if you can find other ways of tracking the vibrations, you should similarly be able to retrieve the audio. Which is exactly what the MIT CSAIL researchers did: by analysing video footage of objects that vibrated in sympathy (albeit minutely) to sounds in their environment, they were able to generate a recovered audio signal.
As the video shows, in the case of capturing a background musical track, whilst the audio was not necessarily the highest fidelity, by feeding the input into another application – such as Shazam, an application capable of recognising music tracks – the researchers were at least able to identify it automatically.
In the post Augmented TV Sports Coverage & Live TV Graphics, we saw how live TV graphics could be used to overlay sports events in order to highlight particular elements of the sports action.
One of the thing things you may have noticed in some of the broadcasts was that as well as live “telestrator” style effects, such as highlighting the trajectory of a ball, or participant tracking effects, many of the scenes also included on pitch advertising. So was the pitch really painted with large adverts, or were they digital effects? The following showreel from Namadgi Systems (which in its full form demonstrates many of the effects shown in the previously mentioned post) suggests that the on pitch advert are, in fact, digital creations. Other vendors of similar services include Broadcast Virtual and BrandMagic.
So-called virtual advertising allows digitally rendered adverts to be embedded into live broadcast feeds in a way that makes the adverts appear as if they are situated on or near the field of play. As such, to the viewer of the broadcast, it may appear as if the advert would be visible to the spectators present at the event. In fact, it may be the case that the insert is an entirely digital creation, an overlay on top some sort of distinguished marker or location (determined relative to an easily detected pitch boundary, for example), or a replacement of a static, easily recognised and masked local advert.
EXERCISE: Watch the following video and see how many different forms of virtual advertising you can detect.
So how many different ways of delivering mediated reality ads did you find?
The following marketing video from Supponor advertises their “digital billboard replacement” (DBRLive) product that is capable of identifying and tracking track or pitched advertising hoardings and replacing them with custom adverts.
EXERCISE: what do you think are the advantages of using digital signage over fixed advertising billboards? What further advantages do “replacement” techniques such as DBRLive have over traditional digital signage? To what extent do you think DBRLive is a mediated reality application?
As well as transforming the perimeter, and event the playing area, with digital adverts, sports broadcasters often present a mediated view of the studio set inhabited by the host and selected pundits to provide continuity during breaks in the action, as the following corporate video from vizrt describes:
So how do virtual sets work and how do they compare with the “chroma key” effects used in TV and film production since the 1940s? We’ll need another post for that…
In the post Augmented TV Sports Coverage & Live TV Graphics, we saw how sports broadcasters increasingly make use of effects that highlight tracked elements in a sporting event, from the players in a football match to the ball they are playing with. So how else might we apply such tracking technologies?
According to Melvin Kranzberg’s first law of technology, “Technology is neither good nor bad; nor is it neutral”. In the sports context, we may be happy to thing that cameras can be used to track – and annotate – each player’s every move. But what if we take such technological capabilities and apply them elsewhere?
EXERCISE: As well as being used to support referees making decisions about boundary line events, such as whether a tennis ball landed “in” or “out”, or whether a football crossed the goal line, how might virtual boundaries be used as part of a video surveillance system? To what extent could image tracking systems also be used as part of a video surveillance system?
One way of using virtual boundaries as part of a video based surveillance system might be to use them as virtual trip wires, where breaches of a virtual boundary or fence can be used to flag a warning about a possible physical security breach and perhaps start a detailed recording of the scene.
ASIDE: The notion of virtual tripwires extends into other domains too. For example, for objects tracked using GPS, “geo-fences” can be defined that raise an alert when a tracked object enters, or leaves, a particular geographic area. The AIS ship identification system used to uniquely identify ships – and their locations – can be used as part of a geofenced application to raise an alert whenever a particular boat, such as a ferry, enters or leaves a port.
Video surveillance might also be used to track individuals through a videoed scene. For example, if a person of interest has been detected in a particular piece of footage, they might be automatically tracked through that scene. If multiple cameras cover the same area, persons of interest may be tracked across multiple video feeds, as described by Khan, Sohaib, Omar Javed, Zeeshan Rasheed, and Mubarak Shah. “Human tracking in multiple cameras.” In Computer Vision, 2001. ICCV 2001. Proceedings. Eighth IEEE International Conference on, vol. 1, pp. 331-336. IEEE, 2001.
Where the environment is rather more constrained, such as an office block, tools such as the FXPAL DOTS Video Surveillance System allow for individuals to be tracked throughout the building. Optional filters also allow tracking or identification based on the colour of clothing, which may be meaningful in an environment where different colour uniforms or protective clothing are used to identify people by role – and perhaps by different access permission levels.
Once a hard computer science problem to solve, a wide variety of programming libraries and tools now support object identification and tracking. There are even Javascript libraries available, such as tracking.js, that are capable of tracking objects and faces streamed from a laptop camera using code that runs just in your browser.
Tracking is one thing – but identification of tracked entities is another. In some situations, however, tracked entities may carry clearly seen identifiers – such as car number plates. Automatic Number Plate Recognition (ANPR) is now a mature technology and is widely deployed against moving, as well as stationary, vehicles.
With technology firmly in place for tracking objects, and perhaps even identifying them, analysts are now turning their attention to systems that are capable of automatically identifying different events, or behaviours, within a visual scene, a step up from the simple “threshold crossing” behaviours used to implement virtual tripwires.
Once behaviours have been automatically identified, the visual scene may be overlaid with a statement of, or interpretation of, those behaviours.
Many technologies are developed for a particular purpose, but that does not prevent them being adopted for other purposes. When new technologies emerge, there are often many opportunities for businesses and entrepreneurs to find ways of using those technologies either on their own or in combination with other technologies. However, there are also risks, not least that the technology is used for a harmful purpose, or that that we do not approve of. More difficult is to try to predict what the consequences of using such technologies widely may be. As technologists, it’s our job to try to think critically about how emerging technologies may be used, whether for good, or evil, and contribute to debates about whether we want to approve the use of such technologies, or limit them in some way.
In the post From Magic Lenses to Magic Mirrors and Back Again we saw how magic lenses allow users to look through a screen at a mediated view of the scene in front of them, and magic mirrors allow users to look at a mediated view of themselves. In this post, we will look at how remote viewer might capture a scene that is then mediated in some way before being presented to the viewer in near-real-time. In particular, we will consider how live televised sporting events may be augmented to enhance the viewer’s understanding or appreciation of the event.
Ever since the early days of television, TV graphics have been used to overlay information – often in the “lower third” of the screen – to provide a mediated view of the scene being displayed. For example, one of the most commonly scene lower third effects is to display a banner giving the name and affiliation of a “talking head”, such as a politician being interviewed in a news programme.
But in recent years, realtime annotation of elements within the visual scene have become possible, providing the producers of sports television in particular with a very rich and powerful way of enhancing the way that a particular event is covered with live TV graphics.
EXERCISE: from your own experience, try to recall two or three examples of how “augmented reality” style effects can be used to enhance televised sporting events in a real-time or near-realtime way.
Educators often use questions to focus the attention of the learner onto a particular matter. For example, an educator reading an academic paper may identify things of interest (to them) that they want the learner to pick up on. The educator then needs to find a way of twisting the attention of the learner to those points of interests. This is often what motivates the questions they set around a resource (its purpose is to help the students learn how to focus their attention on a resource and immediately reflect back why something in the paper might be interesting – by casting a question to which the item in the paper is the answer). When addressing a question, the learner also needs to appreciate that they expected to answer the question in an academic way. More generally, when you read something, read it with a set of questions in mind that may have been raised by reading the abstract. You can also annotate the reading with questions which that part of the reading answers. Another trick is to spot when part of the reading answers a question or addresses a topic you didn’t fully understand: “Ah, so that means if this, then that…”. This is a simple trick, but a really powerful one nonetheless, and can help you develop your own self-learning skills.
EXERCISE: Read through the following abstract taken from a BBC R&D department white paper written in 2012 (Sports TV Applications of Computer Vision, riginally published in ‘Visual Analysis of Humans: Looking at People’, Moeslund, T. B.; Hilton, A.; Krüger, V.; Sigal, L. (Eds.), Springer 2011):
This chapter focuses on applications of Computer Vision that help the sports broadcaster illustrate, analyse and explain sporting events, by the generation of images and graphics that can be incorporated in the broadcast, providing visual support to the commentators and pundits. After a discussion of simple graphics overlay on static images, systems are described that rely on calibrated cameras to insert graphics or to overlay content from other images. Approaches are then discussed that use computer vision to provide more advanced effects, for tasks such as segmenting people from the background, and inferring the 3D position of people and balls. As camera calibration is a key component for all but the simplest applications, an approach to real-time calibration of broadcast cameras is then presented. The chapter concludes with a discussion of some current challenges.
How might the techniques described be relevant to / relate to AR?
Now read through the rest of the paper, and try to answer the following questions as you do so:
what is a “free viewpoint”?
what is a “telestrator” – to what extent might you claim this is an example of AR?
what approaches were taken to providing “Graphics overlay on a calibrated camera image”? How does this compare with AR techniques? Is this AR?
what is Foxtrax and how does it work?
what effects are possible once you “segment people or other moving objects from the background”? What practical difficulties must be overcome when creating such an effect?
how might prior knowledge help when constructing tracking systems? What additional difficulties arise when tracking people?
how can environmental features/signals be used to help calibrate camera settings? what does it even mean to calibrate a camera?
what difficulties are associated with Segmentation, identification and tracking?
The white paper also identifies the following challenges to “successfully applying computer vision techniques to applications in TV sports coverage”:
The environment in which the system is to be used is generally out of the control of the system developer, including aspects such as lighting, appearance of the background, clothing of the players, and the size and location of the area of interest. For many applications, it is either essential or highly desirable to use video feeds from existing broadcast cameras, meaning that the location and motion of the cameras is also outside the control of the system designer.
The system needs to fit in with existing production workflows, often needing to be used live or with a short turn-around time, or being able to be applied to a recording from a single camera.
The system must also give good value-for-money or offer new things compared to other ways of enhancing sports coverage. There are many approaches that may be less technically interesting than applying computer vision techniques, but nevertheless give significant added value, such as miniature cameras or microphones placed in a in cricket stump, a ‘flying’ camera suspended on wires above a football pitch, or a high frame-rate cameras for super-slow-motion.
To what extent do you think those sorts of issues apply more generally to augmented and mediated reality systems?
In the rest of this post, you will some some examples of how computer vision driven television graphics have been used in recent years. As you watch the videos, try to relate the techniques demonstrated with the issues raised in the white paper.
From 2004 to 2010, the BBC R&D department, in association with Red Bee Media, worked on a system known as Piero, now owned by Ericsson, that explored a wide range of augmentation techniques. Watch the following videos and see how many different sorts of “augmentation” effect you can identify. In each case, what sorts of enabling technology do you think are required in order to put together a system capable of generating such an effect?
In the US, SportVision provide a range of real-time enhancements for televised sports coverage. The following video demonstrates car and player tracking in motor-racing and football respectively, ball tracking in baseball and football (soccer), and a range of other “event” related enhancements, such as offside lines or player highlighting in football (soccer).
EXERCISE: watch the SportVision 2012 showreel on the SportVision website. How many different augmented reality style effects did you see demonstrated in the showreel?
For further examples, see the case studies published by vizrt.
Watching the videos, there are several examples of how items tracked in realtime can be visualised, either to highlight a particular object or feature (such as tracking a player, highlighting the position of a ball, puck, or car), or trace out the trajectory followed by the object (for example, highlighting in realtime the path followed by a ball).
Having seen some examples of the techniques in action, and perhaps started to ask yourself “how did they do that?”, skim back over the BBC white paper to see if any of the sections jump out at you in answer to your self-posed questions.
In the UK, Hawk-Eye Innovations is one of the most well known providers of such services to UK TV sports viewers.
The following video describes in a little more detail how the Hawk-Eye system can be used to enhance snooker coverage.
And how Hawk-Eye is used in tennis:
In much the same way as sportsmen compete on the field of play, so too do rival technology companies. In the 2010 Ashes series, Hawk-Eye founder Paul Hawkins suggested that a system provided by rivals VirtualEye could lead to inaccurate adjudications due to human operator error compared to the (at the time) more completely automated Hawk-Eye system (The Ashes 2010: Hawk-Eye founder claims rival system is not being so eagle-eyed).
The following video demonstrates how the Virtual Eye ball tracking software worked to highlight the path of a cricket ball as it is being bowled:
EXERCISE: what are the benefits to sports producers from using augmented reality style, realtime television graphics as part of their production?
The following video demonstrates how the SportVision Liveline effect can be used to help illustrate what’s actually happening in an Americas Cup yacht race, which can often be hard to follow for the casual viewer:
EXERCISE: To what extent might such effects be possible in a magic lens style application that could be used by a spectator actually witnessing a live sporting event?
EXERCISE: review some of the video graphics effects projects undertaken in recent years by the BBC R&D department. To what extent do the projects require: a) the modeling of the world with a virtual representation of it; b) the tracking of objects within the visual scene; c) the compositing of multiple video elements, or the introduction of digital objects within the visual scene?
As a quick review of the BBC R&D projects in this area suggests, the development of on-screen graphics that can track objects in real time may be complemented by the development of 3D models of the televised view so that it can be inspected from virtual camera positions that provide a view of the scene that is reconstrcuted from a model bulit up from the real camera positions.
Once again, though, there may be a blurring of reality – because is the view actually taken from a virtual camera, or a real one such as in the form of a Spidercam?
As well as overlaying actual footage with digital effects, sports producers are also starting to introduce virtual digital objects into the studio to provide an augmented reality style view of the studio to the viewer at home.
The use of 3D graphics in TV studios is increasingly being used to dress other elements of the set. In addition, graphics are also being used to enhance TV sports through the use of virtual advertising. Both these approaches will be discussed in another post.
More generally, digital visual effects are used widely across film and television, as we shall also explore in a later post…
PS In the absence of a more recent round-up, here’s an application reviewed in late 2017:
And here’s an example of an application for annotating sports scenes:
In recent years, commercial outdoor advertising has made increasing use of screen based digital signage. These can be used for video based advertising campaigns as well as “carousel” style displays where the same screen can be used to display different adverts in turn. But in a spirit of playfulness, they may also be used as magic lens style displays, similar in kind to the handheld magic lens applications described in the post “Magic Lenses” and See-Through Displays. In 2014, the Pepsi Max “Unbelievable” ad campaign by Abbott Mead Vickers BBDO tricked passengers waiting in London bus shelters into think a customised bus shelter had a transparent side wall, when it fact it was a large magic lens – the Pepsi Max “Unbelievable Bus Shelter”.
Magic lenses provide both a view of the world in front of the display as well as mediated, augmented or transformed version of it. But what if we replace the idea of a lens with that of a mirror, that augments the scene captured by a front-mounted, user facing camera?
Another part of the Pepsi Max “Unreality” campaign replaced a real mirror with a “magic mirror” that transformed the “reflection” seen by the subject by replacing their face with a virtually face-painted version of it:
Just as mobile phone provide a convenient device for viewing the scene directly in front of the user via a screen, with all that entails in terms of re-presenting the scene digitally, front mounted cameras on smart phones allow the user to display a live video feed of their own face on the screen, essentially using the user-facing camera+live video display combination as a mirror. But can such things also be used as magic mirrors?
Indeed they can. Several cosmetics manufacturers already publish make-up styling applications that show the effect of applying different styles of make-up selected by the user. The applications rely on identifying particular facial features, such as lips, or eyes, and then allow the use to apply the make-up virtually. (You will see how this face-capture works in another post.)
Another application, ModiFace, offers a similar range of features.
For an academic take on how an augmented reality make-up application can be used for make-up application tutorial purposes, see de Almeida, D. R. O., Guedes, P. A., da Silva, M. M. O., e Silva, A. L. B. V., do Monte Lima, J. P. S., & Teichrieb, V. (2015, May). Interactive Makeup Tutorial Using Face Tracking and Augmented Reality on Mobile Devices. In Virtual and Augmented Reality (SVR), 2015 XVII Symposium on (pp. 220-226). IEEE.
In much the same way that the Pepsi Max bus shelter used a large size display as a magic lens, so to can human size displays be used to implement magic mirrors.
Once again, the fashion industry has made use of full length magic mirrors to help consumers “try on” clothes using augmented reality. The mirror identifies the customer and then overlays their “reflection” with the items to be tried on. The following video shows the FXGear FXMirror being used as part as a shop floor fitting room.
EXERCISE: Read the blurb about the FXGear FXMirror. What data is collected about users who model clothes using the device? How might such data be used?
EXERCISE: How else have marketers and advertisers used augmented and mediated reality? Try searching through various marketing trade/industry publications to find reports of recent campaigns using such techniques. If you find any, please provide a quick review of them, along with a link, in the comments.
Augmented Reality Apps for the Design Conscious
When the 2013 Ikea catalogue was first released at the start of August 2012, as part of a campaign developed in association with the McCann advertising agency, it was complemented by an augmented reality application that allowed customers to place catalogue items as if in situ in their own homes. Each year since then, the augmented reality app has been updated with the latest catalogue items, demonstrating Ikea’s ongoing commitment to this form of marketing.
Perhaps not surprisingly, the use of augmented reality in the context of interior design extends far beyond just an extension of the Ikea catalogue.
One of the drawbacks of the current generation of augmented reality interior design applications is the low quality of the rendering of the digital 3D object. As we shall see elsewhere, the higher powered computer processors available in today’s desktop and laptop computers, compared to mobile devices, means that it is becoming possible to render photorealistic objects in a reasonable amount of time with a personal computer. However, meeting the realtime rendering requirement of augmented reality apps, as well as the ability to ensure that that the rendered object is appropriately shaded given the lighting conditions of the environment and the desired location of the artificial object, presents further technological challenges.
what does the report describe as “one of the main goals of any retailer’s digital investment”? How do they claim augmented reality might achieve that goal? To what extent do you think that claim is realistic? What obstacles can you think of that might stand in the way of achieving such a goal using augmented or mediated reality?
according to the report, how might augmented reality be used in retail? The report was published in 2014 – can you find any recent examples of augmented reality being used in ways described in the report? Is it being used for retail in ways not identified in the report?
what does the report identify as the possible business value benefits of using augmented reality? In that section, a table poses the question “What augmented reality use case would increase your likelihood of purchasing the product?”. Can you find one or more current or historical examples of the applications described? Do such applications seem to be being used more – or less – frequently in recent times?
A lot of hype surrounds artificial reality although in many respects is value other than as a novelty are yet to be determined. To what extent do you think augmented reality applications are a useful everyday contribution to the marketer’s toolkit, and to what extent are they simply a marketing novelty fit only for short lived campaigns? What are the challenges to using such applications as part of an everyday experience?
Moshi Monsters is a free online game for kids, in which they adopt a monster and look after it. Kids whose parents give us their approval can become members on our site, and can adopt a Moshi Monster. Kids care for their monster by solving puzzle games, which earn their monster virtual rewards called Rox. Kids can spend Rox on virtual items like food, furniture and other treats and toys for their monster. Over time their monster will increase in level, be able to visit new locations in Monstro City, and earn all kinds of in-game rewards for playing. Monster owners will also be able to make friends with other owners and leave messages on their pages.
It’s claimed that the ‘solve-to-earn Rox’ puzzles make the game “educational” – do you agree?
To provide a faintly serious side to this post(?!), how does Moshi Monsters address issues of child safety and parental control? What is the Moshi Monsters line on advertising on the site, compared to its ‘thematic rival’, Neopets? To what extent do you think Moshi Monsters is simply providing a vehicle for selling Moshi Monsters branded goods?
How does the parental advice offered by Moshi Monsters compare with information for parents provided on other child-friendly social networking sites such as Habbo Hotel, Club Penguin or Barbie Girls?
If you’re after 5 minutes of *really* educational fun (?!), why not have a go at Typeracer…
The game it to type a quote out from a book, movie or song lyric faster than the people you are competing against…
Please not that the Digital Worlds blog only covered audio in the most rudimentary sense, and a full treatment would require one or more dedicated uncourse blogs on the subject to do it justice!
Here’s a brief summary of the standards for sound effects creation:
IM27 CREATE SOUND EFFECTS FOR INTERACTIVE MEDIA PRODUCTS Example job titles: Sound Effects Designer, Audio Engineer
Overview
This unit is about your ability to create sound effects that work in an interactive context.
Knowledge and Understanding
This is what you must know
a. How to interpret and follow specifications or other briefs;
b. How, and to whom, to ask questions to clarify requirements or raise issues in response to the specification or brief;
c. Principles of sound design,sound effects and acoustics;
d. How to locate sources of audio material suitable for meeting the creative brief;
e. The types of audio effects that are available and their suitability for different products and contexts;
f. Ways in which sound effects can be used to enhance the user’s experience and/or give feedback on user interactions;
g. Appropriate file formats for saving sound effects;
h. The effect of audio sampling-rates and bit-depth on file-size and data-transfer rates;
i. When and why a sound effect might be cut-off prematurely,and how to minimise the risk of this adversely affecting the product.
j. The various types of data compression and their relative merits and demerits.
k. How to layer sounds to achieve a combined audio effect or to produce a complex replay of elements with logical replay rules
l. The various techniques for synchronising sounds to moving images
m. How to use and work within bespoke 3D geometry rules
n. The recording,editing and post production of dialogue
Awareness
This is what you must be aware of
i. Project parameters and constraints including target platforms and their capabilities, especially relating to audio playback;
ii. Any other audio, such as background music, that the sound effects you create will need to fit with;
iii. The events or user interactions that will trigger sound effects in the product;
iv. How each sound effect will be used in the product (for example, whether it will play once, loop several times or indefinitely etc.);
v. Compatibility issues between mono, stereo, multi-channel and surround sound;
vi. When permission is needed to use material created by others;
vii. The limits of what you may legally do with material created by others before permission is needed;
viii. Any naming conventions, standards, guidelines or specifications that you need to follow;
ix. The requirements and expectations of other team members who will use the sound effects you create.
Performance Statements
This is what you must be able to do
1. Generate original sound effects to meet a brief or specification;
2. Systematically assess the implementation of your work in iterative versions and specify changes in effects, volume, pitch and panning
3. Edit existing audio material to create sound effects to meet a brief or specification;
4. Save sound effects in an appropriate format for different target platforms;
5. Organise sound effects using appropriate filing and naming conventions so that they can be located easily by others;
6. Provide clear documentation and audio demonstration clips as necessary for others to incorporate your sound effects into the product;
7. Liaise with colleagues, such as designers and developers, to ensure your sound effects are appropriate and meet requirements;
8. Liaise with the relevant authority to obtain approval for your work.
Here’s a brief summary of the standards for music composition:
IM28 CREATE MUSIC FOR INTERACTIVE MEDIA PRODUCTS Example job titles: Composer, Musician, Music Writer
Overview
This unit is about your ability to compose and record music for use in interactive products. It assumes you already know how to compose music generally and now need to apply this skill in an interactive media context.
You might need to save your compositions as:
• MIDI files
• AIFF sound files
• WAV sound files
• AC3 files
Knowledge and Understanding
a. How to interpret and follow specifications or other briefs;
b. Leading the process of assessing and specifying music requirements as necessary
c. How, and to whom, to ask questions to clarify requirements or raise issues in response to the specification or brief;
d. The different technologies used in a computer-based music studio, including samplers, sequencers, MIDI devices, ‘outboard’ recording studio hardware and mixing desks;
e. How to sample audio from legitimate sources and use sound samples in your composition;
f. How to use appropriate software to record, sequence and mix audio;
g. Different formats in which music can be output, and when it would be appropriate to use them;
h. The effect of audio sampling-rates and bit-depth on file-size and data-transfer rates;
i. How to address the challenges of scoring music for non-linear medium with scenes of indeterminate length by employing techniques like branching segments and the use of music layers mixed dynamically at run-time.
j. How to articulate designs for bespoke development tools to enable auditioning of your work out of context
Awareness
This is what you must be aware of
i. Project parameters and constraints including target platforms and their capabilities, especially relating to audio playback and data-transfer rates;
ii. How the music will be used in the product (for example, whether it will play once, loop several times or indefinitely, whether it needs to sync with specific parts of the product, etc.);
iii. How the music content will work in conjunction with sound effects and dialogue
iv. Any requirement for the music to change in response to events or user interactions (for example by changing key or tempo, or by segueing into another piece);
v. When permission is needed to sample or use material created by others;
vi. The limits of what you may legally do with material created by others before permission is needed;
vii. The overall purpose and mood of the product and its intended user experience;
viii. How music has been used to enhance comparable products including competitor products.
Performance Statements
This is what you must be able to do
1. Compose music that is appropriate for the purpose and mood of the product;
2. Record music in an appropriate format that can be reproduced within the capabilities of the target platforms;
3. Mix and edit music in an appropriate format that can be reproduced within the capabilities of the target platforms;
4. Create music that can respond to events and user interactions as required;
5. Organise your work using appropriate filing and naming conventions so that it can be located easily by others;
6. Provide clear documentation as necessary for others to incorporate your work into the product;
7. Liaise with colleagues, such as designers and developers, to ensure your work is appropriate and meets requirements;
8. Liaise with the relevant authority to obtain approval for your work.
You might also find the following Open University course of interest: TA212 The technology of music. Here’s a summary: This joint technology/arts course starts with an introduction to music theory and notation and the technological techniques needed in a study of music technology. The principles of sound and acoustics are studied and the course relates musical terms and fundamentals to their physical equivalents. You will study the operation and characteristics of various musical instruments, how music can be represented and stored, the fundamentals of recording, manipulation and transmission of sound, MIDI, current developments and some associated legal/commercial issues.
You can find sample material from the course in the following OpenLearn Units:
One of the things I keep mentioning in the context of game design is the role of sound. Go to any games arcade, and one of the first things you’ll notice is that it’s an attack on all your senses.
Audio can play an important part in manipulating the emotions of the player – and their engagement with the game – as well as providing feedback when a particular action has occurred – or equally as a warning about something that is about to happen.
Today, many games consoles have part of their hardware dedicated to managing the sound – even surround sound in some cases. But high quality audio production was not always the order of the day!
Read A History of Video Game Music (GameSpot). As you do so, see if you can use the article to answer the following questions – or at least act as the starting point for a wider search that will turn up the answers!
The Grammy Awards are best known as music industry awards. To what extent are game soundtracks eligible for recognition in the Awards? Is there a category for game soundtracks in the awards?
How was sound used to create tension in Space Invaders? (Unfortunately, the links to the audio clips no longer appear to work. If you can find copies of sound files for any of the games mentioned, please comment back here with a link. If you want to share your own recordings, DivShare is one place to share them from (it’s not quite ‘YouTube for audio clips’, but then, is anywhere?).
what is claimed to be “the first stand-alone audio soundtrack in the video game industry”? And just what is a “stand-alone video-game audio soundtrack”? ;-)
when did stero sound start to appear in video games?
to what extent was sound supported in the original GameBoy? How does this compare with audio supported in the current generation of handheld consoles (such as the Sony PSP, or Nintendo DS Lite?)
when did sports titles first start to use continuous “play-by-play” commentary? What exactly is continuous “play-by-play” commentary anyway?;-)
how did games from the late 1990s start to use music as an important part of the actual gameplay or game mechanic?
If you are maintaining your own timeline of notable events in game and interactive media history, why not add some important dates in the history of in-game audio to it? (You might also like to refer to the alternative game audio timeline given in the first part of Adaptive Audio: A report by Alexander Brandon.)
Digital Worlds - Distorted Reality 
Recent Comments