Modelling Music as a Dynamic Texture

My paper "Modeling Music as a Dynamic Texture" has been accepted for publication in IEEE Transactions on Audio, Speech and Language Processing! This is work with my colleague and good buddy, Prof. Antoni Chan, and my advisor and good buddy, Prof. Gert Lanckriet and is awesome for at least 5 reasons:

1. It presents a new, statistical model of musical audio that really takes time into account
2. It introduces - and solves - the "Bohemian Rhapsody problem" (in case you don't know what this is - try describing the sound of BoRhap in 5 words or less...)
   
3. Our model can automatically separate the verse, chorus, solo, bridge, etc parts of any song as well or better than anything else
4. It will be a chapter in my thesis
   
5. I get to talk about Freddie Mercury's falsetto, Bjork and the Beatles in an academic paper

Lookout for the paper in a library near you before the end of the year! We're working on extending the model to more Music Information Retrieval tasks like tagging and similarity. In the meantime, check out some of our results on automatically segmenting music.

Listen to the 35-second version of Bohemian Rhapsody!

Calling library objects from a loaded SWF

Herd It is a Flash application that loads lots of external SWFs - to play minigames, show results, etc. - into the main SWF loaded by the player. The problem is that, any object that is defined in the loaded, external SWF is not known to the main SWF in advance and so can not be referenced by the main SWF's ActionScript code.

When you load the external SWF as a Loader, you can refer to objects on it's stage using:
var myLoader:Loader = new Loader();
var myUrlReq:URLRequest = new URLRequest("data.swf");
myLoader.load(myUrlReq);
var myObject:* = myLoader.content["myTimelineObject"];

I always had thought that you couldn't access anything from the external SWF's library - until now! Thanks to this blog post by Koen De Weggheleire (another good thing to come from Belgium), I have learned that it IS possible to access the external SWF's library.
First, we load the loader:
var myLoader:Loader = new Loader();
var myUrlReq:URLRequest = new URLRequest("data.swf");
myLoader.load(myUrlReq);

Once the loader has loaded (you may want an Event.COMPLETE handler to wait for this), we can now find objects in the external SWF's library using the contentLoaderInfo's applicationDomain:
var LibraryClass:Class = loader.contentLoaderInfo.applicationDomain.getDefinition("LibraryClassName") as Class;
var myLibraryObject:MovieClip = new LibraryClass as MovieClip;

Nice one, Belgium!

naku penda piya, naku taka Piya

...mpenziwe

A light-stepping,
gravity-defying,
time-traveling,
super-modeling,
puppet racing,
spaceship-flying,
grave-robbing,
effect-cheezing,
tux-rocking,
child-prodigy,
crazy-dancing,
dirty-rocking

legend.

From a Queen Song to a Better Music Search Engine

My recent ICASSP paper described a Dynamic Texture Model that can capture both the timbral AND temporal qualities of music. An article about this and how it contributes to our music search project by solving the "Bohemian Rhapsody Problem" just got posted on the Jacob's School of Engineering Site, UCSD's site and EurekAlert.org!

Read the article here and check out how the system can automatically divide Bohemian Rhapsody into homogenous segments and then we can tag each segment individually, rather than treating the whole song as a single unit:

Herd It in the news

Phenomenal reception to the Beta release of my music annotation game Herd It!

First we got written up in Voice of San Diego and that led to a news spot on Channel 6 (unfortunately, I was in Taiwan at ICASSP so Gert had to present!)

Then Robyn Williams interviewed Gert and I on the Science Show, resulting in loads of new users (at some times, too many for our server!).

Play Herd It.

Harmonograph

Ever since reading the funny little book, "Harmonograph: A Visual Guide to the Mathematics of Music", I've been rather intrigued by the Harmonograph. A harmonograph is a music visualization tool that was first devised in Victorian times. It consists of two (or more) pendulums which oscillate with different frequencies and are both connected to a single pen or marker. By tuning the frequencies to various harmonic intervals (octaves, fifths, thirds, etc.), the pattern drawn out by the pen traces various intricately beautiful geometric shapes. Back in ye olde days, well-heeled toffs used to set up their harmonographs at dinner parties and everyone would get together and marvel at the trippy patterns it would create. Presumably they were all bombed on opium at the time...

Anyway, after reading the book, I realized that it would be reasonably simple to build my own "virtual" harmonograph. I started some Actionscript code to implement the equations of motion for the pen:
$P_x = sin(\omega_x t)$ and $P_y = sin(\omega_y t)$
where $\omega_x$ is the frequency of oscillation in the x-direction, corresponding to the first note in the interval. Thus, to plot the interval between A4 and A5, we set $\omega_x =$ 440Hz and $\omega_y =$ 880Hz. The resulting pattern looks like this:

The next step was to add the sound of the interval that is being plotted - this is something that the Victorian gadabouts couldn't do and I thought that it would really help to get some nice synesthesia going on. Enter Flash 10 and the new sound synthesis capabilities!

So, invite over your dandy friends, bust out the opium pipe and enjoy the virtual harmonograph!

Herd It on Amazon Web Services

I've spent the past month figuring out all the details and complications of hosting my Facebook music annotation game. Herd It, on Amazon Web Services. We figured out that one of the reasons why the game didn't work well when more than 5 people connected was lack of bandwidth on our server to serve the Flash components. Also, we want to be ready for when Herd It becomes the next Desktop Tower Defense so AWS seemed like a great solution. It was an omen that, on the day that I was debating whether or not to bother to try to figure it all out, a senior manager from AWS gave a talk at UCSD...

Anyway, now that I've figured it all out, I think that AWS is great. However, there are a lot of hurdles to overcome in getting it to work so hopefully this will benefit someone (maybe even me, when I forget what I did).

Step 1 - Register for an AWS account.
If you've ever bought anything Amazon, this is as simple as adding AWS to your existing account. In particular, you will need 2 of their (many) web services:
EC2 (elastic compute cloud) - this is the "cloud" of servers that does all the processing.
S3 (simple storage service) - this is the storage bucket where you will keep all your data.

Step 2 - Figure out EC2.
EC2 works as follows:
You create an AMI - an "Amazon machine image" which is basically a complete copy of the OS, programs and data of the machine that you want to run in Amazon's cloud. Imagine you wanted to backup your computer's entire hard disk so that you could reconstruct the entire system - this is what you would need. You will replicate this image on one or more of Amazon's cloud machines.

I highly recommend following the AWS tutorial. It covers everything you'll need to know and doesn't have any distracting details. You will learn how to use images that Amazon have pre-made, how to set them up, how to change them, how to save them, and how to kill them.

Step 3 - Create your own AMI
For this, I started with the most current Ubuntu AMI at alestic.com. Some nice guy (Eric Hammond) has created a bunch of basic AMI's that have nothing more than a simple OS. From here, you will need to install all the programs that you're going to need. As someone who wasn't very familiar with Unix administration, this seemed daunting but it was surprisingly easy and I like Ubuntu a lot now. For example, this page shows you that, by typing 4 lines, you can get an Apache web server and PHP running (this is all I needed for Herd It). You will also need to copy all your data onto the AWS machine using FTP or scp (for example, I copied all the PHP and Flash files that make Herd It work).

Once you've got everything on the AMI running as you want it, you'll need to bundle your AMI and copy it into your S3 bucket. Again the AWS tutorial covers all of this.

Step 4 - Elastic DNS
This was the trickiest part... If you only want to run a single instance then you don't need to worry about this. But, the whole point of AWS is to let you create many instances to power your new web app that's going to take over the world. To achieve this, I found some help from this page but there was still a bit of work to do:

Step 4a - Register your domain.
There are a million places where you can get the domain "mykillerapp.com" or whatever.
I got www.herdit.org for $9/year from NameCheap.com
(I just discovered that, of course, there is already a site a "mykillerapp.com" and that it's a sweet applied maths quiz! For the rest of this tutorial, I'll just refer to my domain: herdit.org).

Step 4b - Set up a DNS forwarding service.
The domain name of all your new EC2 machines will be something like
http://ec2-a-bunch-of-numbers.compute.amazonaws.com/ and
http://ec2-more-different-numbers.compute.amazonaws.com/
In order for these to all map to your new domain name, herdit.org, you need to set up DNS forwarding. For this, you need a DNS service provider. You domain name service may provide this but, whether or not it does, ZoneEdit is a free service that gets the job done. You will need to transfer the DNS from your domain name provider and set up ZoneEdit (or whatever DNS service you use) to handle your new domain (it may take a day or two for these changes to register).

Step 4c - Tell you AMIs to register with your DNS service
Once your DNS service is running, you want it to forward requests for your domain ("herdit.org") to the EC2 machines ("ec2-0112358132134.amazon.com", etc.). To do this, you need those EC2 machines to tell the DNS service that they are ready.

The DNS registration works using a program called "ez-ipupdate" that you can install on your (Ubuntu) AMI by typing:

sudo apt-get install ez-ipupdate

Now all you need to do is to get ez-ipupdate to run whenever you start a new instance so that you don't have to log in manually. The Spatten Design blog has a good post on how to do this using Ruby. However, since I don't use Ruby, I wrote an init.d script that you will run when the instance starts. Copy the following and save it on your AMI as '/etc/init.d/update-dynamic-dns'

#!/bin/sh

### BEGIN INIT INFO
# Provides: update-dynamic-dns
# Required-Start: $local_fs $remote_fs
# Required-Stop: $local_fs $remote_fs
# Default-Start: 3 4 5
# Default-Stop: S 0 1 6
# Short-Description: Update dynamic DNS on startup
# Description: Uses ez-ipupdate to send the current Dynamic IP address
# to ZoneEdit Dynamic DNS provider
### END INIT INFO

# Author: Luke Barrington <lukeinusa@gmail.com>

DYNAMIC_DNS_CONFIG_FILE=/etc/ez-ipupdate/dynamic_dns.yml
AMAZON_INSTANCE_DATA_ADDRESS=http://169.254.169.254
API=latest

# Read current instance URL from Amazon service
IP=`curl $AMAZON_INSTANCE_DATA_ADDRESS/$API/meta-data/public-ipv4/`
echo "Instance Dynamic IP Address = $IP"

SERVICE=`sed -n -e "s/^service:[ ]*/\l/p" $DYNAMIC_DNS_CONFIG_FILE`
USERNAME=`sed -n -e "s/^username:[ ]*/\l/p" $DYNAMIC_DNS_CONFIG_FILE`
PASSWORD=`sed -n -e "s/^password:[ ]*/\l/p" $DYNAMIC_DNS_CONFIG_FILE`
HOST=`sed -n -e "s/^host:[ ]*/\l/p" $DYNAMIC_DNS_CONFIG_FILE`

case "$1" in
start)
echo "Using dynamic DNS service = $SERVICE"
echo "Connecting with username = $USERNAME"
echo "Mapping IP to host = $HOST"

# ZoneEdit server name has changed since ez-ipupdate was last built
if [ "$SERVICE" = 'zoneedit' ]; then
eval "ez-ipupdate --address $IP --service-type $SERVICE --server=dynamic.zoneedit.com --user $USERNAME:$PASSWORD --host $HOST"
else
eval "ez-ipupdate --address $IP --service-type $SERVICE --user $USERNAME:$PASSWORD --host $HOST"
fi
;;
*)
echo "Usage: update-dynamic-dns start"
;;
esac


You will also need to create a file at '/etc/ez-ipupdate/dynamic_dns.yml' (or whatever you call it in the script above) that contains the following:


# service should be one of the services supported by ez-ipupdate.
# Possible values: null ezip pgpow dhs dyndns dyndns-static
# dyndns-custom ods tzo easydns easydns-partner
# gnudip justlinux dyns hn zoneedit heipv6tb
# (The above list is from man ez-ipupdate)
service: zoneedit
username: YOUR DNS SERVICE USERNAME
password: YOUR DNS SERVICE PASSWORD
host: YOUR HOST NAME (e.g., herdit.org)


Finally, you can run the update-dynamic-dns script by typing:

./etc/init.d/update-dynamic-dns

To register this init.d script to run automatically at startup, use this command:

update-rc.d update-dynamic-dns defaults

Now, as soon as the EC2 machine boots (well, after a few minutes), it should register itself with your DNS service and tell it to send requests for "herdit.org" to its address (e.g., ec2-123456789.amazon.com). The cool thing about ZoneEdit (or any DNS service that has "round robin" DNS) is that, if multiple machines all register to the same host, the DNS service will send requests to each one in turn. This will spread your millions of users across all the AMIs that you run.

At this stage, you will want to bundle up the AMI again. Now you are ready for Step 5...

Step 5 - Try and take over the world
Run hundreds of instances, pay thousands of dollars to Amazon, get millions of users, sell your site for billions of dollars.



Notes and next steps
Now that I''ve set all this up, we are testing Herd It to see how it can handle the load of many simultaneous users. Herd It users a Java server to coordinate everything (via XML events) and saves all the info in a MySQL database. These are both still running on my local server. I plan to put them on AWS sometime as well and I expect that this tutorial will help with that.

There are apps out there that can monitor your site's traffic to automatically create or kill new instances based on your traffic but, for the moment, I will be monitoring it manually.

Now, after all that work, why not go and play Herd It?!