Google for Music... by Google?

The big goal of my research for the past few years has been to build a system that we liken to "Google for Music". Using signal processing and machine learning to analyze music, we can automatically associate any songs with hundreds of descriptive tags. This means that you can find music without knowing the name of the band / song / album you want - just by typing in words that describe what you want to hear. To check out a prototype of our system, go to HerdIt.org and sign up to be a Beta tester. While you're there, play Herd It and make the search engine even smarter!

Now it seems that Google is finally getting on board with this themselves. After years of staying away from music (the legal, licensing and logistical difficulties scared even them), it seems like Google music may be coming soon.

Now, all we need to do is get them to use our algorithms to make their search even better!

Modelling Music as a Dynamic Texture

My paper "Modeling Music as a Dynamic Texture" has been accepted for publication in IEEE Transactions on Audio, Speech and Language Processing! This is work with my colleague and good buddy, Prof. Antoni Chan, and my advisor and good buddy, Prof. Gert Lanckriet and is awesome for at least 5 reasons:

1. It presents a new, statistical model of musical audio that really takes time into account
2. It introduces - and solves - the "Bohemian Rhapsody problem" (in case you don't know what this is - try describing the sound of BoRhap in 5 words or less...)
   
3. Our model can automatically separate the verse, chorus, solo, bridge, etc parts of any song as well or better than anything else
4. It will be a chapter in my thesis
   
5. I get to talk about Freddie Mercury's falsetto, Bjork and the Beatles in an academic paper

Lookout for the paper in a library near you before the end of the year! We're working on extending the model to more Music Information Retrieval tasks like tagging and similarity. In the meantime, check out some of our results on automatically segmenting music.

Listen to the 35-second version of Bohemian Rhapsody!

Calling library objects from a loaded SWF

Herd It is a Flash application that loads lots of external SWFs - to play minigames, show results, etc. - into the main SWF loaded by the player. The problem is that, any object that is defined in the loaded, external SWF is not known to the main SWF in advance and so can not be referenced by the main SWF's ActionScript code.

When you load the external SWF as a Loader, you can refer to objects on it's stage using:
var myLoader:Loader = new Loader();
var myUrlReq:URLRequest = new URLRequest("data.swf");
myLoader.load(myUrlReq);
var myObject:* = myLoader.content["myTimelineObject"];

I always had thought that you couldn't access anything from the external SWF's library - until now! Thanks to this blog post by Koen De Weggheleire (another good thing to come from Belgium), I have learned that it IS possible to access the external SWF's library.
First, we load the loader:
var myLoader:Loader = new Loader();
var myUrlReq:URLRequest = new URLRequest("data.swf");
myLoader.load(myUrlReq);

Once the loader has loaded (you may want an Event.COMPLETE handler to wait for this), we can now find objects in the external SWF's library using the contentLoaderInfo's applicationDomain:
var LibraryClass:Class = loader.contentLoaderInfo.applicationDomain.getDefinition("LibraryClassName") as Class;
var myLibraryObject:MovieClip = new LibraryClass as MovieClip;

Nice one, Belgium!

naku penda piya, naku taka Piya

...mpenziwe

A light-stepping,
gravity-defying,
time-traveling,
super-modeling,
puppet racing,
spaceship-flying,
grave-robbing,
effect-cheezing,
tux-rocking,
child-prodigy,
crazy-dancing,
dirty-rocking

legend.

From a Queen Song to a Better Music Search Engine

My recent ICASSP paper described a Dynamic Texture Model that can capture both the timbral AND temporal qualities of music. An article about this and how it contributes to our music search project by solving the "Bohemian Rhapsody Problem" just got posted on the Jacob's School of Engineering Site, UCSD's site and EurekAlert.org!

Read the article here and check out how the system can automatically divide Bohemian Rhapsody into homogenous segments and then we can tag each segment individually, rather than treating the whole song as a single unit:

Herd It in the news

Phenomenal reception to the Beta release of my music annotation game Herd It!

First we got written up in Voice of San Diego and that led to a news spot on Channel 6 (unfortunately, I was in Taiwan at ICASSP so Gert had to present!)

Then Robyn Williams interviewed Gert and I on the Science Show, resulting in loads of new users (at some times, too many for our server!).

Play Herd It.

Harmonograph

Ever since reading the funny little book, "Harmonograph: A Visual Guide to the Mathematics of Music", I've been rather intrigued by the Harmonograph. A harmonograph is a music visualization tool that was first devised in Victorian times. It consists of two (or more) pendulums which oscillate with different frequencies and are both connected to a single pen or marker. By tuning the frequencies to various harmonic intervals (octaves, fifths, thirds, etc.), the pattern drawn out by the pen traces various intricately beautiful geometric shapes. Back in ye olde days, well-heeled toffs used to set up their harmonographs at dinner parties and everyone would get together and marvel at the trippy patterns it would create. Presumably they were all bombed on opium at the time...

Anyway, after reading the book, I realized that it would be reasonably simple to build my own "virtual" harmonograph. I started some Actionscript code to implement the equations of motion for the pen:
$P_x = sin(\omega_x t)$ and $P_y = sin(\omega_y t)$
where $\omega_x$ is the frequency of oscillation in the x-direction, corresponding to the first note in the interval. Thus, to plot the interval between A4 and A5, we set $\omega_x =$ 440Hz and $\omega_y =$ 880Hz. The resulting pattern looks like this:

The next step was to add the sound of the interval that is being plotted - this is something that the Victorian gadabouts couldn't do and I thought that it would really help to get some nice synesthesia going on. Enter Flash 10 and the new sound synthesis capabilities!

So, invite over your dandy friends, bust out the opium pipe and enjoy the virtual harmonograph!