Open Innovation: Netflix Prize

Published Oct-22-09

A competition to develop a powerful algorithm to improve the accuracy of movie predictions for customers of Netflix

BellKor’s Pragmatic Chaos

The Story:

Open Innovation: Netflix Prize It was the three-year computer competition that had mathematicians, statisticians, software engineers and cyber geeks from all over the world working feverishly toward a $1 million prize.

The challenge was to improve online DVD rental and streaming service Netflix’s movie recommendation system.

With viewing choices expanding exponentially Netflix CEO Reed Hastings believed that one way his company could retain subscribers would be to improve the website’s ability to predict which movies customers would like.

These predictions are based on a customer’s own movie preferences. The basic premise is that if you love these movies, then here’s another load that you’re going to be passionate about.

Launch of the Netflix Prize

So in October 2006 the company launched the challenge, inviting anyone to come up with better recommendation software than Netflix’s in-house program called Cinematch. To be in with a chance of winning the prize entries had to be at least 10 percent better than Cinematch.

A 10 percent increase would be a valuable boost to the company coffers. It would help Netflix shift more movies, and increase customer satisfaction.
Hastings knew this was a tough call. He had already wrestled with codes, programs and algorithms, but hadn’t made any progress. And his A-team of crack engineers didn’t have much luck either.

The competition attracted more than 40,000 teams from 186 countries. Competitors were working with a data set of 100 million movie ratings, and with personal information stripped away they had to come up with new algorithms to predict which movies customers would prefer. The answers were then compared to the movies that customers actually picked.

We Have a Winner

In June 2009 a seven-person multinational outfit called BellKor’s Pragmatic Chaos surpassed the ten percent barrier.

Under the rules of the competition that triggered a 30-day period where other teams had the opportunity to beat them. This prompted a slew of competing sides to join forces and when Netflix declared the competition closed in late July two teams had passed the 10 percent threshold – BellKor and Ensemble, with the former being the eventual winner, having achieved a 10.06 percent improvement.

BellKor was actually a hybrid formed from several previously competing teams. They worked in different locations and largely communicated by email. The first time they all met was at the prize giving ceremony. The winners attribute their success to the fact that they were able to combine all their algorithms to create an even more complex one.

Some of the algorithms examined movies in bundles of elements, which might include things like genre or actor. One model looked at what movies were rated rather than how they were rated. By bringing them all together BellKor was able to create a useable model. Team manager Chris Volinksy told CNN Money about BellKor's winning strategy. “You need to think outside the box, and the only way to do that is find someone else’s box.”

Boosting the Bottom Line

Hastings believes that the winning algorithm has already added to his company’s bottom line by increasing its customer-retention rate. And he reckons he has picked up a brand new search engine at a bargain price.

The prize money has represented such a good investment that he has already launched more Netflix contests with shorter time spans of six and eighteen months before awarding the prize.

Far Reaching Implications

The Netflix Prize could have far reaching implications. The contest was widely followed, not because it could help improve movie selections, but because of the way researchers from every corner of the planet were grappling with huge amounts of data.

In addition, the way that teams came together (particularly in the latter stages of the contest) and improved their results suggests that this model of Internet collaboration could be applied to other complex issues and challenges in science, technology, and business.

Share on      
Next Story »

What Can we Solve for You?