Thursday, March 14, 2013

The whizzdom of crowds

I've been enjoying the heck out of the Seavey's "Fantasy Iditarod" game.  It occurred to me that with so many participants (469!) it might be interesting to look at how everybody bet, to see whether or not the game actually had any predictive value.

A few years back James Suroweicki wrote a book called "The Wisdom of Crowds," the basic premise of which is that "a large group's aggregated answers to questions involving quantity estimation, general world knowledge, and spatial reasoning has generally been found to be as good as, and often better than, the answer given by any of the individuals within the group." Over the past decade or so there's been tremendous growth in what are called "prediction markets," in which participants buy and sell prediction shares in things like political elections, Academy Awards, etc.

So, how well can a group of Iditarod fans predict the race outcome?  Not that well, as it turns out, but not that badly, either.  There seems to be some accuracy at the high ends (winners) and low ends (um, not winners), but the pricing and rules of the game have a distortive effect in the middle, I think.

Here's my premise: according to the rules of Fantasy Iditarod, each person has $27,000 to allocate to up to 7 mushers.  Each musher had a "price," with very experienced, successful mushers being priced quite high and rookies or people who hadn't been particularly in the past priced quite low.  The prices were set such that a player couldn't spend it all on top mushers - if they wanted 7 mushers they'd need to bet on some lesser-known or less-successful mushers.  I thought it was possible that the raw counts of how many people had included a given musher might indicate something about how the race would turn out.  So, I wrote a simple script to count the number of bets on each of the mushers (and did the same for the rookie bets - more on that later).

 I was surprised that Martin Buser had gotten as many bets as he had, and he was the jackrabbit early in the race.  Aliy Zirkle was the most popular choice and she came in a very close second.   Joar Leifseth Ulsom was ranked 5th, which is not what you'd normally expect for a rookie, but was quite close to how he actually did (7th).  Mitch Seavey was the 9th most popular choice but had far, far viewer bets than Aliy (74, to her 232).

I think to some extent the results were distorted by our inability to pick the seven teams we thought would place the highest, although I don't think they were distorted that much.  Almost certainly people with no chance received more votes than they would have absent the $27,000 limit, and people with some chance received a bit fewer.  I also think there was a lot of sentimental voting as a way of showing support for mushers people particularly like, regardless of what the expected outcome would be.

In the rookie race, Joar was the hands-down favorite.  If you follow dog mushing at all he would have had to have been your choice.  He received 92 votes for the Rookie Award, and the closest competitor was Travis Beals, at 45.  Josh Cadzow received 42, which surprised me quite a bit - I would have thought he'd get the second-most votes, based on past performance.  If anybody has any insight into why Travis got more votes from fans I'd be really interested to hear your take on it.

The tables are below, with names, votes, and actual placement.  Some teams are still on the trail so the final standings aren't complete, but 36 teams are in and I think that's enough to get a handle on how well the Fantasy Iditarod bets line up with the actual results.

Fantasy Iditarod bet counts
Name Bets Placement
Aliy Zirkle 232 2
Dallas Seavey 206 4
Martin Buser 156 17
Lance Mackey 127 19
Joar Leifseth Ulsom 98 7
Jeff King 97 3
Jake Berkowitz 86 8
Ramey Smyth 80 20
Mitch Seavey 74 1
DeeDee Jonrowe 71 10
Gerry Willomitzer 71 withdrawn
Kristy Berington 70
Mike Ellis 68 30
Travis Beals 68
Brent Sass 65 22
Josh Cadzow 61 14
Matt Giblin 54
Newton Marshall 51 scratched
Peter Kaiser 50 13
John Baker 47 21
Cindy Abbott 46 scratch
Allen Moore 44 33
Aaron Peck 42
Anna Berington 42
Richie Diehl 40 36
Cim Smyth 40 15
Matt Failor 38 28
Paige Drobny 36 34
Mikhail Telpin 32
Christine Roalofs 31
James Volek 30
Aaron Burmeister 28 11
Scott Janssen 28 scratched
Jason Mackey 27 scratched
Paul Gebhardt 26 16
Charley Bejna 25 scratched
Nicolas Petit 25 6
Jessie Royer 23 18
Luan Ramos Marques 23
Mike Williams Sr 23
Wade Marrs 22 32
Jodi Bailey 21
Karin Hendrickson 21
Angie Taggart 20
Jan Steves 18 scratched
Michelle Phillips 18 24
Justin Savidis 17
Ken Anderson 17 12
Ray Redington Jr 17 5
Jim Lanier 15 35
Michael Williams Jr 14 23
Curt Perano 14 27
Kelley Griffin 14 26
Louie Ambrose 13
Bob Chlupach 11
Jessica Hendricks 10 25
Robert Bundtzen 9 scratched
Kelly Maixner 9 31
Sonny Lindner 7 9
Cindy Gallea 7
Rudy Demoski Sr 6 scratch
Linwood Fiedler 5 29
Michael Suprenant 3 scratched
Gerald Sousa 3
Ed Stielstra 3 scratched
David Sawatzky 1 scratched

Fantasy Iditarod rookie bet counts
Joar Leifseth Ulsom 92
Travis Beals 45
Josh Cadzow 42
Paige Drobny 32
Mike Ellis 30
Richie Diehl 23
Cindy Abbott 11
Mikhail Telpin 10
Charley Bejna 9
James Volek 9
Christine Roalofs 8
Luan Ramos Marques 7
Louie Ambrose 6


  1. I think your script to count "bets" is incorrect. There were 466 participants and each participant picked 7 unique mushers. Your total "bets" should therefore sum to 3,262.

  2. I disagree. There are several fallacies to your premise.

    The limitation of $27k to choose 7 mushers altered/impacted how people chose their mushers. It forced people to not select the top 7 mushers but to select a variety of mushers that would finish in all parts of the race. For example, I selected Josh Cadzow because he was a cheap rookie, not because I expected him to do well. Liikewise, there were many mushers I would have likely predicted to do well but couldn't choose because I didn't have the money.

    Also, individual reasons also impacted people's picks. For example, 28.54% of people chose Travis Beals. This wasn't a prediction that Travis would finish high but instead a reflection that Travis lives near the Seaveys and many people picked him because they knew him. Other people selected a musher because the muser was his/her favorite, not because it was a predictive selection that that musher would win. For example, I chose Charley Bejna because my parents were helping out with his team pre-race.

    Also, I think an individual who knows the mushers and is familiar with their dogs will do better than the group of 466 taken together. Another way to put it is that a large number of participants are just casual Iditarod fans with limited information to make an educated guess that's much more than a hunch. For example, I chose Jessica Hendricks because I was impressed with abilities when I volunteered at the Ruby checkpoint during the 2003 race. I picked her on a hunch with no knowledge of what she's been doing the past 10 years.

    Also, people pick the popular mushers which has nothing to do with any sort of prediction. For example, Martin Buser was the 3rd most selected musher chosen by 34.55% of the participants. But in the previous 10 Iditarods since his 2002 win, he's only placed in the top 10 3 times. Is he the 3rd most likely musher to have won the 2013 race or is he a very popular musher?

    Also, the intent of Fantasy Iditarod was not to select the winner but to build a portfolio of mushers with the $27k limit that would create the maximum points. This created trade-offs in which some people didn't select the musher they thought would most likely win in order to have more money to spend on other above-average mushers.

  3. I just found your blog from Norma Delia's link on Facebook. Really interesting!
    I wrote a blog to talk about my picks in the Fantasy Iditarod, so I thought my thoughts my give some clues.