final thoughts on our NCAA football playoff forecasting model

Here are some final thoughts on our model and methodology. First, the pros.

  1. Our models worked really well considering how little information they use. Really well. I thought 12-13 games would not be enough to provide a reasonable forecast, but we ended up being very close to the committee rankings.
  2. All three versions of the model converged on the same results, which wasn’t too surprising given that they all use the same underlying network to gauge quality. But not having agreement would have pointed to a problem.

Now the cons.

  1. There is always that team that clearly doesn’t belong in the rankings, especially early on (I’m looking at you, Minnesota!)
  2. I would want to use additional information next year to instead give teams a fraction of a win based on the point differential in played games. This could help connect our network and provide more refined forecasts.
  3. We don’t use any human and/or crowdsourced information, like point spreads or some of the polls. This can help make the most sense of games that have already been played. There is no need to entirely rely on math models as we did this past year, but I do want to give the math to add value over more traditional models that only rely on human evaluation (like the polls).
  4. It looks like we will need to do some manual tweaking of the results to account for teams from non-major conferences like Marshall. We weren’t sure what to do with them, but it seems like a team like Boise State or Northern Illinois could get invited so long as their schedule passes the sniff test for difficulty (Marshall’s did not) and the team goes undefeated.
  5. We need a better testing and validation plan for selecting model parameters. What we did didn’t work well. We tried to choose the parameters that forecasted the most games “correctly,” but ultimately this ended up yielding nonsensical parameters. In the paper we used, the authors selected parameters based on predicting the outcomes of the most bowl games, and the parameters were pretty nonsensical. For example, home games counted for 3x more than away games, meaning that a team with 6 away wins would be ranked about the same as another team with 2 home wins. I selected parameters that made the most sense and “worked.” For example, home wins/losses counted 15-20% more than away wins/losses to account for the higher expectations associated with home field advantage. But I think we can improve this.

In sum, it sounds like there are a few directions to improve the basic model. We will try to clean up our code and post it in the relatively near future.

Nate Silver wrote a nice post summarizing final thoughts for FiveThirtyEight’s college football model. Their model was based on the previous week’s polls, which injected the human element into the rankings. And this introduced a bias into the rankings. Michael Lopez has a nice post on voter bias and how this affects rankings. I’m not saying that using information from humans shouldn’t play a role, but think math deserves some credit.

Final college football rankings

Our final college football playoff rankings agreed with the committee’s final ranking in terms of who gets in, but our seeding was a bit different (we swapped Ohio State and Oregon).

  1.     Alabama
  2.     Ohio State
  3.     Florida State
  4.     Oregon
  5.     TCU
  6.     Baylor

Let’s not discuss where Wisconsin ended up in the rankings. Final thoughts will be coming in an upcoming blog post. Stay tuned!

NCAA Playoff forecasts, week 14

This will be our last forecasts for the 2014 College Football Playoff. With at most one game left in the season for most teams, we see that for all practical purposes all but nine teams will be excluded from the playoff.

Our top 5 teams to make the playoff are the same as those in the CFP committee: Alabama, Oregon, OSU, Florida State, TCU.

The good news: Wisconsin still has a chance!

Wisconsin has a fairly good chance due to being in the B1G championship game and due to so many upsets last week. However, Wisconsin’s chances are probably overstated here, since the model has (incorrectly?) overvalued the B1G team. LIkewise, there is a good chance that Ohio State’s chances are overstated. Ohio State is currently ranked fifth in the College Football Rankings. Additionally, it’s not clear if the CFP committee will “value” a win over Ohio State given that Ohio State will be playing in the championship game without J.T. Barrett. Still, there is a chance for Wisconsin, and it’s better than it was last week.

The results indicate that there are only 9 teams that have a chance, with one favorite from each major conference. A second team in some of the conferences have a chance: B1G (Go Wisconsin!), SEC (Missouri has a small chance), and the Big12 (Baylor), and the ACC (Georgia Tech).

Week 14 forecasts.

Ensemble model results

The top teams to make the playoff with the fraction of simulations in the playoff (out of 10,000).

  1. Alabama: 0.976
  2. Oregon: 0.713
  3. Ohio State: 0.695
  4. Florida State: 0.59
  5. Texas Christian: 0.345
  6. Georgia Tech: 0.265
  7. Wisconsin: 0.226
  8. Baylor: 0.151
  9. Missouri: 0.0393

The top teams to just miss the playoff with the fraction of simulations ranked fifth.The fraction of playoffs with at least one team from each major conference:

  • ACC: 0.772
  • Big12: 0.444
  • B1G: 0.754
  • PAC12: 0.713
  • SEC: 0.976