Nothing more exciting than watching that RPI in realtime!

(04-30-2014 08:39 AM)NicevilleWRC Wrote:  

(04-28-2014 04:20 PM)JOwl Wrote:  

(04-28-2014 04:14 PM)Gravy Owl Wrote:  

(04-28-2014 12:55 PM)JOwl Wrote:  It's unavoidable when using a binary measure (win/loss) as the basis of the rating. It would be necessary to use something that captured how badly you beat a team in order to avoid the issue.

I think it's probably possible to devise a rating algorithm where losses can't help and wins can't hurt, without using any information beyond wins and losses. I'm not so sure that such a rating would actually work better than ISR.

I'm pretty sure such a rating system would collapse to simply treating all opponents as equal (i.e., completely ignore strength of schedule).

How do you get past the problem I posed for an undefeated team? Every victory except the one vs its highest-SOS opponent would cause its rating to go down vs what it would be from just that one win.

Regression and/or Bayesian logic. If a team is undefeated it's much more likely they were lucky in a few games that otherwise would have been losses than they were actually expected to win all of their games, so you'd regress their 'talent' record to say 0.900 or so (as opposed to their 'actual' record of 1.000). But as they continue to win you'd regress their record less, or to phrase it differently, their actual record becomes an increasingly large part of your estimation.

Using Bayes your prior (assumption) is the team is a 0.500 talent level team and you revise your prior as new information comes in. You'd place a little bit of weight on beating bad teams because that's expected for even 0.500 teams, but you'd still revise your prior slightly upwards because it's less likely we are a bad team. However beating a good team would carry lots of weight, as a 0.500 team isn't likely to do that, so our prior would change significantly upwards.

In neither case would beating a bad team lower your estimation of the undefeated team. Maybe the problem could occur when you improve your determination of who are the good and bad teams by looking beyond just their record and including strength of schedule, but I'm not sure why if you've set it up properly. Unless, as Boyd mentions, you are estimated to beat a team by 15 runs but 'only' win by 8 because you took your foot off the gas, except that doesn't happen as much in baseball because you can't freely sub backups or 'run out the clock'.

Bayesian or a Credibility approach would mask the apparent problem (rating), but wouldn't solve the real problem, which would show up in the ranking. Basically, what you're doing is compressing the possible range of ratings at the start of the season, and allowing that range to expand over time. So the teams who are significantly better than the initial estimate will appear to improve over course of the season, measured in terms of rating. But they _won't_ appear to improve with respect to their peers, who are growing with them. The rating without context is meaningless (nobody cares that their team, say, cracked 0.600 in RPI), the rating relative to others is what matters.

Take as a new example two good teams who are undefeated and have identical SOS. Then in their next games, team A plays a team that's of a quality exactly equal to their prior SOS and wins, while team B plays a bad team and wins. In any well-constructed system that relies solely on W/L, team A's rating is going to improve relative to B's (even if the rating for both increases). This is identical to the "problem" that Fitt raised: even if you do everything you possibly can -- i.e., win the game -- you can go down in the ranking. As before, the problem can be overcome if using measurement data that factors in strength of win (e.g. run differential), but not with a binary win/loss measure.

Or to put into other terms, the Bayesian approach is going to do little other than scale the ratings around the midpoint. It doesn't improve the measure any.

(04-30-2014 11:18 PM)grol Wrote:  Nothing more exciting than watching that RPI in realtime!

(05-01-2014 12:30 AM)JOwl Wrote:  

(04-30-2014 08:39 AM)NicevilleWRC Wrote:  

(04-28-2014 04:20 PM)JOwl Wrote:  

(04-28-2014 04:14 PM)Gravy Owl Wrote:  

(04-28-2014 12:55 PM)JOwl Wrote:  It's unavoidable when using a binary measure (win/loss) as the basis of the rating. It would be necessary to use something that captured how badly you beat a team in order to avoid the issue.

I think it's probably possible to devise a rating algorithm where losses can't help and wins can't hurt, without using any information beyond wins and losses. I'm not so sure that such a rating would actually work better than ISR.

I'm pretty sure such a rating system would collapse to simply treating all opponents as equal (i.e., completely ignore strength of schedule).

How do you get past the problem I posed for an undefeated team? Every victory except the one vs its highest-SOS opponent would cause its rating to go down vs what it would be from just that one win.

Regression and/or Bayesian logic. If a team is undefeated it's much more likely they were lucky in a few games that otherwise would have been losses than they were actually expected to win all of their games, so you'd regress their 'talent' record to say 0.900 or so (as opposed to their 'actual' record of 1.000). But as they continue to win you'd regress their record less, or to phrase it differently, their actual record becomes an increasingly large part of your estimation.

Using Bayes your prior (assumption) is the team is a 0.500 talent level team and you revise your prior as new information comes in. You'd place a little bit of weight on beating bad teams because that's expected for even 0.500 teams, but you'd still revise your prior slightly upwards because it's less likely we are a bad team. However beating a good team would carry lots of weight, as a 0.500 team isn't likely to do that, so our prior would change significantly upwards.

In neither case would beating a bad team lower your estimation of the undefeated team. Maybe the problem could occur when you improve your determination of who are the good and bad teams by looking beyond just their record and including strength of schedule, but I'm not sure why if you've set it up properly. Unless, as Boyd mentions, you are estimated to beat a team by 15 runs but 'only' win by 8 because you took your foot off the gas, except that doesn't happen as much in baseball because you can't freely sub backups or 'run out the clock'.

Bayesian or a Credibility approach would mask the apparent problem (rating), but wouldn't solve the real problem, which would show up in the ranking. Basically, what you're doing is compressing the possible range of ratings at the start of the season, and allowing that range to expand over time. So the teams who are significantly better than the initial estimate will appear to improve over course of the season, measured in terms of rating. But they _won't_ appear to improve with respect to their peers, who are growing with them. The rating without context is meaningless (nobody cares that their team, say, cracked 0.600 in RPI), the rating relative to others is what matters.

Take as a new example two good teams who are undefeated and have identical SOS. Then in their next games, team A plays a team that's of a quality exactly equal to their prior SOS and wins, while team B plays a bad team and wins. In any well-constructed system that relies solely on W/L, team A's rating is going to improve relative to B's (even if the rating for both increases). This is identical to the "problem" that Fitt raised: even if you do everything you possibly can -- i.e., win the game -- you can go down in the ranking. As before, the problem can be overcome if using measurement data that factors in strength of win (e.g. run differential), but not with a binary win/loss measure.

Or to put into other terms, the Bayesian approach is going to do little other than scale the ratings around the midpoint. It doesn't improve the measure any.

I see what you're saying now, and you're right (although I think that's a different problem from RPI, because doesn't the actual rating decrease?).

I suppose then you could reframe the conversation - instead of ranking the best teams by talent, you could rank them as who's had the most impressive season in which case it's fair to say Team A has had a more impressive season than Team B. That's cheating in a sense, since we usually care who's better, but when determining who should make the playoffs more impressive regular season is a good criteria.

Boyd has us at #6 this morning, but we're in a virtual tie (all at 0.600) with #4 FSU and #5 South Carolina. #15 ISR and #10 SoS.

(05-01-2014 08:10 AM)NicevilleWRC Wrote:  I suppose then you could reframe the conversation - instead of ranking the best teams by talent, you could rank them as who's had the most impressive season in which case it's fair to say Team A has had a more impressive season than Team B.

I'm not sure that is reframing at all -- it's how the conversation has been framed in the first place.

I don't know anyone who believes that the purpose of RPI or ISR is to rank teams by "talent". The purpose is to measure performance so that it can be usefully compared. The debate is about how good each measure is at doing just that, but not about the purpose.

Leave impressions of "talent" to the ESPN blowhards who view college baseball as nothing more than a personnel training program for the majors, rather than something in which players belong to actual teams and in which teams actually want to win.

(05-01-2014 08:10 AM)NicevilleWRC Wrote:  That's cheating in a sense, since we usually care who's better, but when determining who should make the playoffs more impressive regular season is a good criteria.

I'm not sure the premise is correct. Determining post-season participation by performance rather than perceived "talent" would not seem to be cheating in any sense -- not in college baseball, not in the NFL, not in any league. But the opposite arguably would be.

(05-01-2014 08:39 AM)georgewebb Wrote:  I don't know anyone who believes that the purpose of RPI or ISR is to rank teams by "talent".

From Boyd's website: "The Iterative Strength Ratings, a measure of team quality for NCAA Division I baseball". I've always interpreted that as talent.

(05-01-2014 08:39 AM)georgewebb Wrote:  

(05-01-2014 08:10 AM)NicevilleWRC Wrote:  That's cheating in a sense, since we usually care who's better, but when determining who should make the playoffs more impressive regular season is a good criteria.

I'm not sure the premise is correct. Determining post-season participation by performance rather than perceived "talent" would not seem to be cheating in any sense -- not in college baseball, not in the NFL, not in any league. But the opposite arguably would be.

Maybe we aren't using the same definition of talent and performance. By talent I don't mean potential, but something like a team's true winning percentage based on their inherent skill level. By performance I mean some observed combination of their inherent skill plus some luck/noise/random variation mixed in. We can debate over talent, but we can't debate performance. Whether or not they could ever do it again (talent), Auburn returned the blocked "Kick Six" against Alabama to win the game (performance) and went to the SEC and BCS Championship games as a result.

Participation in the college baseball playoffs uses both methods - teams get automatic bids based on performances in conference tournaments, but at large teams are chosen based on talent (mostly). And then everyone is seeded based on talent (again, mostly). That's why we argue over who should be a national seed and who was snubbed, but not over who won the WAC.

So what is the point of the College World Series? Is it to determine the best team, or to determine a champion? Because those aren't necessarily the same teams; the 2007 Patriots were the best team, but the Giants were the champions. I say it's cheating to rank teams based on who had the best performance in the regular season because when it comes down to it we don't care what happened in the first 40+ games, we care who's going to win today and the best way to determine that is talent. If we didn't care who was going to win today, then why have playoffs at all?

RPI is always tight, but this is really tight:

Boyd pRPI May 1:
4 Florida State 0.600
5 South Carolina 0.600
6 Rice 0.600

(05-01-2014 10:40 AM)NicevilleWRC Wrote:  We can debate over talent, but we can't debate performance. Whether or not they could ever do it again (talent), Auburn returned the blocked "Kick Six" against Alabama to win the game (performance) and went to the SEC and BCS Championship games as a result.

OK, but it sounds like you are arguing that sending Auburn to the SEC championship was "cheating" (sure, they won the qualifying stage, but everyone "knows" Alabama was "better"), whereas sending Alabama would to the title game not have been. Most people would argue the reverse.

I am confused. This statement seems to say that you believe measuring performance is measuring talent*:

(05-01-2014 10:40 AM)NicevilleWRC Wrote:  From Boyd's website: "The Iterative Strength Ratings, a measure of team quality for NCAA Division I baseball". I've always interpreted that as talent.

This statement (especially with the characterization of "cheating") seems to say that you believe that measuring performance is almost antithetical to measuring talent:

(05-01-2014 08:10 AM)NicevilleWRC Wrote:  I say it's cheating to rank teams based on who had the best performance in the regular season because when it comes down to it we don't care what happened in the first 40+ games, we care who's going to win today and the best way to determine that is talent.

*To be clear, that's a perfectly fair point, given that (1) you've defined "talent" as likelihood of success in future games, and (2) performance is generally a darn good predictor of future success.

(05-01-2014 11:30 AM)georgewebb Wrote:  I am confused. This statement seems to say that you believe measuring performance is measuring talent*:

(05-01-2014 10:40 AM)NicevilleWRC Wrote:  From Boyd's website: "The Iterative Strength Ratings, a measure of team quality for NCAA Division I baseball". I've always interpreted that as talent.

This statement (especially with the characterization of "cheating") seems to say that you believe that measuring performance is almost antithetical to measuring talent:

(05-01-2014 08:10 AM)NicevilleWRC Wrote:  I say it's cheating to rank teams based on who had the best performance in the regular season because when it comes down to it we don't care what happened in the first 40+ games, we care who's going to win today and the best way to determine that is talent.

*To be clear, that's a perfectly fair point, given that (1) you've defined "talent" as likelihood of success in future games, and (2) performance is generally a darn good predictor of future success.

Some of both. Observed performance is talent plus random variation, so when trying to evaluate a team you start with measuring performance and then try to weed out the variation so you're only left with talent.

The reason why I'd say they can be antithetical is performance is more retrodictive and looks back at what has happened, while talent is more predictive and looks forward to what will happen. The two aren't exclusively useful for only one or the other - as you said, performance is a good predictor of future success, but random variation is not, so if you can eliminate the random variation then you have an even better predictor of future success (i.e. talent) - but each serves one purpose better than the other.

I didn't mean to say it was cheating to send Auburn to the championship game, I meant determining who had a better season is easier - everyone would say Auburn. The harder and (I think) more interesting question is which was the better team, so to 'settle' for who had a better season is a bit of a cop out.

(05-02-2014 01:32 PM)NicevilleWRC Wrote:  I didn't mean to say it was cheating to send Auburn to the championship game, I meant determining who had a better season is easier - everyone would say Auburn. The harder and (I think) more interesting question is which was the better team, so to 'settle' for who had a better season is a bit of a cop out.

That's a good distillation, and perhaps zeroes in on where I think you and I diverge. Personally, I don't view "which was the better team" as more interesting; I think who wins is more interesting. Similarly, I don't consider "who had the better season" to be settling; I consider it to be the whole point.

After all, the proof of the pudding is in the eating.*


*Granted, that's not exactly on-point, but it's close, and I hate missing a chance to correctly state this often mis-stated maxim.