(04-28-2014 09:08 AM)NicevilleWRC Wrote:  

(04-28-2014 08:19 AM)waltgreenberg Wrote:  

(04-28-2014 07:51 AM)NicevilleWRC Wrote:  "Opponents marked in red will do damage to the team's RPI, even if the team beats them. Opponents marked in yellow will do damage if the team loses to them. Opponents marked in green will improve RPI, even if the team loses to them."

I feel like that should be enough to show the problems with RPI. When simply having a team on your schedule improves or hurts your ranking regardless of the result you have screwed up.

This is true of ISR, as well, but in both cases it's only true in the extreme. In other words, the only time losing to another team will improve your RPI is if you have a very poor RPI ranking (e.g., sub-150) and lose to a team in the Top 25 - 50. Conversely, a team will only hurt RPI by winning if they're a Top 25 team and lose at home to a sub-150 RPI team.

According to Boyd, this is not at all true. We are currently #5 in RPI, Houston #2, and yet if we lose to them it will raise our RPI. Same for ULaLa (#13), Lousiana State (#36), Sam Houston State (#50), Dallas Baptist (#73), and TCU (#103). I don't know how exactly he determined that, but that's what his website claims.

What is true about Walt's statement is that this also happens with the ISR. Imagine, as an extreme case, a team that's undefeated against solid competition. If it then plays a crappy team and wins, its ISR will drop.

It's unavoidable when using a binary measure (win/loss) as the basis of the rating. It would be necessary to use something that captured how badly you beat a team in order to avoid the issue. As an example, per Boyd's expanded RPI details, we've played 4 games against teams that lowered our RPI regardless of game outcome. Three of those were Purdue; given the whupping we put on them, I would bet those games wouldn't hurt (probably help) under a system that includes runs or run differential.

PS - Great new detail on the RPI, Boyd. You're always welcome to hang out here and join in as much as you want.

(04-28-2014 12:55 PM)JOwl Wrote:  

(04-28-2014 09:08 AM)NicevilleWRC Wrote:  

(04-28-2014 08:19 AM)waltgreenberg Wrote:  

(04-28-2014 07:51 AM)NicevilleWRC Wrote:  "Opponents marked in red will do damage to the team's RPI, even if the team beats them. Opponents marked in yellow will do damage if the team loses to them. Opponents marked in green will improve RPI, even if the team loses to them."

I feel like that should be enough to show the problems with RPI. When simply having a team on your schedule improves or hurts your ranking regardless of the result you have screwed up.

This is true of ISR, as well, but in both cases it's only true in the extreme. In other words, the only time losing to another team will improve your RPI is if you have a very poor RPI ranking (e.g., sub-150) and lose to a team in the Top 25 - 50. Conversely, a team will only hurt RPI by winning if they're a Top 25 team and lose at home to a sub-150 RPI team.

According to Boyd, this is not at all true. We are currently #5 in RPI, Houston #2, and yet if we lose to them it will raise our RPI. Same for ULaLa (#13), Lousiana State (#36), Sam Houston State (#50), Dallas Baptist (#73), and TCU (#103). I don't know how exactly he determined that, but that's what his website claims.

What is true about Walt's statement is that this also happens with the ISR. Imagine, as an extreme case, a team that's undefeated against solid competition. If it then plays a crappy team and wins, its ISR will drop.

It's unavoidable when using a binary measure (win/loss) as the basis of the rating. It would be necessary to use something that captured how badly you beat a team in order to avoid the issue. As an example, per Boyd's expanded RPI details, we've played 4 games against teams that lowered our RPI regardless of game outcome. Three of those were Purdue; given the whupping we put on them, I would bet those games wouldn't hurt (probably help) under a system that includes runs or run differential.

PS - Great new detail on the RPI, Boyd. You're always welcome to hang out here and join in as much as you want.

Thanks for the kind words.

By the way, the ISR also includes an adjustment for margin of victory, although it's capped at ten runs (*), so it's hard to find a team that you can't beat badly enough that they actually hurt you.

(*) It's not capped because I believe in not running up the score (aka not trying as hard as you can) but because there's no continued predictive power once you go past that point.

Boyd, thanks for jumping in. It's obvious that you love the college baseball game, and there's lots of folks here who share your enthusiasm.

Aaron Fitt's article is here, if you are interested. His idea of a fix seems overly simplistic.

Guy

While I disagree with Fitt on the problem (or at least the poster child he puts forward -- what school can self-prohibit playing certain schools and then complain about the downside of not being able to play same?), I don't think the "drop 3" is a terrible solution; dropping outliers is a common method to diminishing extraordinary benefit/detriment in data.

That said, if done I don't think it should be asymmetrical; his proposal simply gives teams the greenlight to find powderpuffs. Instead, it should be drop top 3/bottom 3 or some similar balanced approach. If you also lose the benefit of your best opponent, you're going to want to step up the quality of your second-best opponent as well.

(04-28-2014 02:14 PM)grol Wrote:  Boyd, thanks for jumping in. It's obvious that you love the college baseball game, and there's lots of folks here who share your enthusiasm.

Aaron Fitt's article is here, if you are interested. His idea of a fix seems overly simplistic.

Guy

Right, dropping outliers doesn't really intersect with self-selection, so it's not a particularly useful measurement tool in this case. Add to that the fact that the college season is just barely long enough for good measurements anyway, and you find that you really don't want to be throwing away data. I suspect that all you'd really accomplish is for everyone to schedule a throwaway weekend against an HBCU or equivalent.

Boyd,

Thanks for all you do (and I miss the Breadcrumbs!). I think you've got a couple of bugs/issues with the school location data. I was puzzled why Texas wasn't showing up on the 'nearby potential opponent' lists for Rice. I clicked over to UT's page, and they only show 2 nearby opponents - Nebraska at 60(!) miles away and UNM at 264. This being Texas, 264 miles from Austin barely gets you to a border.. and not the New Mexico one, at that.

(04-28-2014 12:55 PM)JOwl Wrote:  It's unavoidable when using a binary measure (win/loss) as the basis of the rating. It would be necessary to use something that captured how badly you beat a team in order to avoid the issue.

I think it's probably possible to devise a rating algorithm where losses can't help and wins can't hurt, without using any information beyond wins and losses. I'm not so sure that such a rating would actually work better than ISR.

(04-28-2014 04:14 PM)Gravy Owl Wrote:  

(04-28-2014 12:55 PM)JOwl Wrote:  It's unavoidable when using a binary measure (win/loss) as the basis of the rating. It would be necessary to use something that captured how badly you beat a team in order to avoid the issue.

I think it's probably possible to devise a rating algorithm where losses can't help and wins can't hurt, without using any information beyond wins and losses. I'm not so sure that such a rating would actually work better than ISR.

I'm pretty sure such a rating system would collapse to simply treating all opponents as equal (i.e., completely ignore strength of schedule).

How do you get past the problem I posed for an undefeated team? Every victory except the one vs its highest-SOS opponent would cause its rating to go down vs what it would be from just that one win.

(04-28-2014 01:00 PM)boydnation Wrote:  Thanks for the kind words.

By the way, the ISR also includes an adjustment for margin of victory, although it's capped at ten runs (*), so it's hard to find a team that you can't beat badly enough that they actually hurt you.

(*) It's not capped because I believe in not running up the score (aka not trying as hard as you can) but because there's no continued predictive power once you go past that point.

Thanks much for the correction, Boyd. That's a very useful tidbit.

And while I have you... it would be great to see an updated ratings FAQ ( http://www.boydsworld.com/baseball/faq.html#4 ) that reflected the current state of the ISR (or at least one that deleted the "why don't you include my favorite factor" question). I know there's confusion on this board, as not too long ago someone posted that the ISR doesn't involve any home/road adjustments.

(04-28-2014 03:06 PM)NoodleOwl Wrote:  Boyd,

Thanks for all you do (and I miss the Breadcrumbs!). I think you've got a couple of bugs/issues with the school location data. I was puzzled why Texas wasn't showing up on the 'nearby potential opponent' lists for Rice. I clicked over to UT's page, and they only show 2 nearby opponents - Nebraska at 60(!) miles away and UNM at 264. This being Texas, 264 miles from Austin barely gets you to a border.. and not the New Mexico one, at that.

Funny you should mention that; the data that I started with there was based on pulling from automated Google Maps searches, which worked in most cases but had a few where even the most specific search terms didn't always work. I found a couple of days ago that Most Valuable Rice Grad Jeremy Mills is including lat/lon data for all games this year, so starting tomorrow the distance data will be based on that.

(04-28-2014 04:39 PM)boydnation Wrote:   that Most Valuable Rice Grad Jeremy Mills

Agreed!

Othewise known as "Fly By Night"

(04-28-2014 04:20 PM)JOwl Wrote:  I'm pretty sure such a rating system would collapse to simply treating all opponents as equal (i.e., completely ignore strength of schedule).

How do you get past the problem I posed for an undefeated team? Every victory except the one vs its highest-SOS opponent would cause its rating to go down vs what it would be from just that one win.

I realize that my point was very unclear, so let me put it this way. I think the goal of a rating system should be to most accurately summarize the results to date. Eliminating counter-intuitive individual changes is not impossible but is undesirable since it would reduce accuracy.

I see this as part of the human poll vs computer rating debate. The defenders of human polls like to point out these individual cases. To me it is a very small price to pay for accuracy.

(04-28-2014 06:58 PM)Gravy Owl Wrote:  

(04-28-2014 04:20 PM)JOwl Wrote:  I'm pretty sure such a rating system would collapse to simply treating all opponents as equal (i.e., completely ignore strength of schedule).

How do you get past the problem I posed for an undefeated team? Every victory except the one vs its highest-SOS opponent would cause its rating to go down vs what it would be from just that one win.

I realize that my point was very unclear, so let me put it this way. I think the goal of a rating system should be to most accurately summarize the results to date. Eliminating counter-intuitive individual changes is not impossible but is undesirable since it would reduce accuracy.

I see this as part of the human poll vs computer rating debate. The defenders of human polls like to point out these individual cases. To me it is a very small price to pay for accuracy.

Gravy, while your point is valid and I don't disagree with it, less subjectivity does not necessarily lead to greater "accuracy".

(04-28-2014 07:18 PM)waltgreenberg Wrote:  Gravy, while your point is valid and I don't disagree with it, less subjectivity does not necessarily lead to greater "accuracy".

That's true -- it's possible to create a system that is completely objective and relatively (or even wildly) inaccurate.

(04-28-2014 07:36 PM)Gravy Owl Wrote:  

(04-28-2014 07:18 PM)waltgreenberg Wrote:  Gravy, while your point is valid and I don't disagree with it, less subjectivity does not necessarily lead to greater "accuracy".

That's true -- it's possible to create a system that is completely objective and relatively (or even wildly) inaccurate.

Hear hear! The comments above are perhaps especially true regarding learning: when it comes to assessing what someone has or has not learned, objective measures are not necessarily more accurate -- and are sometimes necessarily less accurate. Objectivity is often a false Grail.

(04-28-2014 03:06 PM)NoodleOwl Wrote:  Boyd,

Thanks for all you do (and I miss the Breadcrumbs!).

+1.

Good night for CUSA last night. ODU beat #1 Virginia and ECU beat UNC, among other results. The conference overall RPI went from 9 to 7 in just one night (passing AAC and Mountain West).

(04-30-2014 07:50 AM)Fort Bend Owl Wrote:  Good night for CUSA last night. ODU beat #1 Virginia and ECU beat UNC, among other results. The conference overall RPI went from 9 to 7 in just one night (passing AAC and Mountain West).

Wow

(04-28-2014 04:20 PM)JOwl Wrote:  

(04-28-2014 04:14 PM)Gravy Owl Wrote:  

(04-28-2014 12:55 PM)JOwl Wrote:  It's unavoidable when using a binary measure (win/loss) as the basis of the rating. It would be necessary to use something that captured how badly you beat a team in order to avoid the issue.

I think it's probably possible to devise a rating algorithm where losses can't help and wins can't hurt, without using any information beyond wins and losses. I'm not so sure that such a rating would actually work better than ISR.

I'm pretty sure such a rating system would collapse to simply treating all opponents as equal (i.e., completely ignore strength of schedule).

How do you get past the problem I posed for an undefeated team? Every victory except the one vs its highest-SOS opponent would cause its rating to go down vs what it would be from just that one win.

Regression and/or Bayesian logic. If a team is undefeated it's much more likely they were lucky in a few games that otherwise would have been losses than they were actually expected to win all of their games, so you'd regress their 'talent' record to say 0.900 or so (as opposed to their 'actual' record of 1.000). But as they continue to win you'd regress their record less, or to phrase it differently, their actual record becomes an increasingly large part of your estimation.

Using Bayes your prior (assumption) is the team is a 0.500 talent level team and you revise your prior as new information comes in. You'd place a little bit of weight on beating bad teams because that's expected for even 0.500 teams, but you'd still revise your prior slightly upwards because it's less likely we are a bad team. However beating a good team would carry lots of weight, as a 0.500 team isn't likely to do that, so our prior would change significantly upwards.

In neither case would beating a bad team lower your estimation of the undefeated team. Maybe the problem could occur when you improve your determination of who are the good and bad teams by looking beyond just their record and including strength of schedule, but I'm not sure why if you've set it up properly. Unless, as Boyd mentions, you are estimated to beat a team by 15 runs but 'only' win by 8 because you took your foot off the gas, except that doesn't happen as much in baseball because you can't freely sub backups or 'run out the clock'.

(04-30-2014 07:50 AM)Fort Bend Owl Wrote:  Good night for CUSA last night. ODU beat #1 Virginia and ECU beat UNC, among other results. The conference overall RPI went from 9 to 7 in just one night (passing AAC and Mountain West).

And Southern Miss just beat #7 Ole Miss tonight 7-6.

Warren Nolan just moved our RPi up to #5.