Hello There, Guest! (LoginRegister)

Post Reply 
The Perfect Computer Ranking?
Author Message
Crayton Offline
All American
*

Posts: 3,328
Joined: Feb 2019
Reputation: 186
I Root For: Florida
Location:
Post: #1
The Perfect Computer Ranking?
The perfect computer ranking

Some of the issues surrounding the committee is that the process is opaque and there always seems to be a chance they’ll get it wrong. Could a computer ranking replace the committee? The Massey Composite is often listed for comparison here; could one of its more “accurate” rankings be used?

What would the “perfect” computer ranking look like?
1. Simple enough that hobbiests could calculate it without the need of a proprietary ESPN/FPI formula
2. Results similar to those of human (AP, for example) polls, not a Power/Vegas-line/predictive poll.
3. “Iterative”, one of the Massey Composite requirements; not just a formula, like RPI
4. Not the final say; a set of rules would need to augment the ranking to select any post-season field (ex. 2 highest ranked teams, then any conference champs ranked in the Top 6, then any at-larges remaining)

Sagarin has been popular because of USA Today and the BCS (some of those BCS ones were pretty horrible). Are there any other good ones we should keep our eyes on?
12-16-2019 05:30 PM
Find all posts by this user Quote this message in a reply
Advertisement


Wedge Offline
Moderator
*

Posts: 19,862
Joined: May 2010
Reputation: 964
I Root For: California
Location: IV, V, VI, IX
Post: #2
RE: The Perfect Computer Ranking?
There is no such thing as a perfect computer ranking. Not possible.

Every computer formula for ranking teams reflects the biases of those who created the formula.

A composite ranking is nothing more than a composite of biases -- which is exactly what the AP poll, the "coaches" poll, and the committee rankings are.

If someone loves a particular computer ranking, it's almost certainly because that person's favorite team(s) are treated well by that ranking, and/or the teams they hate are treated poorly by that ranking.
12-16-2019 05:33 PM
Find all posts by this user Quote this message in a reply
ken d Offline
Hall of Famer
*

Posts: 17,327
Joined: Dec 2013
Reputation: 1209
I Root For: college sports
Location: Raleigh
Post: #3
RE: The Perfect Computer Ranking?
(12-16-2019 05:33 PM)Wedge Wrote:  There is no such thing as a perfect computer ranking. Not possible.

Every computer formula for ranking teams reflects the biases of those who created the formula.

A composite ranking is nothing more than a composite of biases -- which is exactly what the AP poll, the "coaches" poll, and the committee rankings are.

If someone loves a particular computer ranking, it's almost certainly because that person's favorite team(s) are treated well by that ranking, and/or the teams they hate are treated poorly by that ranking.

Exactly. And what good would it do us if there actually were one? It's not as if a "perfect" computer ranking could ever accurately predict the outcome of games. And if it could, then why play the games?

The information available today for fans is better than it's ever been, and is as good as it's going to get. We should celebrate that.
12-16-2019 06:38 PM
Find all posts by this user Quote this message in a reply
TexanMark Offline
Legend
*

Posts: 25,637
Joined: Jul 2003
Reputation: 1326
I Root For: Syracuse
Location: St. Augustine, FL
Post: #4
RE: The Perfect Computer Ranking?
UCF fans swear by Colley, the other 130 teams not so much. LOL

I usually just use Massey Composite as a guide.

Happy Bowl season everyone. I enjoy watching as many games as possible.
12-24-2019 10:02 AM
Visit this user's website Find all posts by this user Quote this message in a reply
orangefan Offline
Heisman
*

Posts: 5,208
Joined: Mar 2007
Reputation: 354
I Root For: Syracuse
Location: New England
Post: #5
RE: The Perfect Computer Ranking?
(12-16-2019 06:38 PM)ken d Wrote:  
(12-16-2019 05:33 PM)Wedge Wrote:  There is no such thing as a perfect computer ranking. Not possible.

Every computer formula for ranking teams reflects the biases of those who created the formula.

A composite ranking is nothing more than a composite of biases -- which is exactly what the AP poll, the "coaches" poll, and the committee rankings are.

If someone loves a particular computer ranking, it's almost certainly because that person's favorite team(s) are treated well by that ranking, and/or the teams they hate are treated poorly by that ranking.

Exactly. And what good would it do us if there actually were one? It's not as if a "perfect" computer ranking could ever accurately predict the outcome of games. And if it could, then why play the games?

The information available today for fans is better than it's ever been, and is as good as it's going to get. We should celebrate that.

The biggest difference of opinion tends to be whether all wins should be given equal weight regardless of the margin of victory, or whether the margin of victory should be considered. Beyond that, different modelers have developed different algorithms to address each of these factors. For instance, weight has to be given to the quality of wins by a team that was defeated to determine the value of the win, but how much weight? There are many additional nuanced decisions built into every model, each of which will affect the output.

With respect to predicting outcomes, all teams in any sport have a range of possible performance levels. They are not just one number. So, it will never be possible for a formula to accurately predict a winner 100% of the time, only a more likely outcome.
12-24-2019 11:25 AM
Find all posts by this user Quote this message in a reply
Attackcoog Offline
Moderator
*

Posts: 44,735
Joined: Oct 2011
Reputation: 2860
I Root For: Houston
Location:
Post: #6
RE: The Perfect Computer Ranking?
The computer rankings and human Committee both suffer from differing types of bias. That’s why I prefer a system where winning your way into the playoff is the basis for filling out most of the playoff bracket and the subjective methods such as a Committee or polls/computers are just used for “at large” wildcards. The 8 team play-off with P5 champs getting an autobid—the top G5 champ—and two wild cards seems like the best compromise system. It fills most of the bracket based simply on the way games play out on the field—-but still uses the subjective measures to pick among the 5 G5 champs (it at least uses "winning your way in" to narrow the G5 field to just 5) for the G5 slot and to determine which two teams should receive “second chance” golden wild card tickets to the playoff. In other words, we would use subjective methods only where live game play on the field can’t reasonably be utilized as the primary method of selection.
(This post was last modified: 12-24-2019 12:19 PM by Attackcoog.)
12-24-2019 11:26 AM
Find all posts by this user Quote this message in a reply
Advertisement


DavidSt Offline
Hall of Famer
*

Posts: 23,010
Joined: Dec 2013
Reputation: 729
I Root For: ATU, P7
Location:
Post: #7
RE: The Perfect Computer Ranking?
I use PRWolfe who do not overhype SEC teams, but his formula ranked UCF as number 1 when they went unbeaten. PRWolfe ranks G5 schools better.
12-24-2019 11:30 AM
Visit this user's website Find all posts by this user Quote this message in a reply
quo vadis Offline
Legend
*

Posts: 50,007
Joined: Aug 2008
Reputation: 2370
I Root For: USF/Georgetown
Location: New Orleans
Post: #8
RE: The Perfect Computer Ranking?
(12-24-2019 10:02 AM)TexanMark Wrote:  UCF fans swear by Colley, the other 130 teams not so much. LOL

Colley is a silly system. I just glanced at their rankings, and they have Memphis at #4. They have Boise at #9 and Washington at #44. Washington just crushed Boise like a bug.

Its a wacko system.
12-24-2019 11:31 AM
Find all posts by this user Quote this message in a reply
quo vadis Offline
Legend
*

Posts: 50,007
Joined: Aug 2008
Reputation: 2370
I Root For: USF/Georgetown
Location: New Orleans
Post: #9
RE: The Perfect Computer Ranking?
(12-24-2019 11:25 AM)orangefan Wrote:  The biggest difference of opinion tends to be whether all wins should be given equal weight regardless of the margin of victory, or whether the margin of victory should be considered.

To me, you have to factor in MOV, just because it is obviously related to how good a team is. For example, if LSU plays UT- San Antonio, they are likely to win by more points than if Texas State plays UT- San Antonio. That's because LSU is better than Texas State. There's just no question that the better you are than another team, the more points you are likely to win games vs similar opponents by.

Of course, that relationship isn't perfect, it has a skew towards offensive-oriented teams. For example, in 1986, Miami and Penn State both played WVU. Miami won 58-14. Penn State won 19-0, a much smaller MOV because PS didn't have Miami's wide-open offense. But when they played, PS won 14-10.

Still, we clearly lose more omitting it than including it.
(This post was last modified: 12-24-2019 11:49 AM by quo vadis.)
12-24-2019 11:48 AM
Find all posts by this user Quote this message in a reply
Crayton Offline
All American
*

Posts: 3,328
Joined: Feb 2019
Reputation: 186
I Root For: Florida
Location:
Post: #10
RE: The Perfect Computer Ranking?
(12-24-2019 11:26 AM)Attackcoog Wrote:  The computer rankings and human Committee both suffer from differing types of bias. That’s why I prefer a system where winning your way into the playoff is the basis for filling out most of the playoff bracket and the subjective methods such as a Committee or polls/computers are just used for “at large” wildcards. The 8 team play-off with P5 champs getting an autobid—the top G5 champ—and two wild cards seems like the best compromise system. It fills most of the bracket based simply on the way games play out on the field—-but still uses the subjective measures to pick among the 5 G5 champs (it at least uses "winning your way in" to narrow the G5 field to just 5) for the G5 slot and to determine which two teams should receive “second chance” golden wild card tickets to the playoff. In other words, we would use subjective methods only where live game play on the field can’t reasonably be utilized as the primary method of selection.

I'll do you one further and grant 1 of those 2 wild-card spots to the winner of a WCG on Championship Saturday (albeit those 2 teams are determined by a ranking). The last spot would go to a 3rd at-large or, more likely, the best CCG loser (Georgia, this year).

Basing the playoff on wins and losses, CCGs and "standings", seems to be the goal. We want teams to be able to "win there way into" the playoff. NFL fans seem fine with the many, convoluted tie-breakers once head-to-head fails.

Similarly, if there were a recognize computer ranking, with simple inputs, that would serve as some type of "standing" for teams to view how well they are doing, who needs to lose, and perhaps even how MOV may factor in to their post-season hopes. It would be transparent and not dependent on voters or a committee.

My current ranking uses a SRS (all FCS teams are 25 points below even) to rate everyone on MOV, but then uses a SOR (strength needed to acheive record, given opponents' ratings) to award higher rankings to teams with more difficult schedules. That way MOV plays a large factor in rating teams but has a near-neutral affect on an individual team's ranking.

The BCS "began" as a formula-heavy ranking. But, the field was very underresearched and underdeveloped (see the Colley as an example, I think Wolfe was also silly but its been a while since I looked) and handicapped by the ban on margin-of-victory factors. The wonky formulas pushed the BCS back toward a voter-heavy system by 2003. Now, 20 years later, there has got to be a computer system that is sufficient to the ADs/fans needs for an FBS-wide standing.
12-24-2019 02:05 PM
Find all posts by this user Quote this message in a reply
IWokeUpLikeThis Offline
Hall of Famer
*

Posts: 13,726
Joined: Jul 2014
Reputation: 1434
I Root For: NIU, Chicago St
Location:
Post: #11
RE: The Perfect Computer Ranking?
I would like to see an 8-team CFP with Colley Matrix replacing the committee.
12-24-2019 03:26 PM
Find all posts by this user Quote this message in a reply
Advertisement


ChrisLords Offline
Heisman
*

Posts: 8,639
Joined: Jun 2007
Reputation: 336
I Root For: Virginia Tech
Location: Earth
Post: #12
RE: The Perfect Computer Ranking?
(12-16-2019 05:30 PM)Crayton Wrote:  The perfect computer ranking

Some of the issues surrounding the committee is that the process is opaque and there always seems to be a chance they’ll get it wrong. Could a computer ranking replace the committee? The Massey Composite is often listed for comparison here; could one of its more “accurate” rankings be used?

What would the “perfect” computer ranking look like?
1. Simple enough that hobbiests could calculate it without the need of a proprietary ESPN/FPI formula
2. Results similar to those of human (AP, for example) polls, not a Power/Vegas-line/predictive poll.
3. “Iterative”, one of the Massey Composite requirements; not just a formula, like RPI
4. Not the final say; a set of rules would need to augment the ranking to select any post-season field (ex. 2 highest ranked teams, then any conference champs ranked in the Top 6, then any at-larges remaining)

Sagarin has been popular because of USA Today and the BCS (some of those BCS ones were pretty horrible). Are there any other good ones we should keep our eyes on?

I think you're missing the point. The purpose of a human selection committee is to keep the G5 from getting into the playoffs. You can't do that with a computer formula.
12-24-2019 04:23 PM
Find all posts by this user Quote this message in a reply
Crayton Offline
All American
*

Posts: 3,328
Joined: Feb 2019
Reputation: 186
I Root For: Florida
Location:
Post: #13
RE: The Perfect Computer Ranking?
I suppose the point of a thread on a "perfect" ranking, is to suss out what attributes such a ranking would have... not just the particulars of its algorithm. To that end, 4 ideal Attributes are proposed in the OP, they are: (A) simple and non-proprietary, (B) similar to human polls, (C] iterative, not a simple formula, and (D) not the final say on postseason placement

(12-24-2019 11:25 AM)orangefan Wrote:  The biggest difference of opinion tends to be whether all wins should be given equal weight regardless of the margin of victory, or whether the margin of victory should be considered. Beyond that, different modelers have developed different algorithms to address each of these factors. For instance, weight has to be given to the quality of wins by a team that was defeated to determine the value of the win, but how much weight? There are many additional nuanced decisions built into every model, each of which will affect the output.

Here is a description of the ranking system I've been using recently (shared mostly because it is the offseason, I have a couple days off, and I'm bored); it fulfills Attribute A, simple and non-proprietary:

#1) A straight SRS; and all FCS teams are assigned a value of -25 (ie. 25 point underdogs to an average FBS team).
#2) A SOR (Strength of Record) formula is used, given each team's above SRS to sort teams into their final ranking. #1 and #2 fulfill Attribute C
#3) Until CCGs are played, preseason rankings have some weight (which diminishes to 0 by December), so that rankings in September and October don't look wacky. This part is effective, but complicated and over a decade old from a previous system; it may not satisfy Attribute C, but it matters 0 by the end of the season, so I'm not interested in altering it if its only for my private consumption.

Many SOR rankings (#2, above) I've seen have an arbitrary quality to them. ESPN, for example, ranks teams by how difficult it would be for "an average Top 25 team" to achieve the same or better record. This is of no help outside the Top 40 and oftentimes we're more interested in how the Top 5 stack up against one another, for playoff purposes. Instead I worked backwards, and rather than ranking based on likelihood of repeating the record, teams are ranked by the strength needed to repeat the record. It smoothes nicely over all 130 teams.

By using only SOR for the final rankings, the rankings reflect "deservedness" rather than "power". Last year, Alabama was #4 in SRS, ahead of playoff-bound Oklahoma; they were a powerful team and certainly played LSU closer than the Sooners did. However the human polls (and the committee) are more retrodictive and by going 12-1 and winning their conference against decent competition, Oklahoma "deserved" the 4th playoff spot more. This fulfills Attribute B.

Still, the first SRS step is necessary to correctly grade SOR. A win over 11-2 Alabama (#4 in SRS) is much more impressive than a win over 12-2 Boise (#40 in SRS). Because SRS is MOV-heavy, running up the score ends up rewarding the teams on your schedule more than rewarding you (though there is no negative effect). This preserves the principle enshrined in the BCS era that MOV should not be used, so that teams hoping to maximize their computer ranking would not run up the score on their opponents. While the intent of the BCS forefathers was noble (good sportsmanship and all), MOV is immensely helpful in calibrating a final ranking. The BCS ended up using the human polls as a greater crutch after the 2001 and 2003 debacles to offset this limitation; the "eye-test" we called it. an Attribute E? or still B?

Finally, no ranking should be the end-all for post-season play. "Tie-breakers" so to say, should be used. While 2018 Georgia may have had a more impressive 11-2 than Oklahoma and Ohio State's 12-1's, the number of losses break the tie and Georgia does not go to the playoff. Similarly in 2017, Ohio State's superior 11-2 performance loses out to the 1-loss campaign of Alabama. This fulfills Attribute D.
(This post was last modified: 06-24-2020 08:40 PM by Crayton.)
06-24-2020 08:39 PM
Find all posts by this user Quote this message in a reply
bullet Offline
Legend
*

Posts: 66,287
Joined: Apr 2012
Reputation: 3285
I Root For: Texas, UK, UGA
Location:
Post: #14
RE: The Perfect Computer Ranking?
(12-24-2019 04:23 PM)ChrisLords Wrote:  
(12-16-2019 05:30 PM)Crayton Wrote:  The perfect computer ranking

Some of the issues surrounding the committee is that the process is opaque and there always seems to be a chance they’ll get it wrong. Could a computer ranking replace the committee? The Massey Composite is often listed for comparison here; could one of its more “accurate” rankings be used?

What would the “perfect” computer ranking look like?
1. Simple enough that hobbiests could calculate it without the need of a proprietary ESPN/FPI formula
2. Results similar to those of human (AP, for example) polls, not a Power/Vegas-line/predictive poll.
3. “Iterative”, one of the Massey Composite requirements; not just a formula, like RPI
4. Not the final say; a set of rules would need to augment the ranking to select any post-season field (ex. 2 highest ranked teams, then any conference champs ranked in the Top 6, then any at-larges remaining)

Sagarin has been popular because of USA Today and the BCS (some of those BCS ones were pretty horrible). Are there any other good ones we should keep our eyes on?

I think you're missing the point. The purpose of a human selection committee is to keep the G5 from getting into the playoffs. You can't do that with a computer formula.
Garbage in Garbage out. A computer system might work with college basketball where you have 25-30 games, but with the 12-13 of football, computers have a weak connection with who is actually better.
06-24-2020 09:20 PM
Find all posts by this user Quote this message in a reply
ken d Offline
Hall of Famer
*

Posts: 17,327
Joined: Dec 2013
Reputation: 1209
I Root For: college sports
Location: Raleigh
Post: #15
RE: The Perfect Computer Ranking?
(12-24-2019 03:26 PM)IWokeUpLikeThis Wrote:  I would like to see an 8-team CFP with Colley Matrix replacing the committee.

I took the top approximately 40 teams and recorded their rank in each of six of the most widely used ratings to come up with a composite rank (like Massey does only without every Tom, Dick and Harry with a computer) by throwing out the best and worst ranking for each team. I then went back to see which ratings were thrown out the most often as being outliers. Colley was by far the most frequently out of line from the others, and the size of their deviation from the norm by far the greatest, so I dropped them, and got a much tighter grouping of ranks.

I don't believe we will ever have a system that doesn't have any beauty contest component to the selection process at all. I just don't see why having a single such contest with by far the fewest "judges" is better or less biased than a composite of reputable computer models. And I don't believe for a second that the purpose of the CFP selection committee is to keep the G5 out of the playoffs, or that the human polls have a similar sinister slant. To me, that's pure paranoia. There's a lot of that going around these days.
06-24-2020 09:34 PM
Find all posts by this user Quote this message in a reply
chargeradio Offline
Vamos Morados
*

Posts: 7,466
Joined: Mar 2007
Reputation: 121
I Root For: ALA, KY, USA
Location: Louisville, KY
Post: #16
RE: The Perfect Computer Ranking?
Winning percentage, ranked 1 to 130 (weighted triple)
Opponents winning percentage, ranked 1 to 130
Points For, ranked 1 to 130
Points Against, ranked 1 to 130
Time of Possession, ranked 1 to 130
Net Turnover Margin, ranked 1 to 130

Obviously with 12 games there may be lots of ties in these categories, so to be ranked 1st in all six categories would be unlikely. But if a team had an average ranking of less than 5.0 amongst these metrics, you'd expect them to be one of the best (if not the best) teams in the country.

Games against FCS opponents would likely have to be weighted differently (i.e., dock so many percentage points off the FCS team's record in opponents' winning percentage)
06-24-2020 11:21 PM
Visit this user's website Find all posts by this user Quote this message in a reply
Advertisement


schmolik Offline
CSNBB's Big 10 Cheerleader
*

Posts: 8,573
Joined: Sep 2019
Reputation: 637
I Root For: UIUC, PSU, Nova
Location: Philadelphia Suburbs
Post: #17
RE: The Perfect Computer Ranking?
In the old BCS era, I hated the computer rankings. I couldn't understand them. I'd rather use the SOS. Teams are evaluated on known criteria. Most of the computer formulas are so difficult to understand (assuming the formulas are even public and some aren't and if they aren't it only introduces the possibility of bias). I hate it that the NCAA men's basketball went to the NET instead of the RPI. The RPI formula is much simpler to understand and calculate. Does anyone even know how to calculate the NET?
06-25-2020 06:46 AM
Find all posts by this user Quote this message in a reply
ken d Offline
Hall of Famer
*

Posts: 17,327
Joined: Dec 2013
Reputation: 1209
I Root For: college sports
Location: Raleigh
Post: #18
RE: The Perfect Computer Ranking?
(06-25-2020 06:46 AM)schmolik Wrote:  In the old BCS era, I hated the computer rankings. I couldn't understand them. I'd rather use the SOS. Teams are evaluated on known criteria. Most of the computer formulas are so difficult to understand (assuming the formulas are even public and some aren't and if they aren't it only introduces the possibility of bias). I hate it that the NCAA men's basketball went to the NET instead of the RPI. The RPI formula is much simpler to understand and calculate. Does anyone even know how to calculate the NET?

How does making a formula public eliminate the possibility of bias? Every formula is biased. There is no way to know which football team is "best", or whether a team ranked #5 is "better" than one ranked #8 or #9. The simple fact is that the better football team does not always win, so plugging results of individual games into a formula is never going to be better than an eyeball test, or "beauty contest".
06-26-2020 11:09 AM
Find all posts by this user Quote this message in a reply
bullet Offline
Legend
*

Posts: 66,287
Joined: Apr 2012
Reputation: 3285
I Root For: Texas, UK, UGA
Location:
Post: #19
RE: The Perfect Computer Ranking?
(06-26-2020 11:09 AM)ken d Wrote:  
(06-25-2020 06:46 AM)schmolik Wrote:  In the old BCS era, I hated the computer rankings. I couldn't understand them. I'd rather use the SOS. Teams are evaluated on known criteria. Most of the computer formulas are so difficult to understand (assuming the formulas are even public and some aren't and if they aren't it only introduces the possibility of bias). I hate it that the NCAA men's basketball went to the NET instead of the RPI. The RPI formula is much simpler to understand and calculate. Does anyone even know how to calculate the NET?

How does making a formula public eliminate the possibility of bias? Every formula is biased. There is no way to know which football team is "best", or whether a team ranked #5 is "better" than one ranked #8 or #9. The simple fact is that the better football team does not always win, so plugging results of individual games into a formula is never going to be better than an eyeball test, or "beauty contest".

Never is pretty strong. Some of these eyeball tests are pretty bad.
06-26-2020 11:13 AM
Find all posts by this user Quote this message in a reply
Hokie Mark Offline
Hall of Famer
*

Posts: 23,719
Joined: Sep 2011
Reputation: 1392
I Root For: VT, ACC teams
Location: Greensboro, NC
Post: #20
RE: The Perfect Computer Ranking?
(06-26-2020 11:09 AM)ken d Wrote:  
(06-25-2020 06:46 AM)schmolik Wrote:  In the old BCS era, I hated the computer rankings. I couldn't understand them. I'd rather use the SOS. Teams are evaluated on known criteria. Most of the computer formulas are so difficult to understand (assuming the formulas are even public and some aren't and if they aren't it only introduces the possibility of bias). I hate it that the NCAA men's basketball went to the NET instead of the RPI. The RPI formula is much simpler to understand and calculate. Does anyone even know how to calculate the NET?

How does making a formula public eliminate the possibility of bias? Every INCORRECT formula is biased. There is no way to know which football team is "best", or whether a team ranked #5 is "better" than one ranked #8 or #9. The simple fact is that the better football team does not always win, so plugging results of individual games into a formula is never going to be better than an eyeball test, or "beauty contest".

The eye test is inherently biased, but mathematical formulas don't have to be.
06-26-2020 08:31 PM
Visit this user's website Find all posts by this user Quote this message in a reply
Post Reply 




User(s) browsing this thread: 1 Guest(s)


Copyright © 2002-2024 Collegiate Sports Nation Bulletin Board System (CSNbbs), All Rights Reserved.
CSNbbs is an independent fan site and is in no way affiliated to the NCAA or any of the schools and conferences it represents.
This site monetizes links. FTC Disclosure.
We allow third-party companies to serve ads and/or collect certain anonymous information when you visit our web site. These companies may use non-personally identifiable information (e.g., click stream information, browser type, time and date, subject of advertisements clicked or scrolled over) during your visits to this and other Web sites in order to provide advertisements about goods and services likely to be of greater interest to you. These companies typically use a cookie or third party web beacon to collect this information. To learn more about this behavioral advertising practice or to opt-out of this type of advertising, you can visit http://www.networkadvertising.org.
Powered By MyBB, © 2002-2024 MyBB Group.