Surprises on PFF

[leoram]

Doesn't this say a lot about their grading system?

It's on the offense only as Junkman pointed out earlier. If you consider their OLine, RB, TE, and QB, it shouldn't be a surprise but it was to me at first.

 

[LosAngelesRams]

Someone should let me borrow that log in info so I can read some stats lol.

 

[jjab360]

Needless to say I disagree with your assessment of PFF. For me, you're guilty of falling into tsome of the same pitfalls that you accuse PFF of but you can tell me that after you've read this.

Let's check off each of your points and I'll see if I can make a counter observation.

"just recorded observations by laypeople" - This is probably the comment I disagree with the most because there is a method to their madness. Using "laypeople" is the best way to go IMO. If a person is asked to do a simple job too much "knowledge" and "expertise" can easily contaminate the product. If I'm asked to look at a play and record whether the ROG blocked the player in front of him so as to keep him from tackling the RB on a running play I require no specialized knowledge to complete that task. If I was a knowledgeable expert like yourself I might over analyze the situation and say to myself, "well, he kept the guy from tackling the RB but the RB had to bounce outside of what looked like the designed hole" so I'm going to give him a fail on that play while the "lay" guy would say he did his job. Who contaminated the data?

A worker on an assembly line doesn't need to know how to design, build and drive a car in order to be able to attach a steering wheel to the column. He need only attach the steering wheel exactly as he was taught without any deviation to not only do his job correctly but to also be part of the team that builds that great car that all the experts rave over.

These laypeople you're talking about aren't put out on an island by themselves. Provided they are given a specific set of instructions that they follow exactly, the data they collect can and will be collated by the "experts" you seem to think don't exist. Do you know what the "laypoeople's" instructions are? Do you know how many "laypeople" are looking at a single play, each looking at a different aspect? If the answer is no, then aren't you making a completely unsubstantiated assumption?

Now, you might quibble with or even disagree completely with the "set on instructions" he was given but as you don't even know (probably) what those instructions are you're just guessing without any facts to back you up. Do you have those facts?

"PFF lacks any sophistication to know what really made a given play successful" - And you know this how? Have you read what exactly it is they measure on each play? If not, you are again just making an assumption that isn't backed up by any facts. I don't have a prescription to PFF but I've been allowed to read what it is exactly that they measure on some of the plays by posters who do have a subscription and considering some of the limitations you yourself have alluded too, I think they do a pretty good job with the resources they have. Sure, they might miss a few things or not realize that what they thought was supposed to happen was actually meant to happen in a completely different way. In addition, don't those mistakes even out over the long run? Like all stats, they are indicators of trends and not the "truth" of every single player. Do you think it's a product of "a guy's "stats" look good is because the guy watching wants him to look good" that JJ Watts has a great score and Dunbar doesn't? If the "stats" are so worthless then you'd expect players like Dunbar to score well too because it's all just random noise right?

"I reported something paltry like 1 knockdown and 3 pressures. I then proceeded to watch them make the graphic with 7 knockdowns and 9 pressures to more match what Madden was saying" - This is the information age Fatbot and nobody would try to get away with stuff like that anymore because it's way too easy to find the truth. Do you realize that on the one hand you're saying these lay people don't know anything and on the other hand they're skewing the data to make the players they know nothing about look artificially good or bad? I can't take that example seriously at all.

Are their scores a perfectly accurate representation of every players abilities? No. When Mason ran for 117 yards in 14 carries did the fact that 89 of those yards came on one play where he ran through a hole that even I could have gained significant yardage in had I been the RB mean that the data is worthless? How about if he runs for 117 yards every game? Does that tell you anything worthwhile? If your using stats for anything other than trends, general information and comparison purposes I think you're looking in the wrong direction with any stat you look at.

They're not the gospel but they're certainly not worthless IMO.

Tell me where and how I'm wrong.

I can't get past the fact that by any measure, PFF's "stats" are subjective by definition. They are PFF's opinion on who they think played well or bad without knowing any player's actual responsibility on any given play. For some reason though, some do in fact (I can think of one guy in particular, not you Alan) take it as absolute gospel.

 

[Alan]

jjab360 looking for perfection:
I can't get past the fact that by any measure, PFF's "stats" are subjective by definition. They are PFF's opinion on who they think played well or bad without knowing any player's actual responsibility on any given play.

So if a player tackles a RB and they record that in the "total tackles" column is that subjective or objective? You and I might ask whether the player tackled him behind the line of scrimmage, at the line of scrimmage or 50 yards down field. We might further ask whether that tackler is a DT or a FS. We didn't get much useful information from that tackle stat did we? But if we look at MLBs and we see that JL has 140 tackles and Mosely has 89 can we make an educated assumption based on that data? Didn't the guy recording the stats make the same subjective analysis on every player thus leveling the playing field? If JL has more tackles than every other MLB can we assume he's a good MLB? I'd say yes. Can we assume he's better than the guy who has the second most tackles? I'd say no.

I don't think their data is all that subjective as in most cases they are simply recording what they see. People who use their data do come to subjective conclusions. Which is why you shouldn't bet the farm on any stats.

 

[Boffo97]

Doesn't this say a lot about their grading system?

Kind of like how ESPN'S QBR garbage was essentially admitting it was a failure when Jay Cutler or some other also ran was rated higher than Peyton Manning?

 

[ZigZagRam]

View attachment 4599 View attachment 4596

As you can see, like the rest of the defense, he started slowly but improved as the season wore on.

Thanks Leo!

That's pretty much in line with what I thought. He was so good in that Denver game.

 

[Fatbot]

Needless to say I disagree with your assessment of PFF... Tell me where and how I'm wrong.

Thanks for the detailed reply and in return here's a long post that I wouldn't blame you for not reading. The TL;dr is I don't think you're wrong, I just think by getting caught up in the details you're not seeing the forest for the trees, which is the trap PFF likes to pull to sell their product.

You can read their methodology, they post it: https://www.profootballfocus.com/about/grading/ . Maybe that convinces some people, but to me it just reads like a snake oil salesman drooling over all the easy money. They even give a couple actual accuracy numbers to make us feel warm & fuzzy, like "we’re confident that our work is very close to 100% by mid-week (actually stands at 99.98%)".

Problem is, that number only applies to their "player participation" data. "As for the grading ... we feel as strongly about the accuracy of the grades."

Oh, ok, well if they "feel strongly", I guess the grading is fine? Why don't they just say their grading is 99.98% accurate, too? Because they don't want to lie. Grading is always subjective, you can't call it any percent of accurate. So they explain, explain, explain this "scientific, foolproof, so-many-pairs-of-eyes-it-can't-fail method" for everyone to trust (and pay them for).

Hey, I take my hat off to them. It's a few guys from the UK that never played football that are now rich off it. I should do the same thing and sell the English Premier League a bunch of my stats that me and my dog watched (four pairs of eyes) -- and I've actually played soccer, er football!

But instead of bogging down into the minutia of their process, the bottom line is no system can do what people want from PFF.

Just an easy example, look at the safety grades posted above. Antoine Bethea is ranked as the #6 top coverage safety in the NFL. But here's what PFF said about Bethea last year: "Where Bethea struggles is in coverage, particularly on the deep ball ... his -4.9 grade in coverage was 17th-worst among safeties ... Now on the wrong side of 30, it’s reasonable to think that his coverage protection will only decline from here."

So Bethea puts on the SF jersey, drinks from the fountain of middle age, and poof! magically went from not being able to cover a wet paper bag to one of the best in the NFL?

Now, note he replaced Donte Whitner in 2013, who "had the fifth-best +10.5 rating in coverage". So let's see, the SF safety was 5th best.. they replaced him with the 17th-worst.. and now he's 6th best.

In hindsight, maybe this makes sense? Maybe the SF defense scheme makes the safety position shine so makes Bethea look great now? Maybe if PFF reported their grades as simply nameless positions like "SF safety position coverage grade is 8.4" that would make more sense. Maybe if they applied a team/scheme discount to each player it would make sense? But what about the fact Eric Reid sucks ballz now, so maybe it's not just the team making safeties look good. And at some point, don't the players make the team?

When SF idiots were crying at the time "SF can't lose Whitner, he's 5th best according to PFF!" and "SF safety is going to suck now because PFF graded Bethea 17th-worst", that was wrong. The PFF grades overrated Whitner and were screwing Bethea.

Perhaps the blame should be put on the media and those that derive the wrong conclusions from the stats, not PFF.

But then the question is, what is the "right" use of the PFF grades that shouldn't get any blame?

Pretty much any way anybody uses the PFF grade -- whether to state a guy is good, sucks, average, whatever -- is so subjective that it is basically worthless. The real success of PFF is it doesn't say anything new. If a guy is good, anybody watching the NFL already knows. But they scurry over to PFF and see it backed a number, and it makes them feel like a big expert to announce it to the world "hey I think so-and-so is the best, and just look at his PFF grade that proves I'm right!"

PFF just makes money off reassuring someone's opinion with the facade of "legitimacy". And next year, when the grade is reversed and it was so obviously wrong in hindsight, nobody remembers to mention it.

 

[Mojo Ram]

I take these stats with a grain of salt.

Like this? Me too.

 

[Alan]

Fatbot finding his second wind:
I just think by getting caught up in the details you're not seeing the forest for the trees, which is the trap PFF likes to pull to sell their product.

Problem is, that number only applies to their "player participation" data. "As for the grading ... we feel as strongly about the accuracy of the grades."

Pretty much any way anybody uses the PFF grade -- whether to state a guy is good, sucks, average, whatever -- is so subjective that it is basically worthless.

It's certainly always better to make decisions about a player by your own personal observations. It's also always dangerous to get caught up in anything that narrows your focus.

I like to take judicious advantage of other types of input too. While I disagree with your contentions, there's more than one way to skin a cat. Many others don't think much of stats either so you're not alone.

 

[leoram]

Fatbot, while I appreciate how well thought out your perspective is, I don't know of a better methodology available to the public. And while they may purport a player as dramatically different from year to year, players and teams can perform much differently on an annual basis (see 49ers).

While their data is empirically flawed, once again, it is a compilation that gives a general reference I find valuable. As for the money I paid, they were entertainment funds happily spent.

Nevertheless, I get your point. Team performance affects individual grading as does the assumption of assignment. I'd say no position is more affected than the QB. Though they try to counteract that issue, there is no perfect system.

In their defense, it seems they knew better about Davin Joseph than the Rams.

 

[RaminExile]

Aaron Donald #1 4/3 DT in the NFL.
Stedman Bailey #15 WR in the NFL.
Alec Ogletree #27, Ouch.
James Laurinaitis #53 Ugh!
EJ Gaines #24 CB, #12 Special Teams. Nice

Cowgirls #1 overall team grade 101.6
Rams -100.7
Bailey, Cook, and Saffold only positively graded offensive starters.
9 Primary defenders positively graded.
Hayes +11.9. Long -4.5.

Cant disagree with them to be honest. Would have been interested to see where Brian Quick would have ended up if he hadnt been hurt though. Clear offseason need is O-line (again....)

 

[fearsomefour]

I will put it this way, in 2007, maybe 2008 they were hiring "scouts" and I thought about being the Rams "film grader". But I didn't apply, but they were hiring fans. Not scouts. I have a fair football background but didn't think I was qualified. then, I saw who they did hire (and many of them may be gone by now) and realized that for the grades, they are not reliable. For their stats where they count things, and tally them they are okay, people can count sacks and passes defelcted and do a fair job, but unless you know the play, it's hard to make a call on a player's performance. Not always, sometimes you can see a guy get beat, but again, unless you know coverage or the protection, you will make errors.

I agree. This is my main issue with how they grade line play. You would really have to know the play and blocking scheme to grade accurately on a lot of O line stuff, run blocking esp.

 

[junkman]

Pretty much any way anybody uses the PFF grade -- whether to state a guy is good, sucks, average, whatever -- is so subjective that it is basically worthless. The real success of PFF is it doesn't say anything new. If a guy is good, anybody watching the NFL already knows. But they scurry over to PFF and see it backed a number, and it makes them feel like a big expert to announce it to the world "hey I think so-and-so is the best, and just look at his PFF grade that proves I'm right!"

There is a huge grey area between perfect and worthless. PFF grades are not as exact as, say, measuring someone's height or weight. It helps to think of PFF grades as quantified subjectivity, where they use the same method to subjectively grade every player in multiple skills on every play. Did that player beat the guy he was lined up against?

They do have acknowledged flaws. For instance, they don't consider quality of competition. If your team has a brutal schedule, that won't be considered in the grading. If a CB is lining up against the #1 WR vs #3 WR, that won't be considered in the grading. The (faulty) presumption is that competition evens out over the course of the year. Yeah... no. So take that all with a grain of salt.

PFF tends to like certain roles (playmakers) more than others. Thankless jobs like a non-penetrating but ran lane clogging NT don't get the high grades that a pass-rush specialist RDE would get. The have to make assumptions about scheme and assignment.

PFF also has a tendency to be too smitten with their own stats. For instance, the Rams were rated #31 coming into this year for the quality of their team. Did the Rams come in 31st place? Yeah, not even close. But because of they way they did the stats, basically just adding up the grades of each individual player, they arrived at the absurd conclusion.

Lastly, they don't consider trends (whereas a properly constructed Power Ranking would). How is the player or team playing now? Well, in most cases, the team is a different beast than when the season started. If you just look at the overall grade for the year, you won't see how TJ McDonald and Alec Ogletree have really come on since the KC game. If you only look at Chris Long's grade of -4.5, you'll be missing out on the fact that he's coming back from an injury, and should excel once more once he gets back to form.

All that said, if you understand the strengths and weaknesses in the grading system, you'll be able to use it as a tool effectively. If you take it at pure face value and consider it to be the be all end all of player grading, you'll be missing the point.

Just an easy example, look at the safety grades posted above. Antoine Bethea is ranked as the #6 top coverage safety in the NFL. But here's what PFF said about Bethea last year: "Where Bethea struggles is in coverage, particularly on the deep ball ... his -4.9 grade in coverage was 17th-worst among safeties ... Now on the wrong side of 30, it’s reasonable to think that his coverage protection will only decline from here."

So Bethea puts on the SF jersey, drinks from the fountain of middle age, and poof! magically went from not being able to cover a wet paper bag to one of the best in the NFL?

Now, note he replaced Donte Whitner in 2013, who "had the fifth-best +10.5 rating in coverage". So let's see, the SF safety was 5th best.. they replaced him with the 17th-worst.. and now he's 6th best.

In hindsight, maybe this makes sense? Maybe the SF defense scheme makes the safety position shine so makes Bethea look great now? Maybe if PFF reported their grades as simply nameless positions like "SF safety position coverage grade is 8.4" that would make more sense. Maybe if they applied a team/scheme discount to each player it would make sense? But what about the fact Eric Reid sucks ballz now, so maybe it's not just the team making safeties look good. And at some point, don't the players make the team?

When SF idiots were crying at the time "SF can't lose Whitner, he's 5th best according to PFF!" and "SF safety is going to suck now because PFF graded Bethea 17th-worst", that was wrong. The PFF grades overrated Whitner and were screwing Bethea.

All stats can be used incorrectly. Would Jordy Nelson be a top WR if he was catching balls from Kyle Orton rather than Aaron Rogers? Would the Rams be 6-8 if they played in the NFC South?

But then the question is, what is the "right" use of the PFF grades that shouldn't get any blame?

With a grain of salt, for sure. The reason PFF fans have no problem with the stats is because they tend to bear out bands of truth. If a guy is lining up every week and beating the snot out of the guy in front of him, PFF will bear this out. If a guy is a sieve, PFF will bear this out. And they will do their best to quantify every level in between. Is it a perfect and precise truth? Despite the quantification, of course not, but I still do value the quantification. If you consider the strengths and weaknesses of the stats, at the end of the day, the raw data they provide can give insights that can't be gotten anywhere else. The only other way you could arrive at these insights is by watching every game and grading every player for multiple skills on every play, just like PFF does.

 

[Dr C. Hill]

There is no working metric in this universe that has Bailey as the #15 WR in the league. I'm as big of a homer as the next guy, but lets be real, he does not even have 400 yards on the season!

I actually think PFF does a decent job at interpreting subjective stats, but their credibility just got T-Boned with that Bailey thing. Total nonsense!

http://espn.go.com/nfl/statistics/p.../sort/receivingYards/qualified/false/count/81

That is #101, and he has 1 touchdown and 27 catches on the year? How can this be?

 

[jjab360]

There is no working metric in this universe that has Bailey as the #15 WR in the league. I'm as big of a homer as the next guy, but lets be real, he does not even have 400 yards on the season!

I actually think PFF does a decent job at interpreting subjective stats, but their credibility just got T-Boned with that Bailey thing. Total nonsense!

http://espn.go.com/nfl/statistics/p.../sort/receivingYards/qualified/false/count/81

That is #101, and he has 1 touchdown and 27 catches on the year? How can this be?

I can actually see where they're coming from if you're looking at efficiency and not just bulk production.

 

[Dr C. Hill]

Please explain to me how Bailey has contributed as much as the #15 receiver in real numbers, AJ Green? Keeping in mind of course that Green is about to top 1,000 yards despite missing almost a quarter of the season with injury....

Again, complete nonsense!

 

[jrry32]

I will put it this way, in 2007, maybe 2008 they were hiring "scouts" and I thought about being the Rams "film grader". But I didn't apply, but they were hiring fans. Not scouts. I have a fair football background but didn't think I was qualified. then, I saw who they did hire (and many of them may be gone by now) and realized that for the grades, they are not reliable. For their stats where they count things, and tally them they are okay, people can count sacks and passes defelcted and do a fair job, but unless you know the play, it's hard to make a call on a player's performance. Not always, sometimes you can see a guy get beat, but again, unless you know coverage or the protection, you will make errors.

The problem with their site is that they try to break football down into an individual sport when doing the grades. It just doesn't work. They don't consider scheme or responsibilities(because they don't have the expertise) and grade individual match-ups. Additionally, they make some mistakes in their actual stats because they don't understand what's happening on the play (on the defensive side of the ball).

For example, they credit the nearest defender as giving up a completion.

 

[jrry32]

Fatbot, while I appreciate how well thought out your perspective is, I don't know of a better methodology available to the public. And while they may purport a player as dramatically different from year to year, players and teams can perform much differently on an annual basis (see 49ers).

While their data is empirically flawed, once again, it is a compilation that gives a general reference I find valuable. As for the money I paid, they were entertainment funds happily spent.

Nevertheless, I get your point. Team performance affects individual grading as does the assumption of assignment. I'd say no position is more affected than the QB. Though they try to counteract that issue, there is no perfect system.

In their defense, it seems they knew better about Davin Joseph than the Rams.

NFL Rewind.

 

[jrry32]

With a grain of salt, for sure. The reason PFF fans have no problem with the stats is because they tend to bear out bands of truth. If a guy is lining up every week and beating the snot out of the guy in front of him, PFF will bear this out. If a guy is a sieve, PFF will bear this out. And they will do their best to quantify every level in between. Is it a perfect and precise truth? Despite the quantification, of course not, but I still do value the quantification. If you consider the strengths and weaknesses of the stats, at the end of the day, the raw data they provide can give insights that can't be gotten anywhere else. The only other way you could arrive at these insights is by watching every game and grading every player for multiple skills on every play, just like PFF does.

And this is why I prefer to discuss football with knowledgeable fans of other teams. Yes, they're going to have some bias but they also are going to be able to give you better insight than PFF or just about any non-football mind out there on what players are worth a damn, what players aren't worth a damn, and what players might be about to turn that proverbial corner.

 

[leoram]

Please explain to me how Bailey has contributed as much as the #15 receiver in real numbers, AJ Green? Keeping in mind of course that Green is about to top 1,000 yards despite missing almost a quarter of the season with injury....

Again, complete nonsense!

First of all, Green is listed above Bailey. Secondly, PFF awards blocking as part of the game Green seems unwilling to contribute. You'll also notice Green has 244 more snaps. Finally, if you were to rearrange the formula for an overall grade, how would it look?