PFF (my first thread as OP)

[junkman]

PFF seems to be getting a lot of flak lately. Imho, PFFs biggest problem is the people who MIS-use their stats.

E.g. this just in:

• Adrian Peterson is the 12th best running back in the league (12 / 48 in YPC)
  
• Tom Brady is the league's 17th best QB (17 / 37 in QB rating, Bradford is 11)
• Calvn Johnson is the league's 132nd best receiver (132 / 151 in catches / targets, Givens is #151 btw)

Any stat can be mis-used. I think that PFF really started getting on people's bad side when someone blindly rolled up PFF's 2013 player scores for each team's 22 projected starters in 2014, and decided that the Rams were the 31st best team in the league. Among the problems with the analysis:

• It didn't consider the relative value of the positions
• It didn't consider the depth of the rosters
• It didn't consider improvement over the year (disfavors the Rams and their young / evolving roster)
• It didn't consider strength of opponent (which is true of all PFF stats but is also true of pretty much any NFL stat, but this really disfavors the Rams)
• It didn't consider strength of teammates (which is true of all PFF stats but is also true of pretty much any NFL stat)

No stats are perfect, and PFF is no exception to that. But there are things that PFF does very well. To repeat my challenges from a different thread,

• does anyone here think they can come up with a ranking of the best LTs in the league? Sorted by best run blocking? Pass blocking? Overall? Which players are ascending or descending year over year or even within the season?
• can you provide evidence that shows where Saffold plays better, G or T?
• can you provide evidence that shows if Davin Joseph over his injury?

Most of the PFF shortcomings (e.g. not considering strength of schedule, or not being sure about the assignment) will even out over the course of the year as statistical background noise. Regarding other statistical shortcomings, those things are in the hands of the people using the stats to apply some common sense before publishing some absurd set of conclusions.

From what I've seen, people who rail against PFF don't really understand what they do, and the reason they don't like PFF is anecdotally, because of articles (like the Rams @ #31 article or the Rams D at #22 article) which base their conclusions on PFF stats where the conclusions don't pass the sniff test.

Any stat can be mis-used, but I'm not abandoning YPC, QB rating or Rec/Target any time soon.

 

[-X-]

I don't have a problem with PFF, and I've used their site frequently in the past. And I do admire their dedication and attention to detail. The obvious complaint is that they can't possibly know the assignments or account for things outside of players' control (e.g., line of sight for QBs or a DE/DB battling an injury like turf toe); and because of that, they do tend to unfairly penalize (see: Cortland Finnegan's anomalous bad year due to an eye injury). But for practical analysis and substantiation of my own opinion, I find them useful to a degree. As you said, you have to use what they provide responsibly. There are far too many people who use them as the authority in player evaluation, and that's the very definition of irresponsible. You can absolutely point to them as a credible source, though, in my opinion.

 

[Memphis Ram]

From what I've seen, people who rail against PFF don't really understand what they do, and the reason they don't like PFF is anecdotally, because of articles (like the Rams @ #31 article or the Rams D at #22 article) which base their conclusions on PFF stats where the conclusions don't pass the sniff test.

I doubt the writer(s) of this piece below care one bit about the Rams articles. Some of us simply agree wholeheartedly with the points being made in it. Points that have been made time and time again even before the article was posted.

http://www.bostonsportsmedia.com/2014/06/can-pro-football-focus-stats-be-blindly-trusted

The Dangers In The PFF Method

Last August, Bill Belichick talked about the dangers of watching film and making conclusions based on it.

It might even look to us like somebody made a mistake but then we look at it more closely maybe somebody besides him made a mistake and he was trying to compensate. I think we need a little closer analysis a lot of times. Sometimes the play calls or what was called on the line of scrimmage might be something that we’re not aware of. That could happen in any game. You think a player did something that he shouldn’t have done but maybe he got a call, a line call or a call from a linebacker or he thought the quarterback said something so he did what he thought was the right thing or maybe it was the right thing but that call shouldn’t have been made or should have been on the other side. But yeah, I think we need to be careful about what we’re evaluating.

So sometimes even the team itself doesn’t know exactly where things broke down and who did what wrong. Belichick then went on to talk about watching opposing team’s game films and the impossibilities of knowing what happened:

But believe me, I’ve watched plenty of preseason games this time of year and you’re looking at all the other teams in the league and you try to evaluate players and you’re watching the teams that we’re going to play early in the season and there are plenty of plays where I have no idea what went wrong. Something’s wrong but I don’t…these two guys made a mistake but I don’t know which guy it was or if it was both of them. You just don’t know that. I don’t know how you can know that unless you’re really part of the team and know exactly what was supposed to happen on that play. I know there are a lot of experts out there that have it all figured out but I definitely don’t. This time of year, sometimes it’s hard to figure that out, exactly what they’re trying to do. When somebody makes a mistake, whose mistake is it?

Bill Belichick doesn’t have it figured out. But Pro Football Focus does? They can provide a grade on every play?

 

[junkman]

I doubt the writer(s) of this piece below care one bit about the Rams articles. Some of us simply agree wholeheartedly with the points being made in it. Points that have been made time and time again even before the article was posted.

http://www.bostonsportsmedia.com/2014/06/can-pro-football-focus-stats-be-blindly-trusted

The Dangers In The PFF Method

...

Bill Belichick doesn’t have it figured out. But Pro Football Focus does? They can provide a grade on every play?

"Bill Belichick doesn’t have it figured out. But Pro Football Focus does? They can provide a grade on every play?"​

Frequently debunked criticism of PFF. They don't provide a grade on every play, just the ones where they are sure of what happened.

https://www.profootballfocus.com/about/grading/

4) The “Rules” of Grading

Because of the nature of the roles, each position is graded in a slightly different way and the definitions for each run on for many pages. Although we’re not going to publish our 30+ page document on how we do this, not least because that’s our IP, below are a few of the key principles in our grading methodology:
.

• DON’T GUESS — If you’re not 95 percent sure what’s gone on then don’t grade the player for that play. The grades must stand up to scrutiny and criticism, and it’s far better to say you’re not sure than be wrong.

It is, however, crucial that this is not seen as an excuse to shy away from making a judgement. What we definitely do not do is raise or lower the grading because we’re not sure. Giving a grade of -0.5 rather than -1.5 for a player on an individual play because you’re unsure is the wrong grade to give. If the grader is 95 percent sure of the severe fault on the play, the grade is -1.5. If, however, the grader is unsure of his judgment, the correct grade is 0.​

~~~~~~~~~~~~~~~


The author of that article goes on to say:

A couple things I don’t like here. How does the grader know whether they are 95% certain or just 90%? How many plays per game are going ungraded because a determination cannot be made?

Later, in the section which asks How subjective is the Grading?

Just like with the more mainstream statistics, there are occasions when the choice is difficult. But the difference on our site is this: If a guy is going to be upgraded or downgraded on a judgment call, we let it ride. We simply make the comment and then put in a 0.

Again, how often is this happening? It seems like it wouldn’t take many “0″ grades to skew the data.

Lastly, I hesitate to bring this part up, but part of me wonders the qualifications for doing this work. It feels like me taking a job to to play-by-play film breakdown on the Premier League. What are the football coaching or scouting backgrounds for these UK analysts making these grades? Is there anyone on staff with an NFL background?
​

Hmmm... soo
1) not surprising that a guy from Boston doesn't like the British.
2) he mentions the things he doesn't KNOW about the grading policy, but instead of calling PFF to find answers, he just dismisses them. Sounds pretty lazy to me.

But he does come to the right conclusions in the end:

Honestly, I don’t know. As mentioned above, I do feel there is some merit and value to the work that Pro Football Focus is putting in. I just don’t get the slavish devotion to their grades that I see when I read many NFL articles.

...

I believe the NFL media as a whole needs to be a little more judicious in how they use these stats instead of blindly accepting what comes out of the PFF factory.
​

Yep. Absolutely true! But the same can be said for any stats (e.g. Calvin Johnson and his low Rec/Target).

 

[-X-]

If the grader is 95 percent sure of the severe fault on the play, the grade is -1.5. If, however, the grader is unsure of his judgment, the correct grade is 0.

This may sound snarky, but how in the FAT hell does one quantify what percentage of "sure" they are?

 

[VegasRam]

I don't have a problem with PFF, and I've used their site frequently in the past. And I do admire their dedication and attention to detail. The obvious complaint is that they can't possibly know the assignments or account for things outside of players' control (e.g., line of sight for QBs or a DE/DB battling an injury like turf toe); and because of that, they do tend to unfairly penalize (see: Cortland Finnegan's anomalous bad year due to an eye injury). But for practical analysis and substantiation of my own opinion, I find them useful to a degree. As you said, you have to use what they provide responsibly. There are far too many people who use them as the authority in player evaluation, and that's the very definition of irresponsible. You can absolutely point to them as a credible source, though, in my opinion.

I am impressed. I have a decent vocabulary, am a voracious reader, never need to use spellcheck, and have never SEEN let alone used, the word anomalous.:bow:

 

[-X-]

I am impressed. I have a decent vocabulary, am a voracious reader, never need to use spellcheck, and have never SEEN let alone used, the word anomalous.:bow:

Well thanks. Glad I could expand your arsenal by one bullet.

 

[Jorgeh0605]

This may sound snarky, but how in the FAT hell does one quantify what percentage of "sure" they are?

With statistics. If you recognize the play as being a certain type of play. Say, a simple dive through the 2 hole. You can compare it to other plays which are the same that the team has run before. Plot a frequency distribution, perform a inferential statistics test (that is set to allow 5% error, which is a norm in statistics) and determine whether or not that particular event is so unlikely to happen that it can't possibly be the same play, or it is likely enough to happen (95%) that you accept that it is the same play.

With that being said, I highly doubt they do that for EVERY play. Though it is probably similar to the evaluation process occurring only it isn't calculated, its the brains "best guess".

My point was that stats are a miraculous thing and have been used to quantify seemingly un-quantifiable information. But you have to use them for what they are worth and always accept that whatever error rate you set. Like I said, the normal amount is 5%, this is applicable to most science and behavioral science stats as well.

 

[-X-]

With statistics. If you recognize the play as being a certain type of play. Say, a simple dive through the 2 hole. You can compare it to other plays which are the same that the team has run before. Plot a frequency distribution, perform a inferential statistics test (that is set to allow 5% error, which is a norm in statistics) and determine whether or not that particular event is so unlikely to happen that it can't possibly be the same play, or it is likely enough to happen (95%) that you accept that it is the same play.

With that being said, I highly doubt they do that for EVERY play. Though it is probably similar to the evaluation process occurring only it isn't calculated, its the brains "best guess".

My point was that stats are a miraculous thing and have been used to quantify seemingly un-quantifiable information. But you have to use them for what they are worth and always accept that whatever error rate you set. Like I said, the normal amount is 5%, this is applicable to most science and behavioral science stats as well.

Interesting. Thanks.

 

[LACHAMP46]

I am impressed. I have a decent vocabulary, am a voracious reader, never need to use spellcheck, and have never SEEN let alone used, the word anomalous.:bow:

Oh my, per Websters:

Full Definition of ANOMALOUS
1
: inconsistent with or deviating from what is usual, normal, or expected : irregular, unusual
2
a : of uncertain nature or classification

b : marked by incongruity or contradiction : paradoxical

I like this board..learn something new errday

Now on to PFF, in the ultimate team game, TEAM Game...almost nothing is equal...I could go on to the comparisons of Barry Sanders vs Emmitt Smith, that is different o-lines, play calls, QB's and so forth..to coaching styles...no to instances are entirely equal, and there are too many variables. G-Rob is rated lower than Taylor Lewan this past week. Is Taylor better, this basic question is what most fans want to know, can PFF discern/tell/grade/estimate this? maybe or maybe not...For a site to proclaim they can, with any statistical certainty is misleading.

 

[VegasRam]

Well thanks. Glad I could expand your arsenal by one bullet.

At my age, I need all the bullets I can get.

 

[LACHAMP46]

To junkman, this is a great first post. Well done. This is an interesting topic, as I have always wondered why some believe so much in PFF & FO and base opinions and ideas on their stats daily. In almost every blog, some writer will say something about a player then say as confirmation, "PFF has so & so rated..." As a ole school fan, it took awhile b4 I accepted the numerous baseball stats as proof, to see them evade the game football finally overcome with them now is, just different. I saw a quote once, : there are 3 types of lies, lies, damn lies, and statistics" Or something like that. I trust my eyes, and a bunch of other eyes help a whole lot too. You can't see everything.

 

[junkman]

Oh my, per Websters:

I like this board..learn something new errday

Now on to PFF, in the ultimate team game, TEAM Game...almost nothing is equal...I could go on to the comparisons of Barry Sanders vs Emmitt Smith, that is different o-lines, play calls, QB's and so forth..to coaching styles...no to instances are entirely equal, and there are too many variables. G-Rob is rated lower than Taylor Lewan this past week. Is Taylor better, this basic question is what most fans want to know, can PFF discern/tell/grade/estimate this? maybe or maybe not...For a site to proclaim they can, with any statistical certainty is misleading.

Anomaly I knew. Anomalous, I had to read twice.

PFFs stats are not that worthwhile after a single game. It's better to look at them after a bunch of games / bunch of seasons to see general levels of performance or performance trends. Any fans who concern themselves with who is better right now between Lewan and GRob would be debating how many angels can dance on the head of a pin.

That said, no stats are really all that worthwhile after a single game. For RBs... YPC, YAC, total yards... after a single game, they just make up a single point on a chart. That said, there are many great ways to quantify what RBs and WRs and QBs do, so I tend not to look at PFF for stats on those guys except when we're talking about blocking. DL / OL / CB / LB, the PFF stats are pretty compelling.

Ask me at the end of the year about GRob and Lewan, and I'll be happy to tell you what PFF said.

 

[RamzFanz]

PFF seems to be getting a lot of flak lately. Imho, PFFs biggest problem is the people who MIS-use their stats.

E.g. this just in:

• Adrian Peterson is the 12th best running back in the league (12 / 48 in YPC)
  
• Tom Brady is the league's 17th best QB (17 / 37 in QB rating, Bradford is 11)
• Calvn Johnson is the league's 132nd best receiver (132 / 151 in catches / targets, Givens is #151 btw)

Any stat can be mis-used. I think that PFF really started getting on people's bad side when someone blindly rolled up PFF's 2013 player scores for each team's 22 projected starters in 2014, and decided that the Rams were the 31st best team in the league. Among the problems with the analysis:

• It didn't consider the relative value of the positions
• It didn't consider the depth of the rosters
• It didn't consider improvement over the year (disfavors the Rams and their young / evolving roster)
• It didn't consider strength of opponent (which is true of all PFF stats but is also true of pretty much any NFL stat, but this really disfavors the Rams)
• It didn't consider strength of teammates (which is true of all PFF stats but is also true of pretty much any NFL stat)

No stats are perfect, and PFF is no exception to that. But there are things that PFF does very well. To repeat my challenges from a different thread,

• does anyone here think they can come up with a ranking of the best LTs in the league? Sorted by best run blocking? Pass blocking? Overall? Which players are ascending or descending year over year or even within the season?
• can you provide evidence that shows where Saffold plays better, G or T?
• can you provide evidence that shows if Davin Joseph over his injury?

Most of the PFF shortcomings (e.g. not considering strength of schedule, or not being sure about the assignment) will even out over the course of the year as statistical background noise. Regarding other statistical shortcomings, those things are in the hands of the people using the stats to apply some common sense before publishing some absurd set of conclusions.

From what I've seen, people who rail against PFF don't really understand what they do, and the reason they don't like PFF is anecdotally, because of articles (like the Rams @ #31 article or the Rams D at #22 article) which base their conclusions on PFF stats where the conclusions don't pass the sniff test.

Any stat can be mis-used, but I'm not abandoning YPC, QB rating or Rec/Target any time soon.

That's the worst first OP I've ever seen. My god man.

 

[CodeMonkey]

Nice post junkman. Thanks for being here.

 

[junkman]

With statistics. If you recognize the play as being a certain type of play. Say, a simple dive through the 2 hole. You can compare it to other plays which are the same that the team has run before. Plot a frequency distribution, perform a inferential statistics test (that is set to allow 5% error, which is a norm in statistics) and determine whether or not that particular event is so unlikely to happen that it can't possibly be the same play, or it is likely enough to happen (95%) that you accept that it is the same play.

With that being said, I highly doubt they do that for EVERY play. Though it is probably similar to the evaluation process occurring only it isn't calculated, its the brains "best guess".

My point was that stats are a miraculous thing and have been used to quantify seemingly un-quantifiable information. But you have to use them for what they are worth and always accept that whatever error rate you set. Like I said, the normal amount is 5%, this is applicable to most science and behavioral science stats as well.

Wow, that was some kind of reply! :notworthy:

I was going to just say it's like predicting the weather (which usually triggers a snarkly reply) and then refer them to Nate Silver's "The Weatherman Is Not a Moron" from the "Signal and the Noise", but helpfully published in the NY TImes. (http://www.nytimes.com/2012/09/09/magazine/the-weatherman-is-not-a-moron.html). Your answer was much better!

 

[junkman]

That's the worst first OP I've ever seen. My god man.

I was like ... WTF?!? Is he serious? No blue font? No smiley? Dude with 3k+ posts should know better! Scroll scroll scroll... oh, there it is.

Oldest trick in the book and I actually fell for it. In my defense, I was probably distracted by your avatar. If that part of my mind is occupied, nothing else works.

 

[Noregar]

I do not like PFF. I have been an engineer for almost 25 years and far too often I have seen people blindly use numbers spat out by an equations/computer programs without considering if the results actually make sense. To me PFF does the same thing.

Related to the Rams 31st team ranking that is referenced in the OP, someone did not "blindly roll up PFF's 2013 player scores" that ranking was actually done by PFF in conjunction with ESPN. PFF also put their name on that ranking so they own it. I had already questioned their methods but that # 31 ranking of the Rams was completely idiotic and goes a long way to prove that there are serious flaws with their methods. The argument for misuse of PFF's data/stats/rankings holds no water with me because the same people (PFF) who chose to misuse the data created the data. PFF (not someone else) used THEIR junk science to rank the Ram’s defense 22nd while ranking the defensive tackles 27th and their linebackers 30th. NONE of those rankings are even realistic for anyone that knows this team.

Please do not try to sell me on the virtues of Pro Football Focus. I do not like them and they disrespect the Rams with their rankings. I do respect that fact that most of us here are Ram fans and thus we have that as common bond but we all have varying opinions on players, coaches, and even PFF.

For those who want to use PFF enjoy the fantasy but you should also understand that quoting/referencing them will actually hurt your credibility in some circles.

 

[Angry Ram]

PFF seems to be getting a lot of flak lately. Imho, PFFs biggest problem is the people who MIS-use their stats.

E.g. this just in:

• Adrian Peterson is the 12th best running back in the league (12 / 48 in YPC)
  
• Tom Brady is the league's 17th best QB (17 / 37 in QB rating, Bradford is 11)
• Calvn Johnson is the league's 132nd best receiver (132 / 151 in catches / targets, Givens is #151 btw)

Any stat can be mis-used. I think that PFF really started getting on people's bad side when someone blindly rolled up PFF's 2013 player scores for each team's 22 projected starters in 2014, and decided that the Rams were the 31st best team in the league. Among the problems with the analysis:

• It didn't consider the relative value of the positions
• It didn't consider the depth of the rosters
• It didn't consider improvement over the year (disfavors the Rams and their young / evolving roster)
• It didn't consider strength of opponent (which is true of all PFF stats but is also true of pretty much any NFL stat, but this really disfavors the Rams)
• It didn't consider strength of teammates (which is true of all PFF stats but is also true of pretty much any NFL stat)

No stats are perfect, and PFF is no exception to that. But there are things that PFF does very well. To repeat my challenges from a different thread,

• does anyone here think they can come up with a ranking of the best LTs in the league? Sorted by best run blocking? Pass blocking? Overall? Which players are ascending or descending year over year or even within the season?
• can you provide evidence that shows where Saffold plays better, G or T?
• can you provide evidence that shows if Davin Joseph over his injury?

Most of the PFF shortcomings (e.g. not considering strength of schedule, or not being sure about the assignment) will even out over the course of the year as statistical background noise. Regarding other statistical shortcomings, those things are in the hands of the people using the stats to apply some common sense before publishing some absurd set of conclusions.

From what I've seen, people who rail against PFF don't really understand what they do, and the reason they don't like PFF is anecdotally, because of articles (like the Rams @ #31 article or the Rams D at #22 article) which base their conclusions on PFF stats where the conclusions don't pass the sniff test.

Any stat can be mis-used, but I'm not abandoning YPC, QB rating or Rec/Target any time soon.

I don't like PFF, because they are taking an ugly game and overanalyzing the hell out of it. And that's positive or negative.

Like I said in the Jake Matthews thread, I could care less if a offensive lineman has a -.01325 whatever grade. Like...OK.

As to your questions.

1. No, I can't but I use the eye test. Is the dude constantly being pushed back or is he the one doing the pushing? What does a arbitrary number gonna do? Compare it to the starting tackle of the Chargers? I really don't care about that.

2. The only evidence is if he's not starting games for any other reasons except injury. If he is and not playing well, that means there is no one better and the position needs to be upgraded somehow. I.E. Chris Williams last year.

3. Yeah, if he's active and/or playing on gamedays. If he's still hurting, chances are his play will reflect that.

Stats are skewed anyway. Every single one of them. But PFF bends and twists them to a point where they suck the joy of watching football.