I don't remember where I read it originally but it was reposted here so if you use the right search parameters you should be able to find it.
I don't think they mentioned the names of the teams. I think it might have been something to do with evaluating college players. Maybe. Or an article about PFF itself. Since it was posted here I'm sure others remember it and hopefully chime in.
Try PFF and draft or evaluating college players for the draft as search parameters to start out with.
Just found this while I was Googling PFF and working with the NFL. It's not about your question but it's interesting.
https://www.profootballfocus.com/about/grading/
Grading
Why do we grade?
The goal of our detailed grading process is to gauge how players execute their roles over the course of a game by looking at the performance of each individual on each play. We look beyond the box score and study the tape to grade how well each player performs on each play, and as a result over the course of a game and a season. How well a lineman blocks on a given play, how much space and help a runner receives, how effectively a pass rusher brings pressure or how well a defender covers a receiver.
We collect lots of extra statistics such as yards after catch, yards after contact, missed tackles, dropped passes, etc., but our real focus is on grading individual performance on each play. Did an offensive lineman seal his block to spring the runner through a hole? Did a defensive lineman beat his block to force a runner to change the play direction in the backfield? Was the crucial third-down completion due to the quarterback beating the coverage or a breakdown in coverage?
We examine not just the statistical result of a play, but the context of that statistic. The defensive tackle may have made a tackle on a play, but if it was 3rd-and-5 and he got blown 4 yards off of the ball to make the tackle after a 6-yard gain, that’s not a good play.
We’re fully aware that statistics can lie, and the grading is designed to give a true gauge of what happened on a given play by isolating the individual performances as best we can in such a team-driven sport. If the quarterback throws an accurate first down conversion that is dropped, the quarterback receives the same credit as he would have with a catch.
Conversely, a cornerback who gets beat and “benefits” from either a dropped pass or a poor throw from the quarterback, gets docked just as he would have with a completion. In both examples, the result is 0-for-1 — a poor statistical look for the quarterback and a statistical positive for the cornerback — but our grades will accurately reflect their true performance as a positive and a negative, respectively.
The combination of grades and unique statistics allow us to evaluate individual player performance in each game. We present base statistics alongside more advanced statistics together with a grade for every player. The marks are presented as overall composite grades constructed from a number of key areas:
OffenseDefense
● Running
● Run Defense
● Passing and Receiving……….● Pass Rushing
● Pass Protection● Pass Coverage
● Run Blocking● Discipline & Procedure
● Screen Blocking
● Discipline & Procedure
What Do We Grade?
Throughout the course of the season (regular season and playoffs) we grade every single offensive, defensive and special teams snap. We log data such as the point of attack of a running play, the location a pass was thrown and hang time of kicks and punts before moving on to the player-performance analysis.
Our detailed comment system allows us to go well beyond the grades to dig even deeper into a player’s skill set. Instead of just looking at a player’s grade to determine that he’s a good run blocker, we can see how successful he was on pull blocks or blocks at the second level. If a cornerback is getting beaten often, is there a certain route causing him difficulties? All of the data is collected in order to build a detailed picture of each player’s performance and production over the course of a season.
The Basics
Each grade given is between
+2.0 and
-2.0, with
0.5 increments and an average of
0.0. A positive intervention in the game earns a positive grading and vice-versa. Very (very) few plays draw a +/-2.0 rating.
Another way of looking at the “0” grade is viewing it as the “expected” grade for an NFL player. We expect an NFL quarterback to accurately throw a 6-yard curl route against off-coverage on 1st-and-10, and whether or not his WR gets tackled for a 6-yard gain or breaks a tackle for a 70-yard touchdown, the quarterback’s grade remains the same.
The average grade, or what we would typically expect of the average player, is therefore defined as zero. In reality, the vast majority of grades on each individual play are zero and what we are grading are the exceptions to this.
The varying degrees of positive and negative grades add a little bit more context than a simple plus and minus systems. An offensive lineman might surrender a sack on a given play, but how quickly was he beaten? Allowing a defender to slip past and get into the quarterback’s face in 1.9 seconds is obviously much worse than allowing that same sack in 2.7 seconds, so while both plays are negatives, they certainly won’t carry the same exact grade.
Using that same example, if the pressure is surrendered in 1.9 seconds and the quarterback sidesteps the sack, it’s certainly not indicative of the offensive lineman’s pass-blocking acumen, so there’s no reason to change his grade because he “only” surrendered a pressure and not a sack. The goal of the grading is to isolate individual performance as much as possible, fully realizing that there is always a certain level of dependency on teammates in football.
Performance-Based Scouting
We offer a different type of scouting, strictly based on performance and not technique or upside. Obviously after watching thousands of plays per week we have a feel for what good technique looks like, but we’re looking for how well a player did his job. We are looking for the result of that poor technique, not the poor technique itself. If poor technique results in a positive play, that is graded at the same level as good technique yielding a positive play. Did the lineman make the block he attempted, by whatever means? We realize that, over time, poor technique will lead to more negative plays, but our emphasis is on each individual play, and sometimes poor technique gets the job done.
Traditional scouting may describe a player’s explosiveness off the edge, but we can tell you how often a rusher actually got pressure to the outside shoulder of the offensive tackle. Other scouting reports will often describe a cornerback’s hip swivel, but we’ll look at how well he actually plays in coverage. A player’s athleticism is irrelevant in our system, unless it leads to productive on-field performance.
Essentially, we’ve created a new type of scouting that strictly looks at performance, not necessarily the process that gets there. In our dealings with NFL clients, we’ve referred to this as supplying the “what” as they supply the “why.” We can tell a team that an offensive tackle gives up an inordinate amount of bullrush pressure and they can determine if it’s a lack of technique, functional strength, or perhaps a combination of the two.
Quantifying what we see on film allows us to funnel out the “highlights” that our brain tries to create when evaluating players.
How We Grade
There are two main processes that are used to come up with the grades: player participation and the actual grading process. Player participation is more black and white as it consists of which players are on the field, where and general information about the actions they performed (rush the passer, drop into coverage, stay in to pass block, etc).
This process is performed by two separate analysts on two separate spreadsheets. Each spreadsheet is then compared and any discrepancies are reviewed by a third analyst in order to ensure near-100 percent accuracy. We’ve found that both blind runs are generally at least 99.7% accurate and once the games are checked, we’re confident that our work is very close to 100% by mid-week (actually stands at 99.98%).
As for the grading process, it starts as soon as a game is completed. Each play consists of two basic sections, one for the grading and one for stat collection, which would include detail such as (but by no means limited to) which gap the run goes through or which route was thrown on a pass.
The grading for each game passes through three individuals (two who grade every player on every play) to ensure that the grading and stat collection is thoroughly checked and critiqued. This ensures that both our procedures and the interpretation of each play is rigorously checked and consistently applied.
Using All-22
While we feel strongly about our ability to grade games based on the broadcast footage, the All-22 has been an invaluable addition to our processes. The original analyst is instructed to flag any plays from the broadcast footage that need more information or a better view from the coach’s film. The second and third analysts are then able to pinpoint these plays along with others to get a clearer, more decisive look at every play. The use of All-22 has also allowed us to expand our analysis of special teams plays into greater depth and breadth than is possible from broadcast footage.
When We Grade
Each game is graded within the first 24 hours of completion and new for 2014, our goal is to have every game completed and up on the site by 8:00 AM Monday morning. Our review processes are then completed no later than the end of Tuesday which allows us to finalize our grades and statistics for the week by the end of Wednesday.
Normalization and What the Grades Mean
Once we’ve got the raw grades we could leave it there, but this would lead to a number of problems.
For example, our pass protection grading methodology is a fault based system, only negative grades are awarded. Consequently the “perfect score” as a raw grade is 0, However, what if a lineman plays half the number of passing plays of another guy and they both score 0? What allows you to understand the second has done the better job? This is where Player Participation comes in: To fully understand how a player has performed, we need to know how many plays he’s participated in and what role he performed.
So when we look at, say, a tight end, we need to know how many plays he spent out in pass routes, how many times he blocked for the run and how many times he stayed in to block for the pass. To this number we then apply a normalization factor to set the AVERAGE player in that facet of the game to 0.
What you may notice looking at the “By Position” tab is that not all of the average grades in each season comes to 0. This is because the same normalization factors are calculated from a number of seasons and are applied to all seasons. This allows for comparison of performances at the same position not only within seasons but across them.
Normalization gives the grades their full setting as a performance indicator for an individual over his full body of work on a per-game, per-season and per-play basis.
How subjective is the Grading?
Many people say that as soon as you start grading, you bring subjectivity into your work. Obviously, to some degree, that’s true.
However, there’s also subjectivity around whether a play was a QB run for negative yardage or a sack, if an assist on a tackle should be awarded and if a catch was dropped or not. Sure, you can come up with a set of rules to determine which is which, but in the end, at the borderline between one and the other, it’s always subjective. It comes down to a judgment call.
The real trick of grading is to define a clear set of rules, encompassing each type of play. If your rules are thorough and precise enough, the answers just fall out. It becomes as easy as determining the dropped pass that hit the TE right between the numbers.
Statistics in their raw form are considered objective. But in our opinion, with the small number of NFL games played each season, raw stats are very often unintelligent. If a QB throws three interceptions in a game but one came from a dropped pass, another from a WR running a poor route and a third on a Hail Mary at the end of the half, it skews his stats by far too great an amount to be useful. Our grading allows us to bring some intelligence to the raw numbers and with many different sets of eyes getting a look at each game, we’re able to avoid the individual biases that may arise if only one person was responsible for grading the game.
How accurate are the Statistics and Grading?
Our player participation data has been confirmed as 99.96% accurate. As for the grading, through our interactions with many NFL teams, we feel as strongly about the accuracy of the grades. Whether looking at teams in a broad snapshot or on a play-by-play basis, our feedback from NFL personnel departments has always been extremely positive.
Although we are more than happy with the accuracy (and the constant improvement in accuracy) of the individual areas of our grades, the final “overall” grade is one that is cause for much discussion among NFL fans and the PFF staff as well.
For example, we’re happy with how the pass-blocking and run-blocking grades are constructed for offensive linemen, the balance those two skills are given to create an overall grade will vary from fan to fan, coach to coach and scheme to scheme. This is an area we are hoping to improve in the future, allowing fans to input their own weightings at the “By Position” page. This would provide the option to come up with their own overall grading depending upon how they view the different skills that make up an individual position.
FAQ
How accurate can grading be if you don’t know the call?
For the most part, it’s pretty clear what each player’s assignment is on a given play. Either way, we are grading what a player does and how he reacts on each play, regardless of the call. For an offensive lineman, even if we don’t know if an offensive lineman is asked to execute a reach block or simply ride his man out of the play, if he’s stood up at the line of scrimmage, it’s clear that he didn’t execute his assignment well.
There are certainly some instances during games present a clouded view of a player’s assignment, and in those instances, we err on the side of “0” grade as we try to avoid guessing as much as possible.
What makes you qualified to grade?
We have an extremely detailed grading process that is gaining more respect in NFL circles every year. Feedback from NFL personnel departments has been extremely positive, whether looking at players in a general sense or on a play-by-play basis. We feel that the process that has been developed creates a very accurate representation of what happens on NFL playing fields every season.
As for the training process, each analyst is put through a rigorous training process in order to get up to speed on our system so as to keep all analysts viewing the game through similar eyes. Our grading process has continued to evolve every year as we constantly try to improve how we evaluate football players.
While we acknowledge that our grading is not perfect, after watching thousands of plays every year, we feel that we have a good idea of what “good” and “bad” football looks like and positive feedback about our grading system has backed up that point.
Do NFL teams use your grades?
Yes. We have a number of direct clients within the NFL that use our data for personnel decisions and game management, we also have non-clients that have used our grades as an internal check against their own pro scouts. Most, if not all, teams have at least the bare minimum website subscription.
Our grades and stats are also heavily involved in contract negotiations as agents and front offices use our data to build their case either for or against potential free agents.
How open to analyst bias is the process?
There is always the chance that bias can creep into the analysis process, but we work hard to contain it through our training and review processes. Each analyst is trained extensively before being allowed to grade live games to ensure that everyone is on the same page. For each game, a first run is done on the game that is then checked by another analyst. If there are any questions about a particular grade, it is sent to one of our lead analysts for a final ruling. Due to the training and review processes, we feel strongly about our ability to remain as consistent as possible despite football being such a complicated game.
How can you be right 100% of the time?
This is similar to our previous question about grading plays without the play call. We don’t necessarily expect that our work will be 100 percent accurate (though it’s always the goal), but we feel very confident that we’re extremely close to that mark. When going through our grades on individual plays with members of NFL front offices, scouts, and former players we’ve gotten very strong feedback with very few grades questioned during our studies.
Why don’t you name the analysts who work on certain games?
Given the numbers of eyes that see each game, we don’t feel it’s necessary to name our analysts (though a simple look at our re-focused articles can be quite telling). Each game of analysis goes through at least three sets of eyes, just for the grading. Because of our rigorous training process and extensive review process for each game, we try to eliminate individual analyst bias as much as possible.
