QB Grading – Our Normalization Process

| December 8, 2011

For those of you that have premium memberships, you may have seen some odd things going on with the quarterback rankings this morning as we worked through the early hours to implement some changes to how we present our quarterback grades.

The results are now up and our QB grades have finally been brought into line with the rest of the site, in so much as they have now been normalized. Not a word I personally use in my every day conversations with people but one that is pretty significant regarding what we do at PFF.

I’m going to tell you a little bit about why that is and what that means.

 

 

 

The Why

Back in 2008 when PFF was barely on the radar of most NFL fans, we had a decision to make: How to get grades to accurately reflect what a player did given how much they were on the field? We didn’t just want to reward the iron men who never came off the field, and needed to give those who’s playing time was limited their due. It led to us realizing we needed to collect player participation data, and from there, to us normalizing our analysis grades.

And so we did, except at quarterback. Some logistical issues meant it was something we shied away from, but after some soul searching, we’ve tackled that today and now our grades reflect how well a quarterback has played and take into account how many times he has dropped back to make a throw.

 

What it means

No longer are our grades such that it rewards those who throw all the time, but there are still some disclaimers to note, particularly when it comes to the ‘rushing’ grade a quarterback receives. Here you have players like Michael Vick and Aaron Rodgers with lower grades rushing than say Matt Ryan; obviously we’re not saying Ryan is a better rusher, but because of his limited attempts he has been more effective rushing when he has scrambled compared to Rodgers and Vick who may do more running but have a smaller percentage of plays that earn a positive grade. As ever with our grades, it’s not gospel and it requires some thought and perspective from the viewer to truly understand what they are saying.

 

The Gradings

If the proof is in the pudding, then it’s the rankings themselves. The top three, by a considerable distance, are Aaron Rodgers (+48.7), Drew Brees (+38.5) and Tom Brady (+36.9). Would anyone argue these guys haven’t been the Top 3 QBs in the league this year? Following them, you have six players with a passing grade of between +17.4 (Matt Ryan) and +10.6 (Matt Moore and Matt Schaub) in a sign of the gap between the upper and middle classes of quarterback play.

One player really impacted by this change is Cam Newton (+8.0) who is now solely reliant on his rushing grade (+13.4) for holding onto a spot in our Top 10. To some, his -4.3 passing grade may be blasphemy given the highlight reel plays he’s made and positive impact he has had on the Panthers’ franchise, but it indicates for all his success there have been a lot of inaccurate throws and bad decisions along the way as well. We can’t quantify his impact but we can grade his throwing.

Down at the other end, things get even worse for Blaine Gabbert with his rating dropping to a -37.1 to fully confirm his first year in the league as being as bad as anything we’ve ever seen. At least he has the excuse of being an ill-prepared rookie; what about Curtis Painter (-22.9) or John Skelton (-19.4)? Considering Skelton has only dropped back 117 times, it’s all the more remarkable he’s achieved such a horrible mark. Those are the names you’d expect to see, but spare a thought for the failings of recent high draft picks and especially the class of 2009. Matthew Stafford (-7.7), Josh Freeman (-10.6) and Mark Sanchez (-13.7) are all in the Bottom 10, though they do at least have Joe Flacco (-14.6) for company.

 

So …

As with all of our gradings, I can’t emphasize enough that they’re not gospel. It requires some understanding of how players are used and doesn’t take into account strength of opponent (though one day it may). We’re happy to bring this more advanced form to our subscribers and will continue to develop what we offer in order to give you what you want.

 

 

  • Steve Palazzolo

    Normalization! Exciting day!

  • phillyphan321

    I liked the old ratings better.

  • 52decletzu

    Strength of opponent is a huge factor which must be considered in some way.

    • Nathan Jahnke

      There isn’t really a good way to do strength of opponent, because opponents strengths are always changing. That is one of the reasons why the rating isn’t everything, and context like opponent need to be taken into account when you are interpreting the grades.

  • sunnym

    Before, a +5.0 grade or higher was considered a very good game. What about now?

    • sunnym

      and also, what’s now a bad game? Below 0.0?

    • Nathan Jahnke

      Just like every position now, a rating of 0 is average, anything above a 0 is an above average game, and everything below 0 is a below average game. It completely depends on what you want to say a very good game is. If you think there are roughly five very good games a week, then anything above +3.0 is very good.

  • motorcycle

    I don’t think strength of opponent should ever be taken into account. PFF rated Miami as the best overall defence in 2010, so then if anyone beat Miami early on in the 2011 season it meant more? Or because Miami’s defence were playing badly in the beginning of 2011 season, then overall for the season as a whole they are still a bad defence now?

    About your QB grades, for me it’s impossible to grade a QB running with the ball. If a QB runs for a first down but gets injured is that a positive or negative graded play? What if everyone is covered so he runs downfield but gets clobbered by a defender, is that down to bad blocking (which the playcall may not have accounted for) or down to bad judgement to run by the QB when he should have thrown the ball away. Do you take into account if wide receivers are covered downfield, because the camera angle does not always show the downfield receivers and whether or not they are covered. To grade a QB based on incomplete information does not seem fair. For example if a receiver is open but the QB runs, then the QB should get a negative play. But if you can’t see that receiver on the camera angle, then you can’t grade the QB’s decision to run.

    Does the normalisation process just take into account positively and negatively graded plays as a percentage or something else as well? If you could post how the added total of graded plays changes to the grade after the normalisation then that would help me understand it. Thanks.

  • motorcycle

    Also maybe QB running plays should be normalised differently, because if Rodgers is ranked 35th in QB rushes, something must be wrong!

  • skerwood

    when it comes to the ‘rushing’ grade a quarterback receives. Here you have players like Michael Vick and Aaron Rodgers with lower grades rushing than say Matt Ryan; obviously we’re not saying Ryan is a better rusher, but because of his limited attempts he has been more effective rushing when he has scrambled compared to Rodgers and Vick

    does that mean that scrambling is considered rushing or is this strictly running the ball not scrambling??

  • Ben24626

    I don’t know that un-normalized grades aren’t better, for all positions. It really isn’t that hard to take into account the amount of work a guy has had to get his grade, and something just seems wrong with these grades, very wrong.

    • Ben24626

      *It really isn’t that hard to take into account the amount of work a guy has had to get his grade when reading and interpreting the grades

  • visible

    First of all, any rating that has Matt Ryan as the #4 QB in the league has no merit. Khaled either had to adjust the formula to increase Matt Ryan’s grade or crawl in a hole for listing Matt Ryan as the #13 best player in the NFL. Anyone who understands football knows there is way more to grading a QB than you guys include. While you may not be able to figure out how to incorporate the strength of defense (which would expose Matt Ryan BTW) you should at lease incorporate scheme. If you look at Matt Stafford and Detroit’s non existent running game, you would see the coverages being thrown at Detroit. However, when evaluating Matt Ryan you will see the scheme is to stop the running game and teams actually challenge the Falcons to beat them passing.

    Another thing I find pretty amusing is your QB pressures. When evaluating Tom Brady, he puts his team in better plays at the line of scrimmage, which ultimately avoids pressure. Matt Ryan obviously struggles reading defenses and cannot make the adjustments that Brady does. So how does PFF handle this situation. Here’s how… “Oh, poor Matt Ryan has so many pressures it’s not fair”. Then PFF will actually somehow in some twisted world give him credit over Brady.

    Brady also has the ability to step up and maneuver in the pocket to avoid pressure. While at the same time it is absolutely painful to try to watch Matt Ryan maneuver in the pocket. Instead Ryan would rather throw a rocket over his receiver’s head. But all PFF will remind us of are the drops. I mean who cares about the erratic throws when we can focus on some drops. I feel so bad for Ryan because he is so handcuffed with White, Jones, Gonzalez, Turner, etc. I mean poor Matt Ryan, how can he work with these guys? (yes this is how ridiculous PFF sounds when trying to defend him).

    I could go on and on, but what’s the point. I don’t want to convince the PFF team to take down their Matt Ryan fatheads. I mean if you have Matt Stafford in the bottom 10 and Matt Ryan in the top 5, and yet Stafford is beating him in every single category against tougher coverages and playing 4 games with a broken finger, then you guys have serious issues.

  • http://www.profootballfocus.com Khaled Elsayed

    I never said Ryan was the 13th best player in the NFL – I had him ranked as my 13th best player for 2010. 2010 has no bearing on 2011. Big difference there.

    Clearly he hasn’t played as well as last year, hence the drop off on his grade from this year to this time last year – his ranking is a reflection on the general averageness of quarterbacks outside the top three – hence how close four through to ten is with no-one standing out. I don’t think anyone has defended him to the degree you make out, though to ignore the drops or struggles of the offensive line seems a bit of you tailoring facts to suit your argument – much the same way it would be folly not to look at the woes of the Rams/ Chargers offensive line as contributing to production issues for their QBs.

    Given that Brady has a grade, what almost twice the size of Ryan I don’t see where the beef comes from in that regards. As for Stafford, I think if you watch every throw he’s made compared to every throw other quarterbacks have made this year then you’ll see why stats can be very misleading for quarterbacks. That’s what our grading says but it’s what the eyeball test should tell you as well. Maybe that’s down to a broken finger but we admit to not accounting for injury because you can’t quantify the impact of an injury on a player (only speculate).

    You think we have an irrational love of Matt Ryan, that’s your opinion. We don’t – he graded poorly in 2009 and has a number of bad games this year while struggling to find the form that made him such a clutch player last year. It’s reflected in our grade, and more importantly reflected in Atlantas’ record. With a defense that has got better since last year, imo, Ryan has to shoulder a portion of the blame (along with the line) in terms of why the Falcons have taken a step back.

    We make no bones that we evaluate them as a passer – the throws and decisions they make after the snap. We don’t know what QBs check in and out of and from so how can anyone possibly grade that without knowing what play was called in the huddle?

  • skerwood

    I like this version

  • speirs

    Being a Lions’ fan and watching Stafford this year, I would have to agree with his rating. His numbers look good in the end, but he misses a lot of throws but Calvin also makes up for a lot of those bad throws with great catches.

    Just curious, in the game last week against the Saints, Stafford threw a long pass to Johnson who was wide-open, but he severely under-threw it. Calvin caught the ball, but was tackled right away I believe, does that go down as a positive play for Stafford even though it should have been a touchdown?

    • http://www.profootballfocus.com Ben Stockwell

      speirs,

      Have you got a quarter & time for the play you’re referring to? Having trouble locating the play you’re referring to from the description of it.

      Thanks

  • visible

    Khaled there is definitely an irrational love for Matt Ryan. Every time you come out with a new statistic or formula, you use Matt Ryan as the example, in an attempt to cover up how bad he has played. “Oh poor Matt Ryan” is your quote!

    You’re right, you did not have him #13…you had him as the #7 overall player in the league for 2011 predictions. “When I had him up at number seven in my top 101 players of 2011, I expected some criticism and it came”…your quote again.

    Ryan is a solid QB, but let’s be honest, he is nothing more. As Atlanta’s running game gets worse, Ryan gets exposed more and more. ATL thought drafting a WR would be the answer, but really Ryan’s best friend is the running game, which is the case with most QB’s. Only true franchise QB’s can put a team on their shoulders and take them to victory. Ryan is clearly not a franchise QB, and therefore until ATL has a superior running game again, we will see more of the same from him.

    Regarding Stafford vs Ryan…I admit Stafford is inconsistent and his recent injury has hurt him. But let’s look at Stafford and Ryan’s ratings against common opponents this year…

    Tampa – Stafford 118 / Ryan 75
    Chicago – Stafford 107 / Ryan 76
    GB – Stafford 66.5 (broken finger) / Ryan 55.1
    Carolina – Stafford 121.9 (broken finger) / Ryan 101.1
    New Orleans – Stafford 97.5 / 81.5

    I know you love the pressure and drop argument. But Stafford against Minnesota yielded one of the most high pressure games of the year and Stafford still had a 108 rating. I know the QB rating is far from telling the whole picture….But it if your formula has Ryan as #4 and Stafford in the bottom 10, yet Stafford has outperformed him in every category AND outperformed him 100% of the time against common opponents, while having a worse running game – then even you have to realize there are major flaws to the grading system.

    • drgarnett

      One can argue about Stafford vs. Ryan. But any rating system that has Matt Moore ranked ahead of Stafford *definitely* has some serious flaws.

      • sgtrobo

        agreed. Sometimes you have to take a step back and take a look at the whole body of work and decide if the grades are matching reality. Certainly AR, Brees and Brady are a full echelon above the rest, but sometimes reality just doesn’t match the grading. Matt Moore is rated higher than Tony Romo? I don’t even like Dallas.Aaron Rodgers as one of the worst rated running QBs? C’mon guys.
        I mentioned this elsewhere, didn’t see a response, but the pressure rating under the Signature Stats is borked. You’re giving credit to QBs who throw the ball away when under pressure and it is greatly skewing the stats there.

  • kyleplum

    I would really like to know why Aaron Rodgers is the 34th ranked rushing QB. He always graded highly in running beforehand, and he is definitely one of the best at it. Of all the pocket passer he is the best at running (only when he needs to). Every once in a while his running makes a big difference, and he is so good at picking up first downs. I think misleading nature of the rating speak for itself. Sure, Tebow or Newton can grade out higher running, because that is a big part of their game, but running is only a small part of Rodger’s game and he does it so well.

  • kyleplum

    How does Rodgers have a -.7 for rushing against the Denver Broncos when he had two rushing touchdowns, and they were not just sneaks. Each of his rush TDs took athleticism that i do not think any other pocket passer in the NFL has, but certainly not 33 QBs with his athleticism.