Neil’s NFL Daily: May 3, 2013

Today's Daily ponders the knock-on effects of the Ravens retaining Bryant McKinnie, before discussing the intricacies of draft analysis and player evaluation.

| 4 years ago

Neil’s NFL Daily: May 3, 2013

Yesterday I concluded my draft round-up and waited for the NFL news to flood in. Naturally that didn’t manifest quite as I hoped, but at least Baltimore continued their excellent offseason with another positive move.

Additionally, yesterday’s NFL Daily drew an interesting response in the comments section regarding draft analysis. I felt this interesting topic warranted a longer answer, so for those interested in the subject and some discussion about player valuations, read on.


Friday, May 3rd

Bryant McKinnie Re-signs with the Ravens

It’s not that Bryant McKinnie is performing brilliantly at this stage of his career which makes this a good move for Baltimore — he’s not — it’s more the positive ripple effect across the rest of the line. He didn’t start a single game until the playoffs in 2012, and other than the wild card game against the Colts was broadly average in every regard from then on in. However, starting him at left tackle allowed Michel Oher to go back to right tackle which, in turn, gave them the facility to move Kelechi Osemele to left guard. Now, moving Oher didn’t immediately make him a better player but I’m convinced all the changing positions is having a detrimental effect. To get better at any task requires constant repetition — the facility to ingrain the physical into the subconscious and the two positions require different somatic responses. How can a player who is being asked to play two different positions be as good as if he was asked to concentrate on just one? If it was down to me (Ravens fans send a silent blessing to whichever god they worship that I’m not) I’d put him at RT and leave him there — here’s why.

Over the course of his career, starting in 2009, here is our grading for Oher broken down by position:


Left Tackle2540-9.2
Right Tackle236423.8


Draft Analysis

Yesterday I wrote, “I’m sure some of the draftees will work out, but Bill Belichick’s track record over the past five years has been average”, which elicited the following question in our comments section:

“Neil, on your criticism of Bill Belichick’s drafting, how do you respond to evidence-based analytics that validate both his strategy and performance?” Bill Marcellino

Firstly Bill, thanks for sending me the report, due to my incompetence at the highly technical task of “clicking on a link”, I only got it a few minutes ago so have only been able to scan read it so far. Hence, if I miss any salient points, please shout.

The most obvious difference is that the time frames are different — I specified five years and this report covers 12. However, it would be a little boring (if entirely reasonable) to simply assert there can therefore be no correlation.

I’d be interested to see the results for the past five years and understand if there is any connection between our view, summarised here, and theirs.

I suspect there will be some link (there usually is if the sample set is broad enough) but this statement, on how the data to value players was arrived at, will lead to many variances.

“We also employ two metrics for measuring the success of drafted players where the first assigns a value to each player’s performance for a season. The second was developed as part of this work and is based on a weighted score for games played, games started and recognition as a top player.”

Essentially, they used the Pro Football Reference stat “Approximate Value” (AV) and their own “Appearance Score” (AS).

Look, I’m not about to start picking at a perfectly valid concept like AV (which is based on a lot of work by someone I really admire) because it’s been used as a tool, in my opinion, beyond that for which it was intended. Despite a massive amount of caveats it’s being employed in this context to say Adam Snyder (AV=30) is a better draft pick than Andy Levitre (24), Louis Vasquez (25) or Jared Veldheer (18), although that’s never what it was intended for.

As for AS, it’s based on starts, played and Pro Bowl nominations. It’s clear that in this form of metric how many snaps a player actually played and how he REALLY performed are secondary. Last year Casey Hayward played in 16 games for 769 snaps, started only seven and didn’t go to Hawaii despite us grading him at +23.2. Jermaine Gresham started 15 games and went to the Pro Bowl even though he was our lowest rated tight end at -19.0 (as you know, the Bengals drafted TE, Tyler Eifert as a likely replacement last week). This would give Hayward an AS of 16+2×7+0=30 and Gresham 16+2×15+14=60.

In short, I don’t doubt the methodology employed on the data is valid, but if the data itself is not of sufficient quality how can the results be valuable? Evidence-based analytics is a great title suggesting wonderful results but the foundations need to be solid.


Other editions of Neil’s NFL Daily can be found HERE



Follow Neil on Twitter: @PFF_Neil


| PFF Founder

Neil founded PFF in 2006 and is currently responsible for the service to the company's 22 NFL team customers. He is constantly developing new insights into the game and player performance.

  • Benjamin McCloskey

    One other suspect conclusion from that paper is that UDFA’s are more valuable than late-round picks because they have more accumulated AV/AS. However I didn’t notice any mention of the sheer volume of UDFAs that come into the league, which is something like 3-4 times the number of total drafted picks, let alone late-rounders (for instance, the Patriots drafted 7 players, then signed 19 FAs this season.)

  • Bill Marcellino


    thanks for the detailed, well thought out response. Agree on your point that the time frames are different–different data pools. But even within the overlap period, I think were you and the study authors differ is in their conception of value.

    You guys are in the assessment business–you’re trying to empirically measure performance. You want to be able to quantify how Joe Blow performs, but also provide an empirical basis for comparing the performance of Joe Blow on Team NEP, and Tony Baloney on Team GBP.

    The study I pointed you to has two metrics–a performance metric (Approximate Value), and a roster based value metric (Appearance Score). Like PFF, PFR has a metric that compares relative performance, because activity is a valuable but limited metric. As Doug puts it: “Starting WRs who had lots of receiving yards are, as a group, better
    than starting WRs who did not have many receiving yards. Starters who
    made the pro bowl are, as a group, better than starters who didn’t, and
    so on. And non-starters aren’t worthless, so they get some points too.”

    But the skillful performance of an individual is not the same thing as return on capital investment. Roster building is distinct from the collection of talent (wait, haven’t I heard something like that before?). That;s something Doug explicitly addressed in justifying AV–appearances are a good proxy for value. As he put it: “That is, ‘number of seasons as a starter’ is a reasonable starting
    point if you’re trying to measure, say, how good a particular draft
    class is, or what kind of player you can expect to get with the #13 pick
    in the draft.”

    What Doug was getting at was something you know as well as I do–if you draft someone in the 1st round and they rarely show up as starter, never make the Pro Bowl, etc. they likely weren’t a good value. Conversely, a 7th round pick with a few starts is likely a great value.

    Which is why I found the main claim in that study so interesting–the 2d round produces more value than the first round, which helps explain why bad teams are always trying to trade up into the first round to get “that one player,” while good teams are open to drafting in the middle rounds, to build a strong roster that in turn wins games.

    I’m ok with your focus on performance here at PFF–I’m a subscriber :) But individual performance is distinct from roster building, and that’s what people like Ted Thompson, Kevin Colbert, Bill Belichick do–they try and put together the best roster and win games, not collect the most talent. If you understood the solid foundation behind evidence-based analytics, you’d be able to understand the results 😉