Photo: Ian Johnson and Dustin Bradford/Icon Sportswire

Over the spring/summer, the SIS R&D staff is convening on the Off the Charts Podcast to talk about their top ten players at a position. To do this, we pit two methodologies against each other: 

• The “Scouts,” which comes down to the film-based opinions of Matt Manocherian and Bryce Rossler, each of whom has a lot of experience breaking down film and scouting players (Matt having been a scout for NFL teams).
• The “Stats,” which involves James Weaver and Alex Vigderman devising a ranking based on a suite of metrics, and having that ranking speak for itself.

Officially, Sports Info Solutions does not condone the dichotomy between scouting and statistical analysis. Each of them provides data in their own way and should inform our evaluation of a player. 

When we originally produced the Football Rookie Handbook before transitioning that content to our NFL Draft site, we put the scouting reports and stats side-by-side with the idea that the reader would bounce back and forth between them and leverage both to come to a conclusion about a prospect.

This week, we flipped sides, with Alex and James providing their scouting insights while Bryce and Matt assumed the mantle of the nerds. With quarterbacks, it was easy for everyone to agree on the No. 1 guy, but there were some differences of opinion along the way.

The Stats List Methodology

The stats-based ranking includes a three-year recency-weighted average of a player’s per-snap performance in Total Points, with their most recent season having a weight of 3, the second-most recent having a weight of 2, and their third-most recent having a weight of 1.

Total Points arguably evaluates quarterback performance better than any other position, if only because there is an abundance of data points related to quarterback play that factor into it.

Matt and Bryce initially thought to blend it with a second weighted EPA model that incorporated variables like Expected On-Target +/- (xOnTgt+/-) and throw depth, but concerns about historical sample with respect to xOnTgt+/- ultimately led them to decide that keeping it simple was best.

It should also be noted that this methodology was not limited to Passing Total Points; Rushing Total Points is equally factored in, and that is reflected in the placement of players like Josh Allen and Jalen Hurts, both of whom have been extremely productive on the ground.

What the Stats Showed

As Matt noted, it would have been difficult to tweak the model in a way to dethrone Patrick Mahomes.

“In terms of Z-scores, the difference between Patrick Mahomes and the second-ranked player, Justin Herbert, was bigger than the difference between Herbert and Derek Carr [the 10th-ranked player],” Matt said, “Mahomes was two standard deviations above average; the next-highest Z-score was Herbert, who was a little over one standard deviation above average.” 

After a clear tier break, Herbert and Allen rounded out the top three, being the two of the remaining three players on the list to have ranked in the top 10 in Passing Total Points/play each of the last two years. (Herbert ranked 6th and 1st in 2022 and 2021, respectively; with Allen ranking 10th and 8th.)

Allen’s rushing also proved to be a windfall for him, with Bryce pointing out that, in 2022, the Bills generated 11 EPA on running plays, with Allen himself accounting for 33 rushing EPA.

Dak Prescott also ranked in the Top 10 in Passing Points/snap each of the past two years, but missed most of 2020 with an injury. Another player whose 2020 hurt them a bit in the stats ranking was Joe Burrow, whose ill-fated rookie season dampened his placement, despite performing well each of the past two years. 

Following Burrow was Tua Tagovailoa, whose extremely efficient Year 3 with Mike McDaniel and Tyreek Hill was enough to propel him into the Top 10, despite not ranking in the Top 20 in Passing Total Points in either of his first two seasons in the league.

What the Scouts Thought

The scouts’ list likewise featured Patrick Mahomes at No. 1 and Josh Allen at No. 3, but Alex and James opted to place Burrow second rather than Herbert.

Their reasoning?

“Burrow just has that ‘It’ factor and has produced more results at this point than Herbert,” James said, “He went into Kansas City and beat Patrick Mahomes, which is something Herbert hasn’t done yet, and there’s something to be said for that.”

After that, there was a tier break with Jalen Hurts and the aforementioned Herbert placed at No. 4 and No. 5, respectively. The scouts expressed a bit of apprehension about Hurts simply because the Philadelphia offense was an extremely good fit for him last year, but still asserted that his leadership, his running ability, and his deep ball are reasons to be optimistic about him moving forward.

The scouts also included Trevor Lawrence (No. 6), Lamar Jackson (No. 8), and Russell Wilson (No. 10) in their list, despite all of them having bad statistical years at some point within the last three seasons. James was bullish on Lawrence’s prospects moving forward, citing his arm talent, his mobility, and his command of the offense in Year 2 as very positive signs. Jackson was given the benefit of the doubt with respect to his injuries and the supporting cast around him the last few seasons, and Wilson likewise got a pass for a dysfunctional first year in Denver, but the scouts did note that he’s likely past his prime.

Want to hear more discussion and debate? Check out this episode of the podcast: