This is something that I wrote back in 2015 when I was looking for a defensive KPI that was more robust in terms of measuring team performance compared to what currently existed. While data from 2015 is referenced in this article, the relationships still hold true.
In retrospect, if there's one thing I could change about what I wrote, I would adjust my tone towards the coaches identified in the article. Professional sports is a very tough and cutthroat environment where you are often held accountable for results that are influenced by factors you have no control over. In addition, it appears that both coaches did rely on the best information that was available to them at the time - however, they just were not aware that it was limited. Those individuals that are fortunate enough to coach in professional rugby are in roles that many of us can only dream of occupying. To be named a Head Coach of a Super Rugby franchise is a remarkable achievement and this should be acknowledged.
Below is the article in its original form.
-------------------------------------------------------------------------------------------------------------------------------The use of descriptive statistics to measure team performance is prevalent in many sports today. One of the risks of using specific key performance indicators (KPIs) as a tool to guide decision-making in sports is that if your metrics do not accurately identify and track the right factors that you may not achieve the desired on-field success.
For example, consider the post-match quote of Queensland Reds coach Richard Graham after a 32-5 defeat to the Brumbies:
"We missed a couple of tackles one-on-one, but if you look statistically we were still the best in the league with the first-up tackle percentage made the other night," Graham said.
"We missed 19 tackles, which is probably about average for the game, They were just a little more costly, unfortunately."
Tackle Completion Percentage (Tackle%) as well as Missed Tackles (MT) are two measures which have traditionally been used in rugby to measure defensive performance. When using these metrics the Reds were not able to identify any obvious defensive "red flags" yet they lost and conceded 32 points. Graham did allude to the fact that the missed tackles by the Reds were "more costly", but is this something that can be measured?
Missed Tackle Impact (MTI) is a performance metric that was developed on the premise that defensive success isn't necessarily tied to making tackles or minimizing the number of missed tackles, but rather, stronger defensive teams are better at minimizing scoring/impact from missed tackles. Thus if Quade Cooper misses a tackle on David Pocock which results in a try, we would assign him an adjusted points value of -7 points for that missed tackle. Furthermore, if Cooper misses another tackle in the game but it doesn't result in any scores for the Brumbies, his MTI would be -3.5 (-7 AP/2 MT). Thus MTI measures the average adjusted points conceded per missed tackle at both the team and player levels of analysis. (NOTE: For you rugby nerds who have been paying attention, the MTI is essentially the expected points for a missed tackle).
Consider the 2015 Super 15 season. John Kirwan's Auckland Blues were identified as being a team with many strong tacklers and they had the highest tackle completion percentage as well as lowest missed tackle count (Table 1) yet they finished 14th in the table and conceded the 2nd most points (-428 points) in the competition. Only the Reds were worse (-434 points conceded). However, when we look at the MTI for the Blues, we see that they conceded -1.797 points per missed tackle which ranked 14th in the competition. It's also interesting to note that the Reds were the only team that was worse with -1.963 points per missed tackle - see a recurring theme here?
TABLE 1: Tackle Completion Percentage, Missed Tackles, and Missed Tackle Impact and the corresponding correlation to winning for all teams in the 2015 Super 15 competition. These numbers were from the regular season and did not include the playoffs. Numbers were color coded along a stoplight continuum to indicate the top ranked teams (green) and the bottom ranked teams (red)
If we look at the correlation to winning for these 3 metrics we see that MTI is superior with r = 0.704 compared to Tackle% (r = 0.175) or MT (r = -0.037). With respect to correlation, a coefficient of 1.00 represents a perfect positive relationship, a coefficient of -1.00 represents a perfect negative relationship, and a coefficient value of 0 represents no relationship. A correlation coefficient between +/-0.7 and +/-1.0 is considered a strong relationship. To further validate MTI as a performance metric, we analyzed the last 3 full Super 15 seasons (2012-2013, 2013-2014, and 2014-2015) and assessed how all 3 metrics correlated with winning (Table 2). With more data points, we start to see that the correlation coefficients for Tackle% (r = 0.034) and MT (r = 0.080) approach 0, which suggests that the relationship to winning is pretty much random. In contrast, the correlation coefficient for MTI (r = 0.657) still remains strong and suggests a positive relationship with winning.
TABLE 2: Tackle Completion Percentage, Missed Tackles, and Missed Tackle Impact and the corresponding correlation to winning for all teams in the 2013-2015 Super 15 competitions. These numbers were from the regular season and did not include the playoffs.
It is important to note that while MTI is a stronger correlate to winning than traditional tackle metrics, it is most informative when used to supplement existing information. One of the limitations of Tackle%, MT, and MTI is that they do not identify the context of missed tackles. For example, did Richie McCaw miss a tackle because he was covering for a teammate who was out of position defensively or did he flat out fail to make the tackle? In addition, some players may be poor defenders but may not have a high MTI because the do not even attempt to make tackles and therefore cannot be credited with a missed tackle. We have already developed ways of teasing these situations out of the numbers and can certainly present this to you in future reports.
MTI is a tool that can be used to supplement existing information (traditional statistics, video, coach knowledge, etc.) to assess defensive performance. It is novel in that it quantifies something that coaches might have been aware of, but previously had no way of measuring. As mentioned before, it is important the performance KPIs actually measure what is important to winning, and given that Richard Graham and John Kirwan are no longer coaching in Super Rugby, it is possible that they could have changed their situations had they been using the correct KPIs to assess their team's performance?
Do you think MTI is a metric that could benefit your team?