The Quality of A Forecast Set
by David Luciani
Published November 26, 2001

A reader wrote me an interesting note last week telling me that another site is advertising that its great track record shows that in a certain year, its projections landed within 1 home run 37% of the time, 2 home runs 52% of the time, etc.  The site then suggests that readers take a look "at a couple of different teams" from whatever projections, other than theirs, that they used the previous year,  to see if that other set is as good as this site's.  In other words, on one hand they advocate the use of summary level data at rating their projections but want anecdotal or casual evidence to be used when the reader reviews the data from their competition.  The problem with doing an exercise such as this is that it doesn't exactly convey the quality of the projections because it doesn't single out forecasts that are extraordinarily difficult to make.  Moreover, it is not difficult to make summary-level data appear to be good once you figure out what it is you want to say.

Though I will not openly engage another site in a web-style debate on what constitutes a quality projection,  the reader who wrote me felt it was obvious that this other site had incredible results, projecting the number of stolen bases a player will steal within 2 steals just over 61% of the time.  These are the sort of results that can easily be misleading.  In fact, a 61% rating in this category is actually worse than if you had just used the previous season's stolen base totals!  If you had used the number of stolen bases a player stole in 2000 to project his 2001 performance, you would be within 2 stolen bases on 393 of 636 players or 61.79% of the time.  This is even  assuming that you project 0 steals for any player who did not appear in the majors in 2000, including Ichiro Suzuki.  In other words, if you have a 61% success rating here, you would be better off at using the previous year's totals, regardless of whether the player even appeared in the previous season!

Now I don't want to single out a category but this same site also boasts that it was within 1 stolen base about 50% of the time, which might seem on the surface to be outstanding.  Well, considering that 59% of the players in baseball this past season stole between 0 and 2 bases, then if you were to blindly project 1 stolen base for each and every player, you would be even better than this site's apparently extraordinary results in that almost 60% of the players would be within one stolen base.

As much as I prefer not to be blunt, having 60% of your players land within 3 home runs is actually not an outstanding success rate either.  Again, that's the same exact average you would record if you had simply used 2000 statistics to project 2001 statistics (59% of 2001 players hit within +/- 3 home runs of the total they recorded in 2000).

Were one to undertake a lengthy evaluation, the field would benefit from a method by which we focus on the typical deviation between our forecasts and the ultimate outcome as compared to how well we would have done using alternate methods, such as previous year numbers, three year averages, weighted averages, etc.  For example, in our own final pre-season 2001 forecasts, if you had used our stolen base projections, the standard deviation between the forecast and the eventual outcome was +/- 5.8 stolen bases.  Had you used 2000 statistics instead, you would have experienced a standard deviation of +/- 6.8 stolen bases.  In other words, our forecast tightened the gap by a stolen base on each end when compared to using previous year's totals but I won't make any claim as to how much that helps compared to other sites as it wouldn't be fair.  As should be obvious, the more we narrow this deviation in our forecasts, the more useful they are than the other free and effortless methods already at our disposal.  At least if we state things this way, it offers some useful information about the forecast without making grand claims about being "better" than another site which doesn't even know it's the target of comparison.