YEAR-END REVIEW: PART 1
published Monday, December 1, 2008
by David Luciani

Up until a few years ago, we used to publish an annual year-end review of forecasts for the season just completed.  We stopped doing it for many reasons.  One of these was that despite our best efforts to emphasize the aggregate results over the individual best and worst forecasts, readers would constantly write us to remind us of a particular forecast that wasn't highlighted in the series, arguing that our choices were not representative of the set as a whole.  Undoubtedly, this is true of any selection and we never were attempting to focus on an official best and worst list.  Sometimes, we just like to highlight a handful of forecasts before getting to the most important aspect and that is the overall summary level performance of a forecast set.

We had discontinued this sort of annual review after the 2004 season but we have determined from our reader surveys the past couple of years that at least some would like to see the return of this annual review.  So long as readers know that there's no real science involved in how we choose to pick the best and worst forecasts and that the piece is, as it always was, for entertainment and fun, there's really no good reason not to reintroduce the piece.  In fact, it's a fitting one to kick us off as we prepare to not only re-launch our forecast publishing season for another year but also as we've recently unveiled a streamlined new look to the site.

We head into 2009 with a bit of a different approach than in the past.  In 2007, I had published a personal blog and prior to that, I used to publish a frequent "Ask Baseball Notebook" piece that was equally or even more popular.  For 2009, we're planning to bring both back in sort of a mixed format.  That is, I won't blog in the traditional sense of this era that expects the modern writer to check in daily with random thoughts.  Rather, I will try to condense my ideas and comments into more of an occasional essay format, even if it means shorter essays than I am used to writing.  Also, when good questions (that I am capable of answering anyway) pile up to our mailbag, it will be an appropriate time for an "Ask BN" essay with eight or ten of the best questions submitted.

Other than this multi-part year-end review, which will be spread out over the next few weeks, my primary focus right now is on finalizing the first 2009 projection set.  We will likely announce the official target date for our first set within the next ten days - we have almost always published by the middle of December and I see no reason this won't be true this year - and as we did last year, many of the newer features of recent years (such as player comments and improvements to the site tools) will be gradually introduced over the subsequent months as we head toward spring training.  In fact, the comments from last year, a first for us, do better afford us to recall our own thinking on certain forecasts as we look back to see what went right and wrong in 2009.

It may surprise readers but I actually always prefer to start these retrospectives with at least a handful of the worst projections, which is what I will do in this first piece in this series as I select a few of my least favorite.  It's not that I like focusing on bad forecasts but sometimes, it can be educational to look back at why a forecast missed and also, it's sometimes useful to see just how bad the forecasts that are most perceived to be bad ones actually were.  Some are so bad as to be self-evident misses of a catastrophic nature and others sometimes can show flaws in the forecasting methods, enough so that our desire to constantly improve the methods benefit from the close examination of such forecasts.

So, let's take a look at a few of my least favorite forecasts of 2008.  In part two of the series next week, we'll look at some of the best and then in the final part of the series, we'll consider the overall performance of the forecasts from a summary level.  All forecasts referenced in this set are those that were published in our official Opening Day set at the end of March 2008.

There is no player more mentioned in year-end comments, much as he was a year earlier as well, than Greg Maddux.  By no means do I consider Maddux among even the twenty or fifty worst forecasts of 2008 but it seems that a large chunk of readers do view his forecast that way.  I won't hesitate to concede that his forecast would fall in the miss column but the frequency with which his forecast was mentioned by those who weren't entirely satisfied doesn't seem to be justifiably proportional to how bad the result was.  Let's have a look at the forecast, the actual outcome and then consider what went wrong here:

GREG MADDUX

Unlike forecasts which miss because of an error in projected playing time, there can't be any complaints with that aspect as we were right on with the games started and within three innings overall.  I don't consider the strikeouts category to be a complete miss here but it was clearly a bit off.  The WHIP category was almost perfect and where the forecast comes apart some is in the ERA column and to a greater extent, in the wins column.

I'm not sure what the acceptable margin of error is in the ERA column before a forecast moves into the miss category.  Maddux's actual ERA was 0.64 higher than projected and I suspect that the tolerable margin relates to how good you project a pitcher to be.  For example, if we had projected him to be 0.64 worse than he was, with an ERA projection of 4.86, I strongly doubt the forecast would be remembered as a massive miss.  Also, if we had projected a 4.58 ERA and he had ended up with a 5.22 ERA, again, I don't think it would have been interpreted to be as far off.  In that respect, I've noticed that the forecasts that tend to get tagged as the worst are those where you end up projecting a player to be better than he ended up being and better to the degree of him becoming an important and high-ranked player in the ranking sheets.  In other words, if a player overperforms, you generally don't get much complaint.  That's completely understandable.

Another thing to consider when looking at the general perception that the Maddux forecast was a complete failure is that it reminds me that some categories are so much more important to a player's perceived performance than others.  For example, if we had been bang-on in the ERA column but way off in the WHIP column, to the exact degree of being off by whatever is necessary to overestimate his projected value to a fantasy team, I suspect we would still have heard less about Maddux.  WHIP is a far less-noticed category than ERA, likely correctly because in real life, preventing runs is the primary job of a pitcher.

In the spirit of not just focusing on misses but gaining something by revisiting them, let's consider what went wrong with this forecast.  We can start by looking back at a few sentences taken from our comments published along with the Maddux forecast before last season:

"Maddux actually pitched much better from 2005-2007 than his ERA revealed. In fact, Baseball Info Solutions publishes Bill James' so-called "ERC" or Component ERA which attempts to estimate what a pitcher's ERA should have been based on his primary numbers rather than earned runs allowed. For 2007, 2006 and 2006 respectively, Maddux's ERC was 3.54, 3.39 and 3.77, all significantly lower than his actual ERA...(other comments related to hits per ball in play rate followed)... In a neutral environment, we would still have been projecting an ERA around or above 4.00 at his age as even with the projection, his skills have clearly declined from his glory days."

Bill James' ERC method and our own effort to project how many earned runs a pitcher allowed yet again would have expected Maddux to have had a better ERA than he did in 2008.  Baseball Info Solutions tells us that for this past season, James would have expected Maddux to have an ERA of 3.55 and we would have put our estimate just a hair higher than that, both results based on formulae that considers now perfect hindsight knowledge of how many hits, home runs, walks and so on that he allowed.  We can only guess that unless Maddux is tremendously unlucky, something about both methods fails to account for an element that leads to greater run-scoring against him than we would expect.  Still, he was leaving the ball up much more than we had projected so even if his ERA had been closer to our forecast, he still would have performed differently than we were projecting.

By the way, this is a case where we're not going to blame the shortcomings of the forecast on the trade out of San Diego as Los Angeles is still a pretty good pitcher's environment too.  At the time of the trade, Maddux's ERA was just under 4.00 and then his performance went downhill as the year wound up.  Whether he got tired or showed his age or decline will have to remain a mystery.  The miss in the wins column is somewhat easy to explain, though.   Not only was his ERA worse than we had projected but he also averaged just 3.62 runs of support, the fourth lowest total in baseball of any pitcher with at least 180 innings.  We're content to call Maddux a miss but at least wanted to show the forecast and the final results side by side to keep it in context.

Here's one that was much more off target, not only because the player was hurt but because when he was healthy, he didn't pitch as well as we expected:

CHRIS YOUNG (SD)

This one's tricky because while Young did miss a lot of time, he really didn't look like the pitcher we were forecasting when he was healthy.  He allowed almost as many home runs as our entire full-healthy-season forecast was projecting and his control was off and his strikeout rate down.  There were so many pitchers of the Young variety, as there always are.  I'm talking about pitchers who pitch somewhat like the forecasted player but who fall significantly short of playing time forecasts enough to essentially destroy any possible redeeming value.  Beyond Young, the list could include players like Erik Bedard, John Smoltz and so on.

How about Dontrelle Willis to ruin our day:

DONTRELLE WILLIS

Well, at least we hadn't projected him to pitch that well and even with the complete miss here that would be costly in the wins column, our comments warned readers before the season started: "It is extremely rare for the weight of statistical evidence to be so strong that we argue that a pitcher's skills are declining at twenty-six but Willis seems to register on that scale."

Much worse than Willis, we'd argue, was our Opening Day Matt Morris projection:

MATT MORRIS

On the relief pitching side of the ledger, it's easy for me to find my least favorite forecast.  No doubt, there are worse forecasts in the set but it would be difficult to find a forecast as costly to a fantasy team's projected saves total than this one:

JOE BOROWSKI

There's really nothing to say about the above except that we got it wrong.  Looking back at our pre-season remarks about Borowski, we can also safely say that it wasn't injury alone that caused us to be completely wrong when we said "he's actually a bit better than he showed last year" and that the only useful thing a reader could have taken away from our remarks was the final sentence, which said "we're betting that he'll no longer be a closer by sometime in 2009."  This forecast can safely be dropped in the disaster column.

Costly to the same degree but not as much a fault of forecasting performance as health was Chad Cordero's projection:

CHAD CORDERO

Sometimes, there are players who spend most of the season being mentioned in emails by readers who categorize a forecast as a miss even before the year ends.  Despite his great power, Ryan Howard took all season to get his batting average up to even remotely respectable territory and even though from a value perspective, he may have even exceeded our expectations, we're content to still say that he's a player who stayed healthy and still performed differently than we had projected:

RYAN HOWARD

While we were decent in the power forecast, that's the easy part with a player like Howard and I'm still not satisfied with this forecast.  Despite playing much more than we had expected, his walks were down and his batting average was way off.  The forecast didn't end up as a complete crash and burn but it still deserves to be dropped in the disappointing category when it comes to accuracy.

There were countless players who had seasons shortened by injury where it's hard to read whether we would have been close had they stayed healthy.  Players like Cincinnati's Alex Gonzalez and Toronto's Aaron Hill fall into this category:

ALEX GONZALEZ

AARON HILL

Hill appeared to be well off the pace and we never saw Gonzalez so there's no way to know.  Of course, almost no player could come close to matching the extent of damage Andruw Jones caused to those who were counting on him.  I'm not sure if this next one was the worst forecast of 2008 but it's safe to say that it's in the top ten or twenty anyway:

ANDRUW JONES

While it would be easy to try to blame this one on missed playing time, it just wouldn't be true.  The player we were forecasting, the one we had seen as recently as 2007 even, was completely gone last year.  Amazingly, Jones doesn't even turn thirty-two until April.  It's tough to isolate just one aspect of the failure here to explain the miss.  Not only was Jones' power gone but he couldn't make contact at all (given the projected 512 at bats we had, he would have been on a pace to strike out 186 times).  Because of his age, it's hard to call this an early decline as thirty-two is rare to completely decline for a hitter - declines at an early age are more common for pitchers than hitters.  Weight gain, as we've discussed elsewhere in the pages of our site in the past, usually leads to an increase of power, even if contact rate happens to go down.  Usually, injuries can impact a player's power or speed or even contact rate but an "all of the above" nightmare like Jones suffered through is so unheard of that all we can say is that we got it wrong and based on what we saw this past year, we're not likely to be projecting a comeback in 2009.  For us, he's an embarrassing miss but for the Dodgers, he's now a $36 million gross miscalculation.

Travis Hafner's forecast hurt too:

TRAVIS HAFNER

It's not as clear that Hafner's skills had declined as much as it is for Jones but in any case, the outcome was about the same for 2008 anyway.  Chris Duncan joined the all-disappoint team as well:

CHRIS DUNCAN

Sometimes, speed (or lack thereof) kills.  How about Carl Crawford to let us down?

CARL CRAWFORD

In fact, the only thing we seemed to be on target with in this forecast was our projected drop in batting average, this for a player who had topped .300 three consecutive years and was coming off a .315 season.  We had published a lengthy explanation about his singles per ball in play rate but we'll still drop this in the "easy to call a miss" column as even a healthy Crawford clearly wasn't going to come close to our projected career-high in steals.

One last one I'll single out here, among my least favorites, is this one:

VICTOR MARTINEZ

Injuries alone can't account for this outcome and we'll have to wonder whether Martinez can bounce back for the long run.

I'm sure I've failed to highlight other forecasts that failed on a massive scale and the purpose of this exercise wasn't to necessarily highlight the ones at the extreme end of the error scale.  It always seems appropriate to me that when you're about to highlight some of your favorite forecasts and then get into a summary level analysis of how the forecast set performed as a whole, it's only appropriate to at least start by showing examples of just how badly things can go wrong, no matter how well your methods might work overall.  Next time out, we'll get into some of the better outcomes.