Data Snow Blindness

I am taking a break from everything else that’s going on today to tell the story of an error in data analysis and presentation, seriously affecting the strength of the argument being made, that to me is jaw-dropping – not the error itself, but people’s reaction to its being pointed out.  And that reaction is:

Nothing.

Seriously.  Nothing.  Nothing by commenters on either blog where the original analysis and reactions is prominently mentioned (Weisenthal and Krugman).  Nothing by the bloggers themselves. No change in the analysis as presented on the blog.  Dead silence. Debate continues to be waged based on the uncorrected blog data.

I have racked my brain as to why this might be.  The best reason I can think of is that most people are so focused on the “bright” parts of the data as presented, the fact that the two measures over time involved show a clear relationship, that they ignore the fact that one of the two measures is not quite the correct one to use.  It is, if you will, an example of blindness like the blindness caused by trying to look ahead into a landscape of snow reflecting the sun fiercely – the well-known phenomenon of “snow blindness.”  Data snow blindness.  And, as in the case the other snow blindness, you become truly blind – you simply don’t notice the information indicating that the analysis is off.

The Story

The story begins when Weisenthal reacts to an ongoing debate on the merits of investment in gold by doing a couple of graphs comparing the value of the S&P 500 index over time (1979 and 2007 to now) to the value of gold in the markets ($/troy ounce). His point – a valid one – is that even with recent fluctuations in gold’s price, investment in stocks outperforms investment in gold over time.

Now, as I found out in doing my post-MIT-Sloan-School-of-Management research on investment theory for my own use, it turns out that the S&P 500 is a price-only measure.  That is, the typical value of the S&P 500 that everyone quotes does not include the value of dividends that the companies issue over time, and it doesn’t include the reinvestment of those dividends.  As far as I can tell from observations over the past 15 years as well as from previous data, the dividends themselves average about 2.3% per year, and the reinvestment adds another 0.1% per year (the “geometric mean” of returns, the correct way to measure, suggests that at least until about 2007 (a period of maybe 90 years), the growth in the S&P 500 was about 10.8% per year, which meant that reinvestment of dividends over the course of a year would yield about 2.3% times 10.8% times ½ [to reflect the fact that dividends don’t all occur at the start of the year], or about 0.12%).

If you don’t believe my assessment, go look at the S&P web page where they report S&P 500 values and returns over time.  Above the regular report is a measure of “total return.”  If you look at the description of that return, you find that “total return” does indeed compute return with dividends included (it appears that dividend reinvestment is also included, but I’m not sure about that).  They do it over a much longer period than a year, so at this point the TR index is almost twice the regular S&P 500 index.

If you add in this adjustment, the result of the analysis changes significantly.  In Weisenthal’s original graph (made so that 2013 represents 1), the ratio of S&P 500 to gold price goes from about 0.1 in 1979 to 1 in 2013. Accepting the same starting point, my computed ratio goes from about 0.1 in 1979 to 2.12 in 2013.  A sharp drop in the ratio (reflecting dubious “flight to safety”) from 2007 to 2009 becomes a drop from about 6 to 2, not a drop from 5 to 0.7. It hasn’t been flat or dropping since then; it’s been climbing by about 6%.  Stocks don’t just beat gold over long periods of time – they beat gold over the short and medium term pretty consistently, and over the long term by a huge amount – try 21.2 times as much.

So I posted this in a comment in Weisenthal’s blog and, getting no response, in Krugman’s blog.  As noted, no one took notice (I would have posted my comments in caps, but it’s not netiquette to scream).  In fact, the debate in the comments proceeded as if the original Weisenthal graphs were the issue – is the government understating inflation data (from “gold bugs”) and therefore there is another gold price surge to come once that becomes clear?  Is the advantage of stocks over gold clear enough, or would an even further surge erase it?

Data snow blindness.

Implications For Y’All

At this point, I have to distinguish this from other sources of problematic analyses that have happened recently – e.g., the Reinhart-Rogoff controversy in economics (which apparently revolved partly around a coding error) and the London-trader miscomputation of risk (partially a human Excel miscalculation).  Those are not really a problem of not noticing that one of your sets of data points is not capturing well what you want it to capture – and they have been exhaustively investigated and debated.

For another example of data snow blindness, I’d like to go back to investments again – the idea of 401(k)s (also applicable to IRAs).  What no one seems to be pointing out is that the expense ratios in those 401(k)s are quite high – I believe they’re still above 2%.  If your employer isn’t paying into your 401(k), this means that you must balance the gain 20 years from now in lower taxes when you cash them out with the loss you get from not sticking an equivalent amount in a Vanguard S&P 500 index fund with an expense ratio of 0.1%.  Even if you pay zero taxes when you cash out, somewhere over a 10-20-year dividing line, you may well lose money on your IRA/401(k) compared to the alternative.  And that’s true of 401(k) bond investments as well (vs. bond index funds). Or so the data suggests – but no one seems to notice this enough to discuss it.

Here’s a few more:  stock risk vs. bonds and everything else. If you have your money in a US S&P 500 index fund, what is your risk?  Over any 20-year period, the stock market as a whole has outperformed any other investment – including inflation.  But what if the stock market collapses drastically and stays collapsed?  If you think about it, that would mean that the US government has collapsed, since it’s the government that insures the banks underpinning economic investment by various mechanisms. So the risk of collapse of the stock market’s 500 largest members is pretty much the same as the risk of the US collapsing – in which case, without that government backing, your money is likely to be worthless (and your gold coins). So why are you “diversifying” beyond the stock market, again?  If you’re planning to start drawing it down or keeping it level within the next 20 years, then some amount of, say, a bond index fund or inflation-protected securities (TIPS) that will keep up with inflation is fine; but the reason for doing anything beyond that is not as clear as it might seem.  Data snow blindness.

How about stock investment returns? Today mutual fund companies compare their results to the S&P 500 – is that the regular S&P 500 or the total return one?  Do they include their expense ratios – above 2% until recently, now (afaik) around 1.5% -- and do you compare them to the Vanguard 0.1% and the Fidelity Spartan 0.2% (plus withholding a bit in cash, which right now earns effectively zero)?  There’s a reason why those index funds outperform around 70% of all other stock investments over a 10-year period, and probably close to 90% over a 30-year-period.

In other words, the real implication of data snow blindness is that it is probably hitting you right in the wallet right now – not necessarily yours, since everyone else seems to be doing it too. Or almost everyone else … Gee, I wonder why the Vanguard S&P 500 index fund is one of the two most popular stock investments today?

Anyway, please think about it.  Me, I’m going to go off and check myself for further signs of data snow blindness. 

How To Have Fun Following the Stock Market

Over the course of 40 years, I have been following the stock market. Obviously, I have a bit of a personal interest, but it’s also fun to make up my own mind about what it all means. And, of course, it’s a bit like rooting for a team – with the added fun that, over the long term, the stock market has gone up by about 10% per year, so you tend to win at the end of the day a bit more frequently than you lose – definitely not the case with some of my sports teams.

So let’s talk about how I might do it (bearing in mind, of course, that these are brief glimpses via a TV stock market ticker in places like restaurants or on the radio while travelling on occasional days, plus checking the results at night as part of looking for good news to go to sleep with). The first thing I do is, concentrate on the S&P 500. The Dow Jones, I have found, isn’t always representative of the overall stock market, because its selection of stocks is a bit quirky; and, of course, the NASDAQ is too heavily high-tech-weighted. The S&P contains stocks from both, and so, if the NASDAQ goes one way and the Dow goes another, as happens surprisingly often, you can pretty much count on the S&P being somewhere in the middle.

The next thing I do is remind myself that the S&P 500 does not include income from dividends. According to studies I have seen, those dividends add about 2.15% to the S&P’s return each year. So, if the S&P is down a little on a given day, I can comfort myself with the reflection that so far in the year, it has actually returned, say, 1% more than the number suggests. If you want a further comforter, S&P publishes “total return” (S&P 500 plus dividends for the year so far). And that doesn’t include reinvestment of dividends, which, if the market has gone up by 15% so far, will yield you another 0.15% by the end of the year (p.s., due to the magic of compounding, that will be nothing to sneeze at in 5 years or so). See, I’m winning already!

If I happen to be able to look at the stock market before the opening bell, the only thing I look for is “S&P futures”. That happens to be a fairly reliable indicator of where the stock market will hover around until about 2-2:30 pm … more on that later. Be careful of the name, because there’s another S&P futures market whose abbreviation is very similar, but which tends to be negative when S&P futures is positive, and vice versa.

And then there’s one more thing I can do, and pretty much do every day. That is, look to see if it’s sunny out. You see, I live near Boston, reasonably near NYC – near enough to guess what the weather is like around 8 or 8:30 in the morning, when the traders go into NYSE to prepare for the day’s session. A fascinating study a while back showed that when those traders see sunny weather outside, they tend to be more optimistic in their outlook, and so the S&P 500 tends to go up more often. And so, when it’s sunny out at that time, no matter whether it’s pouring the rest of the day, I am more likely to anticipate a good day. It’s sunny this morning! I’m going to win, yay!

Unfortunately, I find that the news rarely gives a good indication on what will move the market today, no matter when in the day you check. After many years of listening to them, I tend to distrust ex post explanations. So I just check, if I’m lucky enough to see, what is happening around 10:30-11, when the spurt of the S&P 500 to its futures level has been pretty much over for a good half hour. On the rare occasions when the futures projection is countered by sudden events, that’s when the big drops or big gains will show up. Yay! The Fed is going to give guidance today, and traders usually overestimate the effect on the positive side before they announce around 2!

If by some amazing chance I get a chance around lunch to watch the S&P 500 update on an every-half-minute basis, I can watch the dance of the buyers and sellers. The way it plays out is, these days, a surge up of less than a point as buyers buy the prices up, followed by a drop of less than a point as sellers sell the price down. It’s only after 2-3 minutes that you can see whether the overall trend is up or down – unless a thrilling jump up suddenly moves the market up by a point or more. Or an annoying big drop down interrupts the festivities. To heck with the business news – where’s the sports?

But the real surprises tend to occur around 2-2:30, when suddenly the market can go completely away from the futures prediction. It almost always is due to breaking news – the Fed, late-in-the-day government reports, political news, late news from Europe. If no break occurs then, the futures prediction is probably good for the rest of the day. Yay if it was sunny this morning!

And then the final bell rings at 4, and about 4:04 to 4:06 you get the final S&P 500 figure of the day. But, oddly enough, if the S&P is up more than 1% (these days, about 13 points), I don’t get too happy, and if it’s down more than 1%, I don’t get too sad. Because if traders really understood what was happening, there’s no way there would be jumps like that. I’m a cynical guy; as far as I’m concerned, traders can be pretty unsophisticated in their fears and enthusiasms, so what this tells me is that traders are running around like chickens because it’s going to take them forever to see that the latest news, ultimately, is no big deal. And since that kind of thing can last for months, I just sit and wait until sanity arrives, with my own idea of where the S&P 500 should be. On the other hand, if it reaches the point where Intel is at 60 times earnings (as it was in 2000), or 5 times, that’s real craziness.

Next, I factor in yearly cycles. The one I like right now (or maybe “like” is too strong) is the end-of-the-month cycle in which, I suspect, traders start trading the price back towards the value at the beginning of the month, to show gains in their monthly performance report if there are any, or because they have oversold through panic and now are just waiting until the price comes back up through lack of sellers in order to cut their losses. That tends to mean that I pay less attention starting around the 21st. Ho-hum.

The next thing I like to anticipate is the quarterly period from about Month 1 Day 15 to about early in Month 2 when companies make their earnings reports. Those companies, I really suspect, manage their numbers to analysts’ estimates. And that means, most times, many more times where the companies beat the numbers by a little, and the stock market keeps bouncing up, bit by bit. Or, there’s negative news, but the earnings reports match it until the panic goes away (or outlast the reports, in a few cases). Fun times. Less than 1% S&P 500 daily increases. Yay!

And then there is the full yearly cycle in which, many times, the “Christmas rally” tends to take place right after New Years’, followed by continued upward movement to the later part of April, followed by slow and at the end more rapid decline to mid-October, followed by comeback until the end of the year. This varies a lot, but it typically means that about 2/3 of the year is an up-tick. Yay! And if it isn’t, one of my sports teams won again. Yay!

But the best part of it all is those times during the year when the S&P 500 is really on a hot streak, or at the end of the year. If the S&P 500 is on a tear, I can sit down and fantasize what my teeny S&P 500 index fund has grown to. If I’m at the end of a down year, I can sit down and remind myself that the underlying S&P 500 is growing by 10% per year, blithely ignoring inflation and eventual capital gains taxes (don’t rain on my parade!) And at the end of the year I can raise a toast, to my teeny S&P 500 index fund that over the last 25 years of 10% average returns has grown 10-fold, to a slightly less teeny amount, and fantasize that if life were just that index fund, I would be rich, rich beyond the dreams of avarice, and that my stock-market team had therefore won the Super Bowl.

Happy New Year! Yay!