If a rational investor believes that a passive strategy like indexing will usually outperform an active strategy like stock-picking, then why is it so difficult to stick to that passive strategy?

Based on my own experiences as an investor and having read countless stories by others, I’d say that hanging on to a passive strategy is difficult for almost all investors, regardless if you’re investing as a professional or as an amateur. 

The tendency investors have to abandon their passive strategies in favour of active strategies is what I call the Rational Investor’s Fallacy.

If you want to read about a professional money manager who seems to be immune to the fallacy, check out the funny article in WSJ called "What Does Nevada’s $35 Billion Fund Manager Do All Day? Nothing”. The guy in charge of the Nevada Public Employees’ Retirement System is truly running a bare-bones operation, trying to keep costs low and not trying to beat markets. He makes only a few changes to the portfolio a year and sits on his hands during volatile market days.

Most investors are however not like the money manager from Nevada – even though they should be, at least according to Warren Buffett, who is an avid proponent of indexing and passive strategies. Buffett has noted that especially his wealthiest friends are the ones who don’t want to hear through. They basically can't stand the idea that their big money won't buy special performance and thus they end up going to hedge funds instead of buying low-cost index funds.

However, as Buffett recently said: "Professionals, after fees, don’t know how to get a better result. If you take half the people in the country and they don’t do anything, they just own the average, they are going to get average result. The other [half] are going to incur all kinds of fees and they are going to do way worse than the people who do nothing. The difference is incredible.”

He’s basically repeating what Nobel prize-winning economist William Sharpe said in the 1991 paper "The Arithmetic of Active Management”: The average active investor, after costs, must underperform the average passive investor.

One of the reasons investors fail to follow their strategies could be that they (surprise-surprise) aren’t always rational. The rationality of investors has been questioned often, especially within the field of behavioural economics. Amos Tversky famously demonstrated the tendency of investors to make risk-averse choices in gains, and risk-seeking choices in losses – as well as other tendencies that violate economic rationality as usually understood. 

Occasionally rational investors might also get struck by the Gambler’s Fallacy, that is the mistaken belief that if something happens more frequently than normal during some period, it will happen less frequently in the future, and vice versa. One can easily see how certain frequent events in the stock market might lure a passive investor to fall for the Gambler’s fallacy (for example the stock market hitting all-time highs several weeks in a row).

Investors might also, from time to time, be affected by something that another Nobel laureate Daniel Kahneman calls “illusion of skill”. Kahneman, who is definitely not a fan of stock-picking, has several times told investors to "stop trying to guess the future”. 

Yet sometimes we think that we can do exactly that – predict the future better than the rest. It reminds me of illusory superiority, a cognitive bias whereby individuals overestimate their own qualities and abilities, relative to others. Similar biases occur easily when people are asked to compare their driving skills to other people.

Finally, a reason why passive investors might go active is the fact that passive strategies are extremely boring. As the name implies, you're not supposed to do much else than to observe and stay calm, even when the bears are raging in the market. That can be surprisingly hard.  

The only cure against the Rational Investor’s Fallacy that I know of is to make enough costly mistakes. In other words, when you’ve lost enough money due to your missteps, you’ll slowly but surely realize that the best you can do is to simply emulate the humble money manager from Nevada.