When Is It Time to Declare Victory?

I was recently re-reading my favourite biography, David Nasaw’s The Patriarch, a massive book detailing the life and times of Joe Kennedy.

Kennedy is a study in contrasts. He was a staunch family man, telling his kids to look after their siblings first. He was also happily married for more than 50 years, and lavishly spent the family fortune to help get his son elected — first to the senate, then to the presidency. Yet he had dozens of mistresses on the side, which was an open secret for decades. Apparently the only person who didn’t know was his wife.

He also made a lot of his fortune in the opaque world of stock manipulations, using insider knowledge to gain a massive advantage over other market participants. Then, as head of the very first Securities and Exchange Commission in the mid-1930s, he created legislation that put an end to the practice.

Kennedy worked extremely hard in the 1910s and 1920s. At one point he was head of three different film companies at once. He also actively traded stocks on the side and had various other businesses with friends — like one that focused on real estate. He made his fortune in the 1920s, got out before the market crashed, and had enough to not only live in comfort for the rest of his life, but to also set up trusts so his children could do the same.

Despite so much success speculating in stocks, Kennedy changed course completely in the early 1930s. He stopped all stock speculation and put his cash into boring assets — things like quality blue chips, highly rated municipal, federal, and corporate bonds, and well-located residential real estate for his family. He bought things like New York City office buildings, Chicago’s Merchandise Mart (the world’s largest privately owned building at the time), and the Hialeah Park Race Track in Miami.

Although Kennedy was obviously at least partially motivated by the depression, I still think it’s fascinating how he completely changed his strategy. He went from maximizing wealth to maintaining it, and he was excellent at both. Kennedy is famous for potentially stealing the 1960 presidential election for his son and for bootlegging — even though there’s no evidence of the latter. He didn’t get into the liquor import business until after prohibition was over.

What he should really be remembered for is being an excellent investor. There were few better.

Inspired by Kennedy’s abrupt change in strategy, I thought today I’d explore the same topic. At what point should someone join him, declare victory, and take their foot of the proverbial gas?

Risk means very different things to different investors. Much of risk is your own individual measure of it.

I know folks who do option strategies or take flyers on what I view are risky stocks, all without missing a beat. Individual investments might be money losers, but the overall portfolio should still go up. That logic doesn’t work very well in bear markets, but we’ve had about 12 months of bear markets in the last 15 years. Nobody’s worried about no stinkin’ bear markets. There’s money to be made!

On the other side is someone we have all met. These are the folks who are scared of their shadow, ones who insist on a healthy bond or precious metals component in their portfolios. My grandparents were like this, although they had a pretty good excuse for it. They grew up in the depression.

The first step in all this is figuring just how much risk you’re comfortable with as you move your portfolio to safer assets. Safe means different things to different people.

I should also point out that various portfolios with similar asset weightings can offer very different risk profiles. Say two investors have a 100% equities portfolio. One buys the kind of boring companies we like around here, like Canada’s grocers, utilities, and life insurers. The other takes concentrated positions in the latest fad stocks with huge volatility and zero earnings. They are two very different portfolios.

Other asset classes are similar. One investor might buy a collection of government and high-grade corporate bonds for the fixed income part of their portfolio. The other buys junk bond ETFs with leverage attached. One buys quality real estate with low amounts of debt. The other maxes out cap rates by looking for the worst house in the worst neighbourhood, and takes on an much debt as possible to do it. Same asset classes, very different risk profiles.

If it’s a safe asset class and it’s offering a big ol’ yield, that’s almost always a giant red flag.

One thing I use to measure my portfolio’s risk profile is beta. Beta measures how volatile a portfolio is compared to the market as a whole. The market’s beta is 1. A portfolio with half the volatility of the market has a beta of 0.5. A riskier one with 150% the volatility of the overall market has a beta of 1.5. And so on.

(This is the part where I have to acknowledge the argument that volatility and risk are two different things. Volatile stocks aren’t necessarily risky, folks argue, they just move around more than the market does on a given day. My view is the whole reason why someone would de-risk their portfolio is to avoid volatility, so I’m definitely mentioning beta.)

Professionals also use standard deviation to measure their risk. This looks at a portfolio’s results over a number of years and measures how far returns tend to stray from the average. A higher volatility there = more risk.

That’s hard for a lot of folks to measure, because I’ve found a lot of retail investors don’t actually keep track of their returns. And if they do, they generally don’t measure it against the index. Some will measure against an index, but perhaps not the right one. So this stuff can get a little complicated.

I know all this talk of measuring risk is killing everybody’s buzz right now, but it’s important to have an understanding of this stuff. You can’t de-risk the portfolio if you don’t understand just how much risk is in it in the first place. Or, if you’re like me, you might realize that your portfolio is already pretty boring, and maybe you’ve already de-risked it.

It’s easy to look at something like Kennedy’s example and declare this decision is easy. He got to the point where he was financially independent, and then let his foot off the gas. It doesn’t need to be more complicated than that.

I think it’s prudent to reveal just how rich Kennedy was. We don’t have exact numbers, but estimates are by the mid-1930s he was worth something in the neighborhood of $100-$150M, depending on how aggressively his private businesses and real estate were valued. That’s the equivalent of about $4B today.

JPK wasn’t just financially independent; he was one of the richest men in the world. Especially in an era when taxes were higher than they are today. Perhaps he isn’t the right example for those of us with more modest wealth.

My view is Kennedy was actually too late in adapting his thinking. He overshot. Obviously it worked out for him — his heirs are still reportedly worth a combined $1B to $1.5B today, nearly a century later, and that’s after spending much of the income generated by the investments — but in hindsight I’d say he probably took on too much risk.

Saying that, when’s the best time to settle down?

I like simplicity, so I’ll propose an easy rule. The time to de-risk is right when you reach financial independence. Use the 4% rule, figure out your spending, and combine the two numbers together.

At this point, the world is your oyster. Many of you won’t want to quit work. Having a job can give you a sense of purpose, socialization opportunities, or even just something to do. I get that. Being work optional is great. But you know what isn’t great? Being work optional, taking too many risks, and then being forced to work again.

There’s also a certain amount of logic in doing this in steps. For the first ten years of my investing journey, I took on as much debt as I could. I’d buy assets, pay them off, and then buy more. I made the decision to pay back my debts in 2015, and the only debt I had after that was a mortgage (along with some temporary margin debt in March-May, 2020). Once I realized all the heavy lifting was done I took my foot off the gas and let up a bit.

My view here is fairly simple. I took a bunch of risk off the table years ago by doing the following:

I’ve been around long enough to know that when the BIG ONE hits, you want to be prepared. I want to be buying stocks in such a scenario, not being forced to sell them to appease creditors. I also don’t want to have to worry how I’ll be able to afford food or keep the lights on.

I also keep a supply of food and cash (enough for about a week’s expenses, nothing major) at home as a just in case measure. The food thing is mostly because we buy in bulk, and sometimes you need a little bit of cash. I’m no doomer who is worried about the end of the world, but it is nice to know we’re somewhat prepared in that type of situation.

(Aside: I’ve never understood being a doomer. My view is I want to put my energy into living in this world, rather than worrying about some what-if scenario that probably won’t happen.)

I gradually made the portfolio more and more bulletproof as I went, so the de-risking process was easy. It felt almost natural.

I could go further. My portfolio is still mostly equities, even if it is mostly boring equities. I don’t own many bonds; if I do it’s mostly a place to park cash while I’m waiting for good investing opportunities. I could also own some precious metals, which tend to perform well when the rest of the financial world is going to hell. But I’m good. I think I’m fine without those asset classes. I understand the risk and I’m pretty okay with it.

The one change I have made since I started thinking about this is I have sold a few riskier assets in the portfolio. I look a massive loss on Goeasy (TSX:GSY) after I got that one wrong. I thought about hanging on, but realized a subprime lender isn’t something that meshes really well with my boring philosophy. A handful of other names that didn’t meet the criteria were punted, too, and have been replaced with less volatile assets.

These days I run stocks through about six criteria to see if I’m interested.

(Share buybacks are the only optional part of that list. I don’t need a buyback to invest, but I find many of the stocks I’m interested in these days have one.)

I spend much more time on the balance sheet than I ever did before. I used to be much more comfortable with companies that borrow to buy good assets. Now I scrutinize them more carefully. That’s not to say I’m going to sell companies like Enbridge (TSX:ENB) and TC Energy (TSX:TRP), but I am more inclined to take their generous dividends and put them to work in something that meshes better with the list above.

One other thing I’ll mention is my increasing infatuation with rising dividends. I used to be happy with buying stocks that offered more yield than dividend growth. The primary goal was maximizing my income. Growth was secondary. Now I want a little growth with my yield. I value both equally.

My view is a lot of stocks with lower dividends are actually higher quality businesses than those with higher ones. And I can maximize my dividend income by waiting to buy them while they’re on sale.

Ultimately, this is a personal question that only you can answer for your own portfolio. My experiences will only help a certain amount here, the rest you’ll have to figure out on your own.

These types of questions are also best answered by those who have a financial plan. A lot of people just want to accumulate the most amount of money possible. Others, like me (and maybe you) want to live off the dividends. Having a plan makes this all the easier. The plan dictates how you’ll act.

I’m a big fan of financial plans. You should have one — and your spouse should be an active participant.

You can follow Joe Kennedy’s lead. Or you can do what I did and gradually take risk off the table, pay down debt, and sit on more cash. Investing success is less about ending up with the biggest pile of money and more towards building up a portfolio that lets you sleep well at night. The last thing any of us wants to do is reach financial independence twice.