Why Correlation Matters More Than Returns

The Counterintuitive Truth About Diversification

What if adding an asset that loses money to your portfolio could make you more money overall?

Not as a trick. Not as a hedge in the naive sense. But mathematically, rigorously, provably — because of one number: correlation.

This is the foundation of Modern Portfolio Theory (MPT), and it is the reason every systematic fund obsesses over correlation matrices before they look at expected returns. By the end of this post you will:

• Understand the portfolio variance formula at the derivation level
• Build and interpret a correlation matrix from scratch in Python
• Know why diversification breaks down in crises — and what to do about it

1. The Two-Asset Portfolio Variance Formula

Suppose you hold two assets, A and B, with weights $w_1$ and $w_2 = 1 - w_1$.

Each asset has:

• Expected return: $\mu_1$, $\mu_2$
• Volatility (standard deviation of returns): $\sigma_1$, $\sigma_2$
• Correlation between their returns: $\rho_{12} \in [-1, +1]$

The portfolio variance is:

And portfolio volatility is its square root:

The key insight lives in the third term. When $\rho_{12} < 0$, the cross-term is negative — it subtracts from total variance. You get volatility reduction simply because the assets move in opposite directions.

Worked example

Let Asset A be an equity ETF: $\sigma_1 = 20\%$, $\mu_1 = 8\%$ per year. Let Asset B be a bond ETF: $\sigma_2 = 15\%$, $\mu_2 = 4\%$ — worse return, lower volatility. Assume $\rho_{12} = -0.35$ (typical equity–bond correlation in calm markets).

With a 50/50 split:

Asset A alone: 20% vol. Asset B alone: 15% vol. The 50/50 portfolio: 10.2% vol — lower than either asset in isolation. That is the diversification benefit.

2. Interactive: Portfolio Volatility vs Correlation

Drag the weight slider to explore how the 50/50 assumption changes things.

Key observations:

• At $\rho = +1$: volatility equals the weighted average of individual vols — no diversification benefit
• At $\rho = 0$: meaningful reduction from the square-root effect
• At $\rho = -1$: a zero-volatility portfolio is possible — the assets perfectly offset each other
• The curve is convex: risk reduction accelerates as correlation drops below zero

3. Scaling to N Assets: The Covariance Matrix

With more than two assets, the formula generalises via matrix algebra. For a portfolio with $N$ assets and weight vector $\mathbf{w}$:

where $\boldsymbol{\Sigma}$ is the $N \times N$ covariance matrix:

The diagonal is variances. The off-diagonal entries are covariances $\text{Cov}(r_i, r_j) = \sigma_i \sigma_j \rho_{ij}$. The matrix is symmetric.

For 5 assets: 10 unique pairwise correlations to estimate. For 100 assets: 4,950. Covariance estimation is one of the hardest practical problems in quantitative finance.

4. Building the Correlation Matrix in Python

import numpy as np
import pandas as pd
 
# ─── Simulated daily returns for 5 assets ─────────────────────────────────
np.random.seed(42)
n_days = 252  # 1 trading year
 
# True correlation structure (normal market)
corr_true = np.array([
    [1.00, -0.35,  0.05,  0.72,  0.65],
    [-0.35,  1.00,  0.28, -0.20, -0.28],
    [ 0.05,  0.28,  1.00,  0.08,  0.10],
    [ 0.72, -0.20,  0.08,  1.00,  0.55],
    [ 0.65, -0.28,  0.10,  0.55,  1.00],
])
vols = np.array([0.20, 0.08, 0.15, 0.22, 0.25]) / np.sqrt(252)
 
# Convert correlation → covariance matrix
cov_true = np.outer(vols, vols) * corr_true
 
# Generate multivariate normal returns
L = np.linalg.cholesky(cov_true)    # Cholesky decomposition
Z = np.random.randn(n_days, 5)
returns = Z @ L.T
 
assets = ['SPY', 'TLT', 'GLD', 'VNQ', 'EEM']
df = pd.DataFrame(returns, columns=assets)
 
# ─── Compute sample correlation matrix ────────────────────────────────────
corr_sample = df.corr()
print("Sample correlation matrix:")
print(corr_sample.round(2))
 
# ─── Portfolio variance: equal-weight ─────────────────────────────────────
w = np.array([0.2, 0.2, 0.2, 0.2, 0.2])
cov_sample = df.cov() * 252   # annualise
 
port_var = w @ cov_sample.values @ w
port_vol = np.sqrt(port_var)
print(f"\nAnnualised portfolio volatility: {port_vol:.2%}")
 
# ─── Marginal contribution to risk ────────────────────────────────────────
mctr = (cov_sample.values @ w) / port_vol
mctr_df = pd.DataFrame({'Asset': assets, 'MCTR': mctr, 'Weight': w})
mctr_df['Risk Contribution (%)'] = (
    mctr_df['MCTR'] * mctr_df['Weight'] / port_vol * 100
)
print("\nMarginal contributions to portfolio risk:")
print(mctr_df.round(3))

What this code does:

1. Generates synthetic daily returns from a known correlation structure using Cholesky decomposition
2. Computes the sample correlation matrix from the simulated returns
3. Calculates annualised portfolio volatility using $\mathbf{w}^\top \boldsymbol{\Sigma} \mathbf{w}$
4. Computes each asset's Marginal Contribution to Total Risk (MCTR) — essential for understanding which positions are actually driving risk

5. Interactive: The Correlation Heatmap

Visualise the difference between normal-market and crisis correlation structures. Toggle between regimes — notice how the equity–real estate pair moves from 0.72 to 0.92 under stress.

Reading the heatmap:

• Blue = high positive correlation — assets move together. No diversification benefit.
• Red = negative correlation — assets move opposite. Genuine hedging.
• Dark = near zero — uncorrelated. Independent risk exposures.

In normal markets, SPY (equities) and TLT (long bonds) are negatively correlated at roughly −0.35. When equities sell off, investors flee to bonds, pushing bond prices up — the classic 60/40 portfolio logic.

In a crisis, that relationship weakens dramatically. The 2020 COVID crash saw a brief period where everything sold off simultaneously as leveraged funds faced margin calls and liquidated across all asset classes. The equity–real estate correlation jumped from 0.72 to over 0.90.

6. The Crisis Problem: Why Diversification Fails When You Need It

Harry Markowitz gave us the theoretical basis for diversification in 1952. What practitioners discovered over 70 years of live trading is that the correlation matrix you estimate during calm periods is not the one that will hold during stress.

This is called correlation instability, and it is one of the most dangerous hidden assumptions in any portfolio optimisation framework.

Stress-testing your correlation assumptions

import numpy as np
 
def portfolio_vol(weights, cov_matrix):
    return np.sqrt(weights @ cov_matrix @ weights)
 
normal_corr = np.array([
    [1.00, -0.35,  0.05,  0.72,  0.65],
    [-0.35,  1.00,  0.28, -0.20, -0.28],
    [ 0.05,  0.28,  1.00,  0.08,  0.10],
    [ 0.72, -0.20,  0.08,  1.00,  0.55],
    [ 0.65, -0.28,  0.10,  0.55,  1.00],
])
crisis_corr = np.array([
    [1.00, -0.10,  0.30,  0.92,  0.94],
    [-0.10,  1.00,  0.15, -0.08, -0.12],
    [ 0.30,  0.15,  1.00,  0.28,  0.35],
    [ 0.92, -0.08,  0.28,  1.00,  0.88],
    [ 0.94, -0.12,  0.35,  0.88,  1.00],
])
 
vols_annual = np.array([0.20, 0.08, 0.15, 0.22, 0.25])
 
def corr_to_cov(corr, vols):
    D = np.diag(vols)
    return D @ corr @ D
 
cov_normal = corr_to_cov(normal_corr, vols_annual)
cov_crisis = corr_to_cov(crisis_corr, vols_annual)
 
w = np.ones(5) / 5  # equal weight
 
vol_normal = portfolio_vol(w, cov_normal)
vol_crisis = portfolio_vol(w, cov_crisis)
 
print(f"Equal-weight portfolio volatility:")
print(f"  Normal market:  {vol_normal:.2%}")
print(f"  Crisis regime:  {vol_crisis:.2%}")
print(f"  Vol increase:   {(vol_crisis/vol_normal - 1):.1%}")
 
# Diversification ratio: weighted-average vol / portfolio vol
weighted_avg_vol = np.dot(w, vols_annual)
div_ratio_normal = weighted_avg_vol / vol_normal
div_ratio_crisis = weighted_avg_vol / vol_crisis
 
print(f"\nDiversification ratio (>1 = benefit):")
print(f"  Normal: {div_ratio_normal:.2f}x")
print(f"  Crisis: {div_ratio_crisis:.2f}x  ← benefit nearly halved")

Running this shows a 5-asset equal-weight portfolio seeing volatility jump 40–60% moving from normal to crisis correlation assumptions with identical weights.

7. What To Do About Correlation Instability

1. Stress-test with crisis matrices Always compute portfolio volatility under at least two scenarios. The gap is your hidden risk.

2. Include genuine diversifiers Long-dated government bonds, gold, and options-based strategies maintain low or negative correlation to equities even in crises — though their effectiveness varies by regime.

3. Use non-linear diversifiers Put options have convex payoffs — they gain value precisely when correlations spike and portfolios collapse. A small tail-risk hedge dramatically reduces left-tail outcomes.

4. Hierarchical Risk Parity (HRP) Rather than estimating the full covariance matrix and inverting it (which amplifies estimation error), HRP uses hierarchical clustering to build portfolios robust to correlation uncertainty. Covered in Episode 9.

5. Dynamic correlation regimes Fit a regime-switching model or GARCH-DCC to detect when the market has shifted into a high-correlation state and adjust weights proactively.

Key Takeaways

1. Returns are only half the equation. A low-return, low-correlation asset can improve risk-adjusted performance.

2. The portfolio variance formula is not additive. The cross-term $2 w_1 w_2 \sigma_1 \sigma_2 \rho_{12}$ is where diversification lives.

3. For $N$ assets, think in matrix form: $\sigma_p^2 = \mathbf{w}^\top \boldsymbol{\Sigma} \mathbf{w}$

4. Correlations are regime-dependent. Calm-market estimates understate crisis risk. Always stress-test.

5. Diversification ratio > 1 = genuine benefit. If portfolio vol equals weighted-average individual vol, you have gained nothing from the combination.

What's Next

Episode 2 builds on this directly: given a universe of assets, what is the mathematically optimal portfolio for every level of risk? This is the Efficient Frontier — solved via constrained quadratic optimisation. We also tackle the most common failure of mean-variance optimisation — estimation error in expected returns — and derive the fix.

References

Foundational Theory

1. Markowitz, H. (1952). "Portfolio Selection." Journal of Finance, 7(1), 77–91. https://doi.org/10.2307/2975974

2. Markowitz, H. (1959). Portfolio Selection: Efficient Diversification of Investments. Yale University Press / Wiley. ISBN: 978-0-300-01372-6

3. Sharpe, W.F. (1964). "Capital Asset Prices: A Theory of Market Equilibrium under Conditions of Risk." Journal of Finance, 19(3), 425–442. https://doi.org/10.2307/2977928

4. Bernoulli, D. (1954 [1738]). "Exposition of a New Theory on the Measurement of Risk." Econometrica, 22(1), 23–36. https://doi.org/10.2307/1909829

Covariance Estimation

5. Ledoit, O. & Wolf, M. (2004). "Honey, I Shrunk the Sample Covariance Matrix." Journal of Portfolio Management, 30(4), 110–119. https://doi.org/10.3905/jpm.2004.110

6. Ledoit, O. & Wolf, M. (2003). "Improved Estimation of the Covariance Matrix of Stock Returns with an Application to Portfolio Selection." Journal of Empirical Finance, 10(5), 603–621. https://doi.org/10.1016/S0927-5398(03)00007-0

Correlation Dynamics & Crisis Behaviour

7. Longin, F. & Solnik, B. (2001). "Extreme Correlation of International Equity Markets." Journal of Finance, 56(2), 649–676. https://doi.org/10.1111/0022-1082.00340

8. Marti, G., Nielsen, F., Bińkowski, M. & Donnat, P. (2017). "A Review of Two Decades of Correlations, Hierarchies, Networks and Clustering in Financial Markets." Progress in Information Geometry, pp. 245–274. https://arxiv.org/abs/1703.00485

9. Mantegna, R.N. (1999). "Hierarchical Structure in Financial Markets." European Physical Journal B, 11, 193–197. https://doi.org/10.1140/epjb/e1999-00052-2

Portfolio Optimisation Extensions

10. Black, F. & Litterman, R. (1992). "Global Portfolio Optimization." Financial Analysts Journal, 48(5), 28–43. https://doi.org/10.2469/faj.v48.n5.28

11. López de Prado, M. (2016). "Building Diversified Portfolios that Outperform Out of Sample." Journal of Portfolio Management, 42(4), 59–69. https://ssrn.com/abstract=2708678

12. Maillard, S., Roncalli, T. & Teïletche, J. (2010). "The Properties of Equally Weighted Risk Contributions Portfolios." Journal of Portfolio Management, 36(4), 60–70. https://doi.org/10.3905/jpm.2010.36.4.060

Textbooks & Practitioner References

13. Chan, E.P. (2009). Quantitative Trading: How to Build Your Own Algorithmic Trading Business. Wiley. ISBN: 978-0-470-28488-4

14. Tulchinsky, I. et al. (2020). Finding Alphas: A Quantitative Approach to Building Trading Strategies (2nd ed.). Wiley. ISBN: 978-1-119-57178-7

15. de Prado, M.L. (2018). Advances in Financial Machine Learning. Wiley. ISBN: 978-1-119-48208-6