161 trading hypotheses: what survived, and where my tests were wrong

After my last few posts, someone gave me blunt feedback: stop testing retail trading lore. Smart Money Concepts, fair value gaps, opening-range breakouts: millions of people trade them, they were tuned on charts by eye, and there’s no economic reason for them to pay. Test ideas with a reason to exist instead, like trend-following, carry and value, the documented risk premia that professional funds actually run.

It’s good advice. It also made me realise I’d never looked at my own research that way. My lab has now registered 161 hypotheses, each written down before it was tested and run through the same pass marks. So I sorted all of them by where the idea came from and checked whether “ideas with a reason” actually did better.

They did, a bit. Not as much as they should have, and the reasons are more interesting than “my ideas were bad”.

The scoreboard

I put every hypothesis into one of four groups:

• Chart patterns and technical rules: Smart Money Concepts, opening-range breakouts, VWAP rules, moving-average crossovers, volume-spike breakouts.
• Published factors and risk premia: trend-following, carry in currencies, commodities, bonds and crypto, value, quality, low-risk, momentum and reversal in stocks, pairs trading.
• Macro timing signals: rules that own US stocks and step aside when a gauge such as credit spreads or the yield curve looks stressed.
• Calendar, event and intraday effects: turn of the month, the drift before Fed meetings, overnight returns, late-day momentum, and lately bond auctions in Nigeria.

Each dot is a hypothesis with a recorded Sharpe ratio after costs (120 of the 161; the earliest results predate the lab’s structured records). Orange dots passed every test. The buy-and-hold line applies to the macro timers, which all trade the S&P 500 and Nasdaq-100.

The chart-pattern group is as bad as the feedback predicted. None of the 23 passed. Of the 13 with a recorded Sharpe ratio, the best scored 0.30 and the median was −1.06. I wrote about the Smart Money Concepts tests in detail; the rest of the group failed the same way.

The macro timers look like the clear winners: 14 of 54 passed. But I’ve already shown what that’s worth. Run through exactly the same tests, simply holding the market scored a Sharpe ratio of 0.75, right in the middle of the timers. Most of them were invested nearly all the time, so they were mostly buy-and-hold with extra steps. Their median of 0.72 is the stock market’s Sharpe ratio, not theirs.

Which leaves the group the feedback pointed me to. Published risk premia had a median of 0.16 (over the 36 with a recorded Sharpe ratio), and only 3 of 59 passed.

The three premia that passed

• Cross-asset momentum and reversal (Bitcoin, stocks, gold, Treasuries) passed with a Sharpe ratio of 0.51, then lost on the sealed 2023–2026 data (−0.50).
• Pairs trading in seven same-industry stock pairs passed with 0.67, then lost on the sealed data (−1.10). Both are covered in my earlier post on the first 56 hypotheses.
• Commodity carry with a drawdown circuit breaker passed with 0.86 and is the lab’s cleanest result in this group. But it trades only two commodities (oil and natural gas), sits in cash 69% of the time, and depends on its carry-measurement window: 21 days gave 0.86, 63 days gave 0.30. Its sealed-data exam hasn’t been run yet.

Why the textbook premia mostly failed

Mostly, I tested them in a form too small to work.

The research these premia come from uses broad, diversified portfolios. The classic paper on trend-following, Moskowitz, Ooi and Pedersen (Journal of Financial Economics, 2012), traded 58 futures contracts across stock indices, bonds, currencies and commodities. My versions were much narrower:

• Currency carry on 5 to 10 currency pairs. Carry earns small, steady returns and crashes occasionally; with that few pairs, one crash wipes out years. My widest version scored −0.05.
• Commodity trend-following on six commodity ETFs, scoring 0.08 to 0.15. ETFs that hold futures pay to roll them every month, a cost a futures trader manages differently.
• Stock factors on large US stocks, long the best fifth and short the worst: betting against beta −0.35, value −0.49, a simple quality score −1.40.
• Crypto funding-rate carry on Bitcoin and Ether: 0.13 and 0.33.

That’s not evidence these premia don’t exist. It’s evidence that a premium earning a Sharpe ratio of 0.5 to 1 across dozens of markets earns much less in five of them, after costs, over one bad decade. The diversification is the strategy.

Where my tests were wrong

Some premia did clear the Sharpe bar and failed elsewhere, mostly for good reasons, such as a 58% drawdown or too many related attempts. Two were thrown out for reasons I now think were mistakes in my own tests.

Trend-following across 25 markets (the textbook 12-month rule on 24 ETFs and Bitcoin) scored a Sharpe ratio of 0.59 out of sample, above the 0.5 pass mark. It failed two tests:

• Walk-forward efficiency of −37.11. That number is the out-of-sample Sharpe divided by the average in-sample Sharpe, which was −0.02. Dividing by something that close to zero produces nonsense, and because it was slightly negative, the strategy was punished for doing better on unseen data than on the data it was built on. The bottom of that ratio is also an average over single assets, while the top is the whole portfolio, so it isn’t even comparing like with like.
• Regime consistency, 0 of 4. The test checks whether the signal predicts each asset’s next month separately in rising, falling, volatile and sideways markets, and those regimes are defined by the asset’s own return over the past three months. That is close to what a trend signal measures, so inside a “rising” market the signal is nearly always “long” and there is little variation left for the test to score. The money a trend portfolio makes comes from being right across many assets at once, which a per-asset check inside each regime doesn’t see.

Crypto momentum (Bitcoin and Ether) scored 0.60, and failed walk-forward efficiency the same way (−8.40, from an in-sample Sharpe of −0.07), along with the regime test and a parameter check.

Neither of these would have been a sure thing. The trend version wasn’t really diversified: 19 of its 25 markets were stock funds, and the crypto one rested on three test years. But they were rejected on a broken ratio, not on performance.

I can’t go back and pass them now. Changing the rules after seeing which ideas they hurt is exactly what the rules exist to prevent. The honest fix is to write new pass marks for diversified premia first, then test a properly built portfolio on them.

The two flaws point in opposite directions, which is the lesson. The gauntlet was too strict on diversified premia, because of ratios and regime labels that don’t fit them. And it was too lenient on anything that mostly owned the stock market, because it never asked whether a strategy beat simply holding. A fixed set of tests encodes assumptions about what a strategy looks like, and they don’t fit every strategy.

Testing somewhere I might have an edge

The other half of the feedback was to look where few quants do. I’ve built a naira yield curve from public data, so I started there.

The idea is a documented one with a mechanism: when a government sells new bonds, dealers have to absorb the supply and want a discount for it. Lou, Yan and Zhang (Review of Financial Studies, 2013) found US Treasury yields rise in the days before auctions and fall back after. Nigeria’s Debt Management Office auctions FGN bonds every month, so I pre-registered the same test there, on daily FMDQ prices from 2022 to 2026 and weekly DMO reports from 2010 to 2017.

It wasn’t there. The average yield move before an auction minus the move after was −0.4 basis points over 55 auctions (95% interval −11.3 to +10.1), and −3.0 over 56 auctions in the earlier period. The before-minus-after difference was positive in about half the auctions (49% and 46%). The data can rule out a concession bigger than about 10 basis points, though not one as small as the US estimate. One possible reason it doesn’t carry over: Nigerian banks and pension funds buy at the auction itself rather than selling ahead of it.

So that’s 161, and one more negative. But it’s the first test in a market where I know the data better than most people, and it shows the pipeline works there.

What I’m changing

• New pass marks for diversified strategies, written before the next test: judge the portfolio’s Sharpe ratio and its confidence interval, check decades separately instead of dividing two Sharpe ratios, and define market regimes per asset.
• A buy-and-hold benchmark on every test, not just the ones that look like the market.
• A wider trend and carry portfolio across bonds, currencies, commodities and stock indices, on longer free data, tested once against those new pass marks.
• The commodity carry result gets its sealed-data exam.
• More Nigerian tests, run forward. I’ve pre-registered a read of the bond market before each Central Bank rate decision; the next one is on 24 November, so the prediction goes out before the decision.

The feedback was right about the chart patterns: 0 for 23. Where it was too optimistic was in thinking that switching to “ideas with a reason” would fix things by itself. A real premium, tested too narrowly or judged by the wrong yardstick, fails just as cleanly as a fake one. The difference is that a fake one fails for good.

How I checked these numbers

Every hypothesis was written down and registered before it was tested. The groupings, the table and the chart come from a script that reads the lab’s registry, and a separate check script compares every number in this post against the registry and the pre-registrations.