Where MMMs fail

Collinearity: channels that move together

If search and social budgets always rise and fall together, no regression can apportion credit between them; the data contains one signal wearing two names. Symptoms: unstable coefficients between refreshes, or a Bayesian posterior where two channels' estimates are strongly negatively correlated (the model knows the sum, not the split). The fix is operational, not statistical: desynchronize the budgets, or accept a merged "paid digital" estimate and say so on the slide.

Flat spend and small channels

A channel that spent the same amount for two years is statistically invisible; its steady effect is absorbed into base, and the model will cheerfully report it as near-zero ROI. This is a property of the data, not a finding about the channel. Similarly, a channel at 2% of spend cannot usually clear the noise floor; its interval will span "waste" to "best channel". Robyn's docs are candid that variation is what the search algorithm feeds on.¹ Deliberate spend pulses, 20% up or down for a few weeks, are cheap information purchases.

Overfitting to promotions

Promo-heavy calendars create spikes that dwarf media effects. A model without promo controls assigns those spikes to whatever media coincided with them, usually inflating the channels that surge around sales events. The resulting ROIs look great and reallocate money toward coincidence. The tell: remove promotion weeks from the holdout and watch predictive accuracy collapse. The fix is boring: log every promotion as a control variable, as the data requirements page insists.

Detecting a bad model before it spends money

Four checks, all cheap. Out-of-sample accuracy: predict a held-out quarter within useful error. Refresh stability: ROIs that swing 40% per refresh are noise with a subscription. Experiment agreement: one geo test against the model's estimate for a big channel.² And the interval test: a model reporting tight intervals on a 2%-of-spend channel is overconfident somewhere you can verify, so assume it is overconfident where you cannot. A model can pass all four and still be wrong; it cannot fail them and be right.

If several failure modes apply to your data, the honest conclusion may be not using MMM yet.