The Library has 84 entries and a much smaller set of ideas. This page is the ideas: what is summed, what multiplies what, which term the answer turns on, and which entries run on each. Everything is written with symbols and without values.
Weight each outcome by how likely it is and add them up. Then ask whether dollars are the right unit: the same expected dollars can be a fine bet for a large fund and a ruinous one for a small one, which is what utility, as opposed to money, is for.
General formEV = sum over states s of p(s) x X(s)
EU = sum over states s of p(s) x U(X(s)), with U concave for a risk-averse decision maker
Certainty equivalent CE solves U(CE) = EU; the risk premium is EV - CE
What drives the answer
The tail: a low-probability state with a very large payoff or loss can dominate the sum, and the curvature of U decides how much a possible ruin outweighs a likely gain.
Where it breaks
That the probabilities were asserted rather than derived, and that the payoffs were point estimates where distributions were needed. Both are the flaw of averages in another form.
Runs under
Expected value is not the decisionThe base case is not the expected caseSome uncertainties have no defensible numberAttractive on its own, wrong for this portfolioHow much of the bet to take
Sources
Pratt (1964). Risk Aversion in the Small and in the Large. Econometrica 32(1/2), 122. https://doi.org/10.2307/1913738
Kahneman, Tversky (1979). Prospect Theory: An Analysis of Decision under Risk. Econometrica 47(2), 263. https://doi.org/10.2307/1914185
Start from what you believed, look at what the new evidence would look like if the hypothesis were true versus false, and move your belief by that ratio. The odds form is the working version: posterior odds equal prior odds times the likelihood ratio.
General formP(H | E) = P(E | H) x P(H) / P(E)
Odds(H | E) = Odds(H) x LR, where LR = P(E | H) / P(E | not H)
In logs across independent pieces of evidence: log Odds(H | E1..En) = log Odds(H) + sum over i of log LR(i)
What drives the answer
The likelihood ratio, not the evidence itself. Evidence that would look the same whether or not the hypothesis holds moves nothing, however vivid it is. And the log-sum form is valid only when the pieces are independent.
Where it breaks
That the prior was chosen to reach the conclusion, that the likelihood ratios are invented, or that five correlated pieces of evidence were counted five times.
Runs under
The outside view before the inside viewThe thesis as a list of things that must be trueWho is reliable about whatFive sources, or one source heard five timesSeveral partial accounts, one reconstructed factHow to ask an expert for a number
Sources
Bayes (1763). An Essay towards Solving a Problem in the Doctrine of Chances. Philosophical Transactions of the Royal Society of London(53), 370-418. https://doi.org/10.1098/rstl.1763.0053
Information is worth exactly as much as the decision it would change. Perfect information is worth the gap between deciding after you know and deciding before; a real study, which reveals only some of the truth, is worth less, and it is worth pursuing only if that value exceeds its cost.
General formV0 = max over actions a of E[U(a, s)]
EVPI = E[max over a of U(a, s)] - V0
EVSI(study) = E over results z of [max over a of E[U(a, s) | z]] - V0
Net value = EVSI - cost; pursue only if positive, and rank studies by (EVSI - cost) / time
What drives the answer
Whether any plausible result would flip the decision. A study whose every outcome leaves you doing the same thing has zero value regardless of how interesting it is.
Where it breaks
That the decision was already made and the study is theater; that the set of actions was too narrow, so nothing could flip; or that the probabilities feeding the calculation were guesses.
Runs under
What is the answer worth?The one question to answer nextWhen to stop paying for researchThe research budget is itself an allocation
Sources
Howard (1966). Information Value Theory. IEEE Transactions on Systems Science and Cybernetics 2(1), 22-26. https://doi.org/10.1109/TSSC.1966.300074
Howard (1988). Decision Analysis: Practice and Promise. Management Science 34(6), 679-695. https://doi.org/10.1287/mnsc.34.6.679
Lay out the choices and the chance events in the order they happen, put values at the ends, and solve from the right: at each chance node take the expectation, at each decision node take the best branch. The tree makes visible that research, waiting, and staging are themselves decisions.
General formAt a chance node: V = sum over branches i of p(i) x V(i)
At a decision node: V = max over branches of V(branch)
Solved from the leaves back to the root
What drives the answer
The order of nodes. Putting a piece of information before a decision rather than after it changes the value of the whole tree, which is where option value and the value of information both come from.
Where it breaks
That the tree omitted the branch that mattered (usually "wait" or "do it in stages"), or that the probabilities at chance nodes were asserted.
Runs under
Research is a decision, so is waitingWhat is the answer worth?Waiting has a value, and sometimes a priceDecide the order before the answerThe option you did not list
Sources
Raiffa, Howard (1968). Decision Analysis: Introductory Lectures on Choices under Uncertainty. Addison-Wesley. (book)
Bertsimas, Dimitris and Freund, Robert M. (2004). Data, Models, and Decisions: The Fundamentals of Management Science. Dynamic Ideas. (book)
The right, without the obligation, to do something later has value, and it has more value the more uncertain the future is. Waiting is not indecision when the next quarter resolves a variable the decision turns on.
General formValue of the option to wait = E[max(V(t+1) - I, 0)] discounted, against the value of investing now, V(t) - I
Invest now only when V exceeds a threshold V* strictly above I; the gap V* - I grows with the volatility of V
What drives the answer
Volatility and irreversibility together. A reversible decision has no option value to protect; an irreversible one under high uncertainty should wait for a much bigger margin than a simple net-present-value test would demand.
Where it breaks
That the option is illusory because the window closes (a competitor moves, the seller walks), or that the uncertainty will not in fact resolve with time.
Runs under
Research is a decision, so is waitingWaiting has a value, and sometimes a price
Sources
McDonald, Siegel (1986). The Value of Waiting to Invest. The Quarterly Journal of Economics 101(4), 707. https://doi.org/10.2307/1884175
Dixit, Avinash K. and Pindyck, Robert S. (1994). Investment under Uncertainty. Princeton University Press. (book)
Keep gathering evidence while the expected value of the next piece exceeds its cost; stop when it does not. Written as a rule, it tells a research firm when to tell its client to stop paying it.
General formV(state) = max of { U(stop | state), max over studies q of [ -c(q) + E[V(state after q)] ] }
Stop when U(stop) is at least -c(q) + E[V(next)] for every remaining q; equivalently when the best remaining net value of information is at or below zero
What drives the answer
How much the remaining uncertainty still bears on the decision. Once the recommendation is stable across the plausible results of every remaining study, the state is saturated.
Where it breaks
That the stopping call is self-serving in either direction: a firm paid by the hour never stops, a firm paid a fixed fee stops early. Stating the rule in advance is the defense.
Runs under
When to stop paying for researchDecide the order before the answer
Sources
Wald (1945). Sequential Tests of Statistical Hypotheses. The Annals of Mathematical Statistics 16(2), 117-186. https://doi.org/10.1214/aoms/1177731118
Bellman (1954). The theory of dynamic programming. Bulletin of the American Mathematical Society 60(6), 503-515. https://doi.org/10.1090/S0002-9904-1954-09848-8
Instead of one base case with a bull and a bear beside it, draw every uncertain input from its distribution thousands of times and look at the whole spread of outcomes. The average of the model is not the model of the average.
General formY = f(X1, ..., Xk) with each Xi drawn from its distribution, with correlations where they exist
E[Y] is approximated by (1/N) x sum over draws j of f(X(j)); read off P(Y below a threshold), the median, and the tails
By Jensen, E[f(X)] differs from f(E[X]) whenever f is not linear
What drives the answer
The input whose variance explains most of the variance of Y; that is the next thing to research. And the correlations: independent draws understate the tails when the inputs move together.
Where it breaks
That the input distributions were made up, so the impressive histogram is a picture of assumptions; and that correlations were ignored.
Runs under
The base case is not the expected caseNear misses are data about the disaster
Sources
Metropolis, Ulam (1949). The Monte Carlo Method. Journal of the American Statistical Association 44(247), 335-341. https://doi.org/10.1080/01621459.1949.10483310
Jensen (1906). Sur les fonctions convexes et les inégalités entre les valeurs moyennes. Acta Mathematica 30(0), 175-193. https://doi.org/10.1007/BF02418571
A forecaster who says seventy percent should be right about seventy percent of the time. Scoring rules make that checkable, punish confident misses hardest, and turn a research firm's judgment into a track record.
General formBrier score = (1/N) x sum over i of (p(i) - y(i))^2, where y is 1 if the event occurred
Log score = -(1/N) x sum over i of [y(i) log p(i) + (1 - y(i)) log(1 - p(i))]
Calibration: among all forecasts near p, the realized frequency should be near p
What drives the answer
Resolution time. Investment outcomes take years, so the score arrives long after the forecast; a library of predictions must be preserved unaltered to be scorable at all.
Where it breaks
That the probabilities were never recorded, were revised after the fact, or were reported as bands too wide to be wrong.
Runs under
Was the seventy percent right seventy percent of the time?How far ahead anyone can see
Sources
Brier (1950). Verification of Forecasts Expressed in Terms of Probability. Monthly Weather Review 78(1), 1-3. https://doi.org/10.1175/1520-0493(1950)078<0001:VOFEIT>2.0.CO;2
When the outcome depends on what other purposeful actors do, the question is not what they will do in isolation but what each will do given what the others do. An equilibrium is a set of choices from which no one gains by changing alone.
General formStrategies a* are a Nash equilibrium if for every player i, u(i)(a*(i), a*(-i)) is at least u(i)(a(i), a*(-i)) for every alternative a(i)
In sequential settings, solve by backward induction for subgame perfection: each mover optimizes given the best responses that follow
What drives the answer
The payoffs the other side actually faces, which are usually not the ones the client assumes. A plan that is optimal against a passive competitor can be dominated against a responding one.
Where it breaks
That the game was mis-specified (wrong players, wrong moves, wrong information), that multiple equilibria exist and one was chosen for convenience, or that the players are not the calculating actors the model assumes.
Runs under
Assume the competitor is not asleepWill they enter, and can you stop them?Why the truce holds, and when it breaksEveryone runs because everyone else mightA threat is only as good as its cost to withdrawWhat works for a monopolist backfires in an oligopolyRegulators and governments are players with payoffsWhich practices survive in an industryEveryone benefits, nobody paysWho should own the spilloverWhy rivals end up next to each otherCritical mass and lock-inShare wars and the square law
Sources
Nash (1951). Non-Cooperative Games. The Annals of Mathematics 54(2), 286. https://doi.org/10.2307/1969529
Gibbons, Robert (1992). Game Theory for Applied Economists. Princeton University Press. (book)
Tadelis, Steven (2013). Game Theory: An Introduction. Princeton University Press. (book)
People who expect to deal with each other again behave differently from people who will never meet again. A short-term gain from defecting is weighed against the discounted value of continued cooperation, and reputation is a bet on future dealings.
General formCooperation is sustainable when the one-period gain from deviating, G, is at most the discounted future loss: G <= delta x (V(cooperate) - V(punish)) / (1 - delta)
A higher discount factor delta (more patience, more future business) supports more cooperation
What drives the answer
The shadow of the future. Anything that shortens it, an exit, a sale, a retirement, a last period, weakens cooperation and predicts price wars, defections, and broken handshakes.
Where it breaks
That the end of the game is nearer than assumed, that punishment is not credible, or that the parties cannot observe deviations clearly enough to punish them.
Runs under
Will they enter, and can you stop them?Why the truce holds, and when it breaksDeals that should close and do not
Sources
Fudenberg, Maskin (1986). The Folk Theorem in Repeated Games with Discounting or with Incomplete Information. Econometrica 54(3), 533. https://doi.org/10.2307/1911307
Kreps, Wilson (1982). Reputation and imperfect information. Journal of Economic Theory 27(2), 253-279. https://doi.org/10.1016/0022-0531(82)90030-8
Green, Porter (1984). Noncooperative Collusion under Imperfect Price Information. Econometrica 52(1), 87. https://doi.org/10.2307/1911462
What each side gets depends on what each would get if talks failed, on who is more patient, and on who can commit. The outside option, not the argument, decides most negotiations.
General formNash bargaining: choose the split that maximizes (u1 - d1) x (u2 - d2), where d is each party's payoff if no deal is reached
Alternating offers: the more patient party (higher delta) captures more of the surplus, and the first mover an advantage that shrinks as offers get faster
What drives the answer
The disagreement points d1 and d2. Improving your own outside option, or credibly worsening the other side's, moves the split more than any concession in the room.
Where it breaks
That the outside options were misread (the seller has another buyer, or does not), that a commitment was not credible, or that private information about values makes efficient agreement impossible.
Runs under
The outside option decides the splitDeals that should close and do notA threat is only as good as its cost to withdrawRegulators and governments are players with payoffsGovernance is a voting gameWho should own the spillover
Sources
Nash (1950). The Bargaining Problem. Econometrica 18(2), 155. https://doi.org/10.2307/1907266
Rubinstein (1982). Perfect Equilibrium in a Bargaining Model. Econometrica 50(1), 97. https://doi.org/10.2307/1912531
Muthoo, Abhinay (1999). Bargaining Theory with Applications. Cambridge University Press. (book)
Design the rules so that people with private information find it in their interest to reveal it. Auctions are the clearest case: the format determines who wins, what they pay, and whether the winner regrets it.
General formA mechanism is incentive compatible if truthful reporting is a best response for every type
Revenue equivalence: under standard assumptions, formats that allocate to the same bidder yield the same expected revenue
Common values: E[V | you won] is less than E[V | your estimate], the winner's curse, so bids must be shaded
What drives the answer
Whether values are private (each bidder knows its own worth) or common (everyone estimates the same unknown). In common-value settings, winning is itself bad news about the estimate that won.
Where it breaks
That the bidders collude, that the seller cannot commit to the rules, or that the mechanism assumed rational bidders in a room that has none.
Runs under
Deals that should close and do notWhy are you the one winning?The format decides who wins and what they pay
Sources
Myerson (1981). Optimal Auction Design. Mathematics of Operations Research 6(1), 58-73. https://doi.org/10.1287/moor.6.1.58
Vickrey (1961). Counterspeculation, Auctions, and Competitive Sealed Tenders. The Journal of Finance 16(1), 8-37. https://doi.org/10.1111/j.1540-6261.1961.tb02789.x
Milgrom, Weber (1982). A Theory of Auctions and Competitive Bidding. Econometrica 50(5), 1089. https://doi.org/10.2307/1911865
Capen, Clapp, Campbell (1971). Competitive Bidding in High-Risk Situations. Journal of Petroleum Technology 23(06), 641-653. https://doi.org/10.2118/2993-PA
When one side knows more than the other, what is offered tells you something about what is not said. Sellers who know their goods are bad sell readily; claims that cost nothing to make carry no information; actions that would be costly for a liar carry a lot.
General formAdverse selection: the pool of goods offered at price p has quality E[q | seller accepts p], which falls as p falls, and the market can unravel
Signaling: a signal s separates types when the cost of sending it differs across types, c(s, high) < c(s, low), enough that only one type sends it
Cheap talk: costless messages are informative only when the sender's and receiver's interests are aligned enough that the sender prefers to be understood
What drives the answer
Whether the signal is costly to fake. Hiring forty salespeople, refusing to discount, buying stock, and accepting contingent pay are signals; a strong-pipeline sentence in a management meeting is talk.
Where it breaks
That the supposedly costly action was cheap for the sender after all, or that the receiver is reading intent into noise.
Runs under
What they said versus what it cost themWhy is this for sale?Let them sort themselvesOne price leaves money on the table
Sources
Akerlof (1970). The Market for "Lemons": Quality Uncertainty and the Market Mechanism. The Quarterly Journal of Economics 84(3), 488. https://doi.org/10.2307/1879431
Spence (1973). Job Market Signaling. The Quarterly Journal of Economics 87(3), 355. https://doi.org/10.2307/1882010
Crawford, Sobel (1982). Strategic Information Transmission. Econometrica 50(6), 1431. https://doi.org/10.2307/1913390
Rothschild, Stiglitz (1976). Equilibrium in Competitive Insurance Markets: An Essay on the Economics of Imperfect Information. The Quarterly Journal of Economics 90(4), 629. https://doi.org/10.2307/1885326
When effort cannot be observed, pay has to be tied to something that can be, and every observable measure rewards something other than what you wanted. Contracts trade risk for incentive, and the measure chosen tells you what will be gamed.
General formThe principal chooses a payment w(y) on observed outcome y to maximize E[y - w(y)] subject to the agent choosing effort e to maximize E[u(w(y)) - c(e)] and being willing to take the job
Informativeness: a signal belongs in the contract if and only if it carries information about effort beyond what the outcome already carries
What drives the answer
The noise in the measure and the agent's risk aversion together. Noisy outcomes with a risk-averse agent force weak incentives, which is why so many pay plans are weaker than they look.
Where it breaks
That the measure is gameable, that the agent controls the measurement, or that the contract rewards the measured task at the expense of the unmeasured one.
Runs under
The plan rewards what it measuresEveryone benefits, nobody pays
Sources
Holmstrom (1979). Moral Hazard and Observability. The Bell Journal of Economics 10(1), 74. https://doi.org/10.2307/3003320
Salanie, Bernard (2005). The Economics of Contracts: A Primer, 2nd ed. MIT Press. (book)
How much volume a price change buys or loses is the whole of pricing power. Charging different customers different prices, whether by version, bundle, or quantity, is how a seller captures the willingness to pay that one price leaves on the table.
General formElasticity e = (dQ/Q) / (dP/P); a profit-maximizing single price satisfies (P - MC) / P = -1 / e
Second-degree discrimination: a menu of (quality, price) pairs designed so each type selects the one meant for it; the low type's quality is distorted downward to keep the high type from imitating
Bundling profits when reservation values for the components are negatively correlated across customers
What drives the answer
The elasticity at the current price, which is almost never known and almost always asked about directly, which does not work. It is inferred from trade-offs, experiments, or discontinuities in the data.
Where it breaks
That the demand curve was drawn from a survey of stated intentions, that arbitrage undoes the segmentation, or that competitors respond.
Runs under
Let them sort themselvesDo not ask what they would payHow much volume does a price change buy or lose?One price leaves money on the tableWhat works for a monopolist backfires in an oligopolyCustomers, and managers, are not the modelWhy rivals end up next to each other
Sources
Mussa, Rosen (1978). Monopoly and product quality. Journal of Economic Theory 18(2), 301-317. https://doi.org/10.1016/0022-0531(78)90085-6
Adams, Yellen (1976). Commodity Bundling and the Burden of Monopoly. The Quarterly Journal of Economics 90(3), 475. https://doi.org/10.2307/1886045
Frank, Robert H. Microeconomics and Behavior. McGraw-Hill. (book)
People cannot tell you what they would pay, but they can choose between realistic alternatives, and the choices reveal what they trade off. The model turns a set of choices into a probability that each option is picked, and the coefficients into willingness to pay.
General formP(choose j) = exp(V(j)) / sum over k of exp(V(k)), where V(j) = beta x attributes(j)
Willingness to pay for an attribute = -beta(attribute) / beta(price)
Binary case: P(y = 1 | x) = 1 / (1 + exp(-x beta))
What drives the answer
The realism of the choice sets and the independence of the alternatives. A conjoint built on features customers do not actually weigh, or with options that are near-duplicates, produces confident nonsense.
Where it breaks
That stated choices differ from real purchases, that the sample is not the buyer, or that the price range shown anchored the answers.
Runs under
Do not ask what they would pay
Sources
Green, Srinivasan (1978). Conjoint Analysis in Consumer Research: Issues and Outlook. Journal of Consumer Research 5(2), 103-123. https://doi.org/10.1086/208721
Train (2009). Discrete Choice Methods with Simulation. Cambridge University Press (book). https://doi.org/10.1017/CBO9780511805271
Fit a line, or its multi-variable equivalent, to see what moves with what while holding the rest steady. It is the transparent baseline any fancier model must beat, and the place most causal claims quietly begin and should not end.
General formY = X beta + e; the least-squares estimate is beta-hat = (X'X)^-1 X'Y
Each coefficient is the association between one regressor and Y holding the others fixed, not the effect of changing it
What drives the answer
What was left out. An omitted variable that drives both X and Y shows up as a coefficient on X, which is why a regression coefficient is a correlation with manners, not a cause.
Where it breaks
Omitted variables, reverse causation, selection into the sample, and specification chosen after seeing the results.
Runs under
The transparent baseline any fancier model must beatTwo lines that both go upCompare the company to itselfLast year's best performerPredict, or explain
Sources
Galton (1886). Regression Towards Mediocrity in Hereditary Stature. The Journal of the Anthropological Institute of Great Britain and Ireland 15, 246. https://doi.org/10.2307/2841583
Wooldridge, Jeffrey M. Introductory Econometrics: A Modern Approach. Cengage. (book)
Every causal claim compares what happened with what would have happened otherwise, and the second thing is never observed. Every method for estimating a cause is a way of standing in for the world that did not occur: a control group, a comparison period, a discontinuity, an instrument.
General formEffect for unit i: tau(i) = Y(i, 1) - Y(i, 0), only one of which is ever seen
Average effect: ATE = E[Y(1) - Y(0)]; heterogeneous effect: CATE(x) = E[Y(1) - Y(0) | X = x]
Difference-in-differences: tau = (Y(treated, after) - Y(treated, before)) - (Y(control, after) - Y(control, before)), valid under parallel trends
Instrumental variables: Z shifts D and affects Y only through D; the effect is Cov(Z, Y) / Cov(Z, D)
Regression discontinuity: compare units just above and just below a threshold that assigns treatment
What drives the answer
The identifying assumption, which is a statement about the unobserved world and cannot be tested by the data it is applied to. Parallel trends, exclusion, and no manipulation at the cutoff are the three that most often fail.
Where it breaks
That before-and-after on its own proved nothing, that the control group differs in the way that matters, that the instrument has a second channel, or that the effect estimated for the marginal units is not the effect for the population the thesis is about.
Runs under
How much volume does a price change buy or lose?Compared to what?Before and after, with someone who did not get itThe people just above and just below the lineSomething that moved the cause and nothing elseFor whom, not whetherSometimes the answer is to try it on a fewCompare the company to itselfPredict, or explain
Sources
Rubin (1974). Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of Educational Psychology 66(5), 688-701. https://doi.org/10.1037/h0037350
Ashenfelter, Card (1985). Using the Longitudinal Structure of Earnings to Estimate the Effect of Training Programs. The Review of Economics and Statistics 67(4), 648. https://doi.org/10.2307/1924810
Bertrand, Duflo, Mullainathan (2004). How Much Should We Trust Differences-In-Differences Estimates?. The Quarterly Journal of Economics 119(1), 249-275. https://doi.org/10.1162/003355304772839588
Angrist, Imbens, Rubin (1996). Identification of Causal Effects Using Instrumental Variables. Journal of the American Statistical Association 91(434), 444-455. https://doi.org/10.1080/01621459.1996.10476902
Thistlethwaite, Campbell (1960). Regression-discontinuity analysis: An alternative to the ex post facto experiment. Journal of Educational Psychology 51(6), 309-317. https://doi.org/10.1037/h0044319
Imbens, Lemieux (2008). Regression discontinuity designs: A guide to practice. Journal of Econometrics 142(2), 615-635. https://doi.org/10.1016/j.jeconom.2007.05.001
Athey, Imbens (2016). Recursive partitioning for heterogeneous causal effects. Proceedings of the National Academy of Sciences 113(27), 7353-7360. https://doi.org/10.1073/pnas.1510489113
Who you heard from decides what you learned. Customers who answer differ from those who do not; the ten calls that agreed may be ten calls from one implementation partner; and a sample too small to detect the effect you care about produces false reassurance, not evidence.
General formSelection: E[Y | observed] differs from E[Y] whenever the chance of being observed depends on Y; Heckman's correction models the selection equation explicitly
Nonresponse bias in a mean = (nonresponse rate) x (difference between respondents and nonrespondents)
Power: the sample size needed to detect an effect of size d at conventional thresholds scales with 1/d^2
What drives the answer
The mechanism that put a unit in the sample. A survey of surviving customers cannot measure churn; a panel of eager experts cannot measure the base rate of expert error.
Where it breaks
That the sample selected itself, that the study was underpowered and its null result is meaningless, or that the same source was counted several times through several mouths.
Runs under
Sometimes the answer is to try it on a fewThe survivors, the responders, and the eagerIs six in a quarter a signal or a coincidence?Last year's best performerNumbers that were typed rather than measured
Sources
Heckman (1979). Sample Selection Bias as a Specification Error. Econometrica 47(1), 153. https://doi.org/10.2307/1912352
Groves (2006). Nonresponse Rates and Nonresponse Bias in Household Surveys. Public Opinion Quarterly 70(5), 646-675. https://doi.org/10.1093/poq/nfl033
Cohen (1992). A power primer. Psychological Bulletin 112(1), 155-159. https://doi.org/10.1037/0033-2909.112.1.155
Kohavi, Longbotham, Sommerfield (2009). Controlled experiments on the web: survey and practical guide. Data Mining and Knowledge Discovery 18(1), 140-181. https://doi.org/10.1007/s10618-008-0114-1
Some questions are about whether and some are about when, and the two need different machinery. The hazard is the chance the event happens in the next interval given it has not happened yet; from it you get the timing of churn, default, financing, departure, or entry.
General formh(t) = lim over small dt of P(t <= T < t + dt | T >= t) / dt
S(t) = P(T > t) = exp(-H(t)), where H(t) = integral of h(u) du from 0 to t
Proportional hazards: h(t | X) = h0(t) x exp(X beta)
What drives the answer
The shape of the baseline hazard: rising (wear-out), falling (survivorship), or spiking at known dates (renewals, maturities, contract ends). Averages of time-to-event hide all of this.
Where it breaks
That the censoring was ignored (customers still alive at the end of the data were treated as if they had churned or as if they never will), or that covariates were measured after the event began.
Runs under
Not whether, but when
Sources
Cox (1972). Regression Models and Life-Tables. Journal of the Royal Statistical Society Series B: Statistical Methodology 34(2), 187-202. https://doi.org/10.1111/j.2517-6161.1972.tb00899.x
Kaplan, Meier (1958). Nonparametric Estimation from Incomplete Observations. Journal of the American Statistical Association 53(282), 457-481. https://doi.org/10.1080/01621459.1958.10501452
Describe a system as a set of states and the probabilities of moving between them. Aggregate figures like net retention can look stable while the transitions underneath them have already turned.
General formp(t + 1) = p(t) x P, where P is the matrix of transition probabilities P(i, j) = P(next state j | current state i)
The long-run distribution pi solves pi = pi x P; expected time in a state before leaving is 1 / (1 - P(i, i))
What drives the answer
The transition out of the healthy state, which is small, hard to see in aggregates, and compounds. A two-point rise in the monthly hazard of moving to at-risk rewrites the retention curve a year out.
Where it breaks
That the process has memory (last year's state matters, not just this year's), or that the transition probabilities were estimated from too few observed moves.
Runs under
The healthy average hides the cohort that is leavingThe retention number is stable; the transitions are not
Sources
None verified yet; none invented.
Adoption spreads through innovation and imitation and follows an S-curve, not a straight line or a permanent exponential; a count of events over time has a natural rate, and a cluster well above it is a signal worth quantifying rather than a feeling.
General formBass: adoptions at t = (p + q x F(t)) x (M - cumulative adopters), with p the innovation rate, q the imitation rate, M the ceiling
Exponential: X(t) = X0 x exp(r t), which no real adoption curve follows for long
Poisson: P(N(t) = k) = exp(-lambda t) (lambda t)^k / k!, so k events where lambda t were expected has a computable improbability
What drives the answer
The ceiling M and the imitation rate q, neither of which is visible in the early data that looks exponential. Timing of the inflection is the question, and it is answered from comparable diffusions, not extrapolation.
Where it breaks
That the S-curve was fitted to its own first third, that the ceiling was assumed rather than researched, or that the event count was cherry-picked to a window.
Runs under
Exponential for a while, then notIs six in a quarter a signal or a coincidence?Will it spread, or fizzle
Sources
Bass (1969). A New Product Growth Model for Consumer Durables. Management Science 15(5), 215-227. https://doi.org/10.1287/mnsc.15.5.215
A crowd whose members judge independently is often wiser than its wisest member; a crowd whose members watch each other can be confidently wrong together. The same market can be either, and the difference is whether the later voices added information or copied the earlier ones.
General formCascade: once the public history of prior choices outweighs any one private signal, each later actor follows the history regardless of their own signal, so information stops accumulating
Crowd average error falls like 1/n only for independent errors; with pairwise correlation rho it stops falling at rho x (individual variance)
What drives the answer
Whether the fifth source saw the same thing as the first four or heard about it from them. Consensus built by observation is evidence; consensus built by imitation is one observation repeated.
Where it breaks
That the analyst treated a cascade as convergent evidence, or that the "crowd" was three people and a trade publication.
Runs under
Five sources, or one source heard five timesA committee is not a single mind
Sources
Galton (1907). Vox Populi. Nature 75(1949), 450-451. https://doi.org/10.1038/075450a0
Banerjee (1992). A Simple Model of Herd Behavior. The Quarterly Journal of Economics 107(3), 797-817. https://doi.org/10.2307/2118364
Bikhchandani, Hirshleifer, Welch (1992). A Theory of Fads, Fashion, Custom, and Cultural Change as Informational Cascades. Journal of Political Economy 100(5), 992-1026. https://doi.org/10.1086/261849
Decision makers overweight vivid cases, neglect base rates, treat sunk costs as reasons, anchor on the first number, and feel losses about twice as hard as gains. These are not character flaws to be lectured about; they are predictable, and a research design can be built to catch them.
General formProspect theory: value is v(x) relative to a reference point, concave for gains, convex and steeper for losses, with decision weights w(p) that overweight small probabilities
Base-rate neglect: P(H | E) computed as if P(H) were 1/2 when it is not
Reference class forecasting: estimate from the distribution of outcomes of comparable past cases before adjusting for the case at hand
What drives the answer
The reference point and the base rate, both of which the decision maker usually has not stated and the research should.
Where it breaks
That the bias story is unfalsifiable (any decision can be labeled biased after the fact), so it must be turned into a testable prediction before it is useful.
Runs under
The outside view before the inside viewCustomers, and managers, are not the model
Sources
Tversky, Kahneman (1974). Judgment under Uncertainty: Heuristics and Biases. Science 185(4157), 1124-1131. https://doi.org/10.1126/science.185.4157.1124
Kahneman, Tversky (1973). On the psychology of prediction. Psychological Review 80(4), 237-251. https://doi.org/10.1037/h0034747
Kahneman, Tversky (1979). Prospect Theory: An Analysis of Decision under Risk. Econometrica 47(2), 263. https://doi.org/10.2307/1914185
Kahneman, Knetsch, Thaler (1991). Anomalies: The Endowment Effect, Loss Aversion, and Status Quo Bias. Journal of Economic Perspectives 5(1), 193-206. https://doi.org/10.1257/jep.5.1.193
Arkes, Blumer (1985). The psychology of sunk cost. Organizational Behavior and Human Decision Processes 35(1), 124-140. https://doi.org/10.1016/0749-5978(85)90049-4
Kahneman, Lovallo (1993). Timid Choices and Bold Forecasts: A Cognitive Perspective on Risk Taking. Management Science 39(1), 17-31. https://doi.org/10.1287/mnsc.39.1.17
Flyvbjerg (2006). From Nobel Prize to Project Management: Getting Risks Right. Project Management Journal 37(3), 5-15. https://doi.org/10.1177/875697280603700302
Al-Najjar, Baliga, Besanko (2008). Market forces meet behavioral biases: cost misallocation and irrational pricing. The RAND Journal of Economics 39(1), 214-237. https://doi.org/10.1111/j.0741-6261.2008.00011.x
Some uncertainties have no defensible probability, and people are not indifferent between a known risk and an unknown one. Decision theory since Ellsberg treats that as rational rather than a mistake, and it changes what a careful investor should do when the evidence cannot supply a number.
General formMaxmin: choose the action that maximizes the minimum expected utility over a set of plausible priors
Smooth ambiguity: maximize E over priors of phi(E[U | prior]), with phi concave for an ambiguity-averse decision maker
Ellsberg: preferring the known urn to the unknown one at both ends of a bet is inconsistent with any single probability
What drives the answer
The size of the set of priors, which is another way of saying how much the evidence actually constrains the answer. That is a research question, not a personality trait.
Where it breaks
That ambiguity is being used as an excuse not to estimate something estimable, or that a number was reported where a range of priors was the honest answer.
Runs under
Some uncertainties have no defensible numberThe question has no answer worth buying
Sources
Ellsberg (1961). Risk, Ambiguity, and the Savage Axioms. The Quarterly Journal of Economics 75(4), 643. https://doi.org/10.2307/1884324
Gilboa, Schmeidler (1989). Maxmin expected utility with non-unique prior. Journal of Mathematical Economics 18(2), 141-153. https://doi.org/10.1016/0304-4068(89)90018-9
Klibanoff, Marinacci, Mukerji (2005). A Smooth Model of Decision Making under Ambiguity. Econometrica 73(6), 1849-1892. https://doi.org/10.1111/j.1468-0262.2005.00640.x
Kreps, David M. (1988). Notes on the Theory of Choice. Westview Press. (book)
Savage, Leonard J. (1954). The Foundations of Statistics. Wiley; Dover reprint 1972. (book)
Gilboa, Itzhak (2009). Theory of Decision under Uncertainty. Cambridge University Press. (book)
Who is connected to whom decides how information, adoption, and influence move. Centrality finds the actor through whom the most passes; structural equivalence finds two people who occupy the same position without knowing each other, which is how to find an expert on an industry that has no experts.
General formDegree centrality C(i) = sum over j of A(i, j); eigenvector centrality solves A x = lambda x, so a node is central when connected to central nodes
Structural equivalence distance d(i, j) = sqrt of sum over k of (A(i, k) - A(j, k))^2; small d means similar positions
Informant accuracy: with independent informants, agreement between them estimates each one's competence, and the consensus answer can be recovered without knowing it in advance
What drives the answer
The boundary of the network drawn. A centrality computed on the wrong population finds the most central of the wrong people.
Where it breaks
That the ties were self-reported, that the network was sampled by snowball from one starting point, or that centrality was confused with importance.
Runs under
Who is reliable about whatThe person the information passed throughThe expert on a market that has no expertsSeveral partial accounts, one reconstructed factWho will trade with whomWhat breaks first in the supply chain
Sources
Freeman (1978). Centrality in social networks conceptual clarification. Social Networks 1(3), 215-239. https://doi.org/10.1016/0378-8733(78)90021-7
Bonacich (1987). Power and Centrality: A Family of Measures. American Journal of Sociology 92(5), 1170-1182. https://doi.org/10.1086/228631
Lorrain, White (1971). Structural equivalence of individuals in social networks. The Journal of Mathematical Sociology 1(1), 49-80. https://doi.org/10.1080/0022250X.1971.9989788
Granovetter (1973). The Strength of Weak Ties. American Journal of Sociology 78(6), 1360-1380. https://doi.org/10.1086/225469
Romney, Weller, Batchelder (1986). Culture as Consensus: A Theory of Culture and Informant Accuracy. American Anthropologist 88(2), 313-338. https://doi.org/10.1525/aa.1986.88.2.02a00020
Wasserman, Stanley and Faust, Katherine (1994). Social Network Analysis: Methods and Applications. Cambridge University Press. (book)
Many economic quantities are not bell-shaped: a few customers, products, cities, or partners carry most of the weight, and the average is nearly meaningless. Interaction between two entities scales with their sizes and falls with the distance between them, and distance can be commercial or institutional as well as geographic.
General formPareto: P(X > x) = (x_min / x)^alpha for x at least x_min; the mean is finite only for alpha > 1 and the variance only for alpha > 2
Gravity: F(i, j) = G x M(i)^a x M(j)^b / D(i, j)^c
What drives the answer
The exponent alpha. Below two, the variance is infinite and sample averages never settle down, which is why a thesis about the average customer in a heavy-tailed market is a thesis about nothing.
Where it breaks
That a lognormal was mistaken for a power law (they look alike over two decades of data), or that the tail was extrapolated from a handful of observations.
Runs under
Near misses are data about the disasterIs this market a few customers wearing a crowd?Who will trade with whomThe distribution of the worst caseNumbers that were typed rather than measured
Sources
Gabaix (1999). Zipf's Law for Cities: An Explanation. The Quarterly Journal of Economics 114(3), 739-767. https://doi.org/10.1162/003355399556133
Clauset, Shalizi, Newman (2009). Power-Law Distributions in Empirical Data. SIAM Review 51(4), 661-703. https://doi.org/10.1137/070710111
Mandelbrot (1963). The Variation of Certain Speculative Prices. The Journal of Business 36(4), 394. https://doi.org/10.1086/294632
Anderson, van Wincoop (2003). Gravity with Gravitas: A Solution to the Border Puzzle. American Economic Review 93(1), 170-192. https://doi.org/10.1257/000282803321455214
A population described as one customer base is often several, and the healthy average can hide one segment that is deteriorating fast. Mixture models let the data say how many groups there are and which unit belongs to which.
General formp(x) = sum over k of pi(k) x f(x | theta(k)), with the pi(k) summing to one
Estimated by expectation-maximization: assign units to classes given parameters, re-estimate parameters given assignments, repeat
What drives the answer
The number of classes K, which the data can support only weakly; a segmentation that changes with K is a story, not a finding.
Where it breaks
That the segments were named by the analyst before the data confirmed them, or that a mixture was fitted to noise.
Runs under
The healthy average hides the cohort that is leavingFor whom, not whetherWhich markets behave alike
Sources
Dempster, Laird, Rubin (1977). Maximum Likelihood from Incomplete Data Via the <i>EM</i> Algorithm. Journal of the Royal Statistical Society Series B: Statistical Methodology 39(1), 1-22. https://doi.org/10.1111/j.2517-6161.1977.tb01600.x
When scarce things must go somewhere, writing down the objective and the constraints honestly is most of the work, and the answer often follows exactly. A research program under a budget and a deadline is itself an allocation problem.
General formMaximize f(x) subject to g(i)(x) <= 0 and h(j)(x) = 0; at the optimum the gradient of f is a nonnegative combination of the gradients of the binding constraints (Karush-Kuhn-Tucker)
Linear case: maximize c'x subject to A x <= b, x >= 0; the shadow price of a constraint is the gain from relaxing it by one unit
Research portfolio: choose x(q) in {0, 1} to maximize sum of EVSI(q) x x(q) subject to sum of cost(q) x x(q) <= budget and sum of time(q) x x(q) <= deadline
What drives the answer
Which constraint binds. The shadow price tells you what an extra dollar or day is worth, which is the argument for or against enlarging the engagement.
Where it breaks
That the objective was mis-stated (maximizing information rather than decision value), or that a constraint was invented to force the answer.
Runs under
The one question to answer nextThe research budget is itself an allocationWrite down the constraints and the answer often falls out
Sources
Kantorovich (1960). Mathematical Methods of Organizing and Planning Production. Management Science 6(4), 366-422. https://doi.org/10.1287/mnsc.6.4.366
Some markets clear by matching rather than by price: who gets which slot, seat, organ, or partner. Stable matchings exist and can be computed, and the rules of the match decide who can game it.
General formA matching is stable if no pair would both prefer each other to their assigned partners
Deferred acceptance: proposers apply in order of preference, receivers hold the best offer so far and reject the rest, repeat until no rejections; the result is stable and optimal for the proposing side
What drives the answer
Which side proposes. The side that proposes gets its best stable match; the other side gets its worst, which is why the design of the rules is a distributional decision.
Where it breaks
That participants misreport preferences strategically, or that the market has externalities (couples, complementarities) that break stability.
Runs under
Some markets match rather than price
Sources
Gale, Shapley (1962). College Admissions and the Stability of Marriage. The American Mathematical Monthly 69(1), 9-15. https://doi.org/10.2307/2312726
Roth, Peranson (1999). The Redesign of the Matching Market for American Physicians: Some Engineering Aspects of Economic Design. American Economic Review 89(4), 748-780. https://doi.org/10.1257/aer.89.4.748
An investment is not evaluated alone: what it adds to a portfolio depends on how it moves with what is already held. And any stream of future cash is worth its discounted sum, which is convex in the rate, so the rate decides more than the cash flows do.
General formPortfolio: maximize w'mu - (lambda / 2) x w'Sigma w subject to the weights summing to one; the contribution of an asset depends on its covariance with the portfolio, not its own variance
Present value: PV = sum over t of CF(t) x (1 + r)^-t
What drives the answer
Covariance for the portfolio question and the growth-discount spread for the present value question. Both are usually the least-examined inputs.
Where it breaks
That correlations were estimated in calm periods and assumed in stressed ones, or that the discount rate was chosen to make the number work.
Runs under
Attractive on its own, wrong for this portfolioWhat really drives the portfolio
Sources
Markowitz (1952). Portfolio Selection. The Journal of Finance 7(1), 77-91. https://doi.org/10.1111/j.1540-6261.1952.tb01525.x
Write down how fast something changes as a function of where it is, and the equation tells you where it settles, whether it oscillates, and whether a small push dies out or grows. Most business plans are drawn as straight lines; most business quantities obey equations with feedback and lags, which do not produce straight lines.
General formContinuous: dx/dt = f(x); discrete: x(t + 1) = g(x(t))
A fixed point solves f(x) = 0; it is stable when small departures shrink and unstable when they grow
Logistic growth: dx/dt = r x (1 - x / K), which rises like an exponential and bends toward the ceiling K
A lag between decision and effect turns a stable system into a cycling one
What drives the answer
The sign and size of the feedback near the current state, and the length of the lag. Positive feedback amplifies, negative feedback stabilizes, and a long enough lag on negative feedback makes it overshoot.
Where it breaks
That the system was linearized around a point it has already left, that stability was assumed rather than checked, or that the lag was ignored.
Runs under
Why capacity arrives just as demand leavesCritical mass and lock-inWhy inventories, hiring, and capacity oscillateShare wars and the square lawHow far ahead anyone can see
Sources
Ezekiel (1938). The Cobweb Theorem. The Quarterly Journal of Economics 52(2), 255. https://doi.org/10.2307/1881734
Volterra (1926). Fluctuations in the Abundance of a Species considered Mathematically1. Nature 118(2972), 558-560. https://doi.org/10.1038/118558a0
Anything that spreads by contact, a product, a default, a rumor, a practice, spreads only if each carrier passes it to more than one other on average. Below that threshold it fizzles whatever the early numbers say; above it, it runs until the susceptible pool is used up.
General formSusceptible S, infected I, recovered R: dS/dt = -b S I; dI/dt = b S I - g I; dR/dt = g I
Basic reproduction number R0 = b / g; spread takes off only when R0 x S(0) exceeds one
Final size: the fraction ever infected solves a fixed-point equation and is strictly less than everyone
What drives the answer
R0 and the size of the susceptible pool. Cutting the contact rate b or shrinking S below the threshold ends the spread; nothing else does.
Where it breaks
That mixing is assumed uniform when the real network has hubs, or that the pool of susceptibles was overestimated.
Runs under
Will it spread, or fizzleCritical mass and lock-in
Sources
Kermack, McKendrick (1927). A contribution to the mathematical theory of epidemics. Proceedings of the Royal Society of London. Series A, Containing Papers of a Mathematical and Physical Character 115(772), 700-721. https://doi.org/10.1098/rspa.1927.0118
Bass (1969). A New Product Growth Model for Consumer Durables. Management Science 15(5), 215-227. https://doi.org/10.1287/mnsc.15.5.215
A stock changes by its inflow minus its outflow, and decisions are made on what the stock looks like now, which is the result of decisions made a delay ago. People running such systems overshoot and undershoot in a predictable way, and the pattern is the same in inventories, hiring, and capacity.
General formd(Stock)/dt = inflow(t) - outflow(t)
A decision rule that adjusts the inflow toward a target on a delayed reading of the stock produces oscillation whose amplitude grows with the delay
Bullwhip: the variance of orders rises at each stage upstream of demand
What drives the answer
The delay between an action and its observed effect. A manager who treats a delayed system as immediate keeps acting on stale readings and generates the cycle.
Where it breaks
That the delay was ignored, that the target itself was moving, or that the model was fitted to a period without a shock.
Runs under
Why capacity arrives just as demand leavesWhy inventories, hiring, and capacity oscillate
Sources
Sterman (1989). Modeling Managerial Behavior: Misperceptions of Feedback in a Dynamic Decision Making Experiment. Management Science 35(3), 321-339. https://doi.org/10.1287/mnsc.35.3.321
Lee, Padmanabhan, Whang (1997). Information Distortion in a Supply Chain: The Bullwhip Effect. Management Science 43(4), 546-558. https://doi.org/10.1287/mnsc.43.4.546
Forrester, Jay W. (1958). Industrial Dynamics: A Major Breakthrough for Decision Makers. Harvard Business Review, 36(4), 37-66. (book)
Sterman, John D. (2000). Business Dynamics: Systems Thinking and Modeling for a Complex World. McGraw-Hill. (book)
Whatever is waiting, customers, orders, patients, tickets, work in progress, obeys one identity: the average number in the system equals the arrival rate times the average time each spends there. Near full capacity, waiting time does not rise gently; it explodes.
General formLittle: L = lambda x W, for any stable system, whatever its internal rules
Utilization rho = lambda / mu, arrivals over service capacity
Simplest single-server queue: expected wait grows like rho / (1 - rho), so at ninety percent utilization the wait is nine times the service time
What drives the answer
Utilization close to one and the variability of arrivals and service. Adding a little demand to a nearly full system adds a lot of waiting.
Where it breaks
That arrivals were assumed steady when they cluster, or that the plan runs the system at a utilization where the mathematics guarantees queues.
Runs under
Capacity, waiting, and the cliff at ninety percent
Sources
Little (1961). A Proof for the Queuing Formula: L = lambda W. Operations Research 9(3), 383-387. https://doi.org/10.1287/opre.9.3.383
Every sector buys from others to make what it sells. Write those purchases as a matrix and a shock to one sector, a tariff, an outage, a price spike, can be traced through everyone who buys from it, and everyone who buys from them, to the total effect.
General formOutput x satisfies x = A x + d, with A the matrix of inputs per unit of output and d final demand
Total requirements: x = (I - A)^-1 d; the inverse matrix says how much of every sector's output one unit of final demand ultimately calls for
A shock to sector j propagates down column j of the inverse
What drives the answer
The structure of A: a sector that feeds many others, with few substitutes, propagates a shock far. The first place a supply chain breaks is usually a small sector with a large column.
Where it breaks
That the coefficients are fixed when firms substitute, or that the table is national when the chain is global.
Runs under
What breaks first in the supply chain
Sources
Leontief (1936). Quantitative Input and Output Relations in the Economic Systems of the United States. The Review of Economics and Statistics 18(3), 105. https://doi.org/10.2307/1927837
Some quantities wander with no memory, and their next move is unrelated to their last; others are pulled back toward a level, and the pull is what makes them forecastable. Telling the two apart is the whole question behind most claims of a trend.
General formRandom walk: x(t + 1) = x(t) + e(t); the variance of where it ends grows with time, so any level is reachable and no level is home
Mean reversion: dx = theta (mu - x) dt + sigma dW; the half-life of a departure from mu is ln 2 / theta
Geometric Brownian motion, dx = mu x dt + sigma x dW, is the model underneath option pricing
What drives the answer
Whether theta is zero. With theta at zero the series is a random walk and the trend is an illusion; with theta well above zero, departures from mu are opportunities with a known half-life.
Where it breaks
That theta was estimated on a window too short to tell it from zero, or that a regime change moved mu.
Runs under
Is this trend a trend
Sources
Uhlenbeck, Ornstein (1930). On the Theory of the Brownian Motion. Physical Review 36(5), 823-841. https://doi.org/10.1103/PhysRev.36.823
Black, Scholes (1973). The Pricing of Options and Corporate Liabilities. Journal of Political Economy 81(3), 637-654. https://doi.org/10.1086/260062
The largest loss in a hundred periods has its own distribution, and it is not the one the average describes. Extreme value theory fits the tail from the few largest observations and reports the level exceeded once in a given horizon.
General formThe maximum of many draws converges to one of three families (Gumbel, Frechet, Weibull), indexed by a tail parameter
Return level: the value exceeded on average once in T periods, read from the fitted tail
For exceedances over a high threshold, the excesses follow a generalized Pareto distribution
What drives the answer
The tail index, estimated from a handful of the largest observations, which is why the confidence interval on a hundred-year loss is wide and honest.
Where it breaks
That the fit was extrapolated far beyond the data, that the extremes were treated as independent when they cluster, or that the underlying process changed.
Runs under
The distribution of the worst case
Sources
Fisher, Tippett (1928). Limiting forms of the frequency distribution of the largest or smallest member of a sample. Mathematical Proceedings of the Cambridge Philosophical Society 24(2), 180-190. https://doi.org/10.1017/S0305004100015681
Embrechts, Klüppelberg, Mikosch (1997). Modelling Extremal Events. Springer (book). https://doi.org/10.1007/978-3-642-33483-2
Bak, Tang, Wiesenfeld (1987). Self-organized criticality: an explanation of the 1/f noise. Physical Review Letters 59(4), 381-384. https://doi.org/10.1103/PhysRevLett.59.381
A portfolio, a product line, or a P&L that seems to have twenty moving parts often has two or three that explain most of the movement. Principal components finds them: the directions along which everything moves together.
General formFor covariance matrix Sigma, solve Sigma v = lambda v; the first component v1 is the direction of greatest variance and lambda1 / sum of lambdas is the share it explains
Loadings say how much each original variable moves with each component
What drives the answer
The correlation structure. When a few components explain most of the variance, the twenty exposures are really three, and risk was concentrated where it looked diversified.
Where it breaks
That components are treated as causes when they are only directions, or that the result changed because the variables were rescaled.
Runs under
Which markets behave alikeWhat really drives the portfolio
Sources
Hotelling (1933). Analysis of a complex of statistical variables into principal components. Journal of Educational Psychology 24(6), 417-441. https://doi.org/10.1037/h0071325
Regress one rising series on another and you will find a significant relationship whether or not there is one. Time series need their own discipline: is the series stationary, does it remember its past, and does a relationship survive differencing.
General formy(t) = a + b x(t) + e(t): if both series trend, b is significant by construction and means nothing
Difference the series, or test whether they share a common stochastic trend, before believing b
Autocorrelated errors inflate every t-statistic
What drives the answer
Whether the series are stationary. Two random walks regressed on each other produce a spurious relationship most of the time.
Where it breaks
Nearly every chart that plots one rising line against another and draws a conclusion.
Runs under
Two lines that both go upIs this trend a trend
Sources
Granger, Newbold (1974). Spurious regressions in econometrics. Journal of Econometrics 2(2), 111-120. https://doi.org/10.1016/0304-4076(74)90034-7
Wooldridge, Jeffrey M. Introductory Econometrics: A Modern Approach. Cengage. (book)
Nobody in an industry has to be rational for the industry to converge on a strategy: practices that do better than average grow, practices that do worse shrink, and a strategy that cannot be invaded by a rare alternative is where the population ends up.
General formReplicator: dx(i)/dt = x(i) x (f(i)(x) - fbar(x)), the share of strategy i grows when its payoff exceeds the population average
An evolutionarily stable strategy earns at least as much against itself as any invader does, and strictly more against the invader when tied
What drives the answer
Relative payoff, not absolute. A strategy that is merely good can still lose share to one that is better in the current mix, and the mix is what changes.
Where it breaks
That the payoffs were treated as fixed when they shift with the population, or that entry and mutation were ignored.
Runs under
Which practices survive in an industry
Sources
Maynard Smith, Price (1973). The Logic of Animal Conflict. Nature 246(5427), 15-18. https://doi.org/10.1038/246015a0
Taylor, Jonker (1978). Evolutionary stable strategies and game dynamics. Mathematical Biosciences 40(1-2), 145-156. https://doi.org/10.1016/0025-5564(78)90077-9
Given an edge, there is a fraction of capital that maximizes long-run growth, and betting more than it reduces growth while raising the chance of ruin. The penalty is asymmetric: too little costs you a little, too much can cost you everything.
General formFor a bet won with probability p at odds b: f* = p - (1 - p) / b
Growth rate g(f) = p ln(1 + b f) + (1 - p) ln(1 - f), maximized at f* and negative beyond about 2 f*
Continuous version: f* = (mu - r) / sigma^2, the excess return over the variance
What drives the answer
The edge and the variance, both estimated with error. Because the edge is usually overestimated, fractional Kelly, half or less, is the practitioner's rule.
Where it breaks
That the edge was estimated on too little data, that the bets were correlated, or that the horizon was too short for the long-run argument to apply.
Runs under
How much of the bet to take
Sources
Kelly (1956). A New Interpretation of Information Rate. Bell System Technical Journal 35(4), 917-926. https://doi.org/10.1002/j.1538-7305.1956.tb03809.x
Thorp (1969). Optimal Gambling Systems for Favorable Games. Revue de l'Institut International de Statistique / Review of the International Statistical Institute 37(3), 273. https://doi.org/10.2307/1402118
Merton (1969). Lifetime Portfolio Selection under Uncertainty: The Continuous-Time Case. The Review of Economics and Statistics 51(3), 247. https://doi.org/10.2307/1926560
An expert asked for a number gives an anchored, overconfident one. Asked the right way, in a fixed sequence that starts from the extremes and checks for consistency, the same expert gives a range that is worth something. The protocol is the instrument.
General formAsk for fractiles: the value the expert thinks has a ten percent chance of being exceeded, then ninety, then fifty, before any central estimate
Alternate fixed-value questions (how likely is more than X) with fixed-probability questions (what value has a one in four chance)
Calibrate on quantities with known answers; widen ranges that prove too narrow
What drives the answer
The order of the questions. Starting from a central estimate anchors everything that follows; starting from the extremes produces ranges roughly twice as wide and far better calibrated.
Where it breaks
Anchoring on the first number mentioned, an expert with a stake in the answer, or a range reported without the protocol that produced it.
Runs under
How to ask an expert for a number
Sources
Spetzler, Stael von Holstein (1975). Probability Encoding in Decision Analysis. Management Science 22(3), 340-358. https://doi.org/10.1287/mnsc.22.3.340
Tversky, Kahneman (1974). Judgment under Uncertainty: Heuristics and Biases. Science 185(4157), 1124-1131. https://doi.org/10.1126/science.185.4157.1124
Howard, Ronald A. and Abbas, Ali E. (2016). Foundations of Decision Analysis. Pearson. (book)
A model that predicts well can be a black box, and a model that explains must identify a cause. The two are graded on different tests and confused constantly: a predictor that is right nine times in ten says nothing about what happens if you intervene.
General formPrediction: minimize out-of-sample error, measured by holding data back and checking; any method is fair
Explanation: identify a parameter that answers what happens to Y if X is changed, which requires an identifying assumption, not a better fit
A feature that predicts an outcome is not thereby a lever on it
What drives the answer
The question being asked. Which customers will churn is a prediction; what would reduce churn is an explanation, and the same data answer them differently.
Where it breaks
Using a predictor to decide an intervention, or demanding a causal story from a model that only needed to forecast.
Runs under
Predict, or explain
Sources
Breiman (2001). Statistical Modeling: The Two Cultures. Statistical Science 16(3). https://doi.org/10.1214/ss/1009213726
Athey, Imbens (2016). Recursive partitioning for heterogeneous causal effects. Proceedings of the National Academy of Sciences 113(27), 7353-7360. https://doi.org/10.1073/pnas.1510489113
Committees, boards, and investment committees are not single minds. The rule they vote by, the order items come up, and who controls the agenda change the outcome; and a group of imperfect decision makers can be far better or far worse than its members depending on whether their errors are independent.
General form
Arrow: no rule turning individual rankings into a group ranking satisfies all of a short list of reasonable conditions at onceMedian voter: under single-peaked preferences and majority rule, the outcome is the median member's ideal point
Condorcet: n independent members each right with probability p > 1/2 are right by majority with probability rising toward 1 in n; correlated members are not
What drives the answer
Independence of judgment and control of the agenda. A committee that hears the same expert before voting is one mind with several votes.
Where it breaks
That the aggregation rule was chosen by whoever wanted the result, or that the votes were not independent.
Runs under
A committee is not a single mindWhoever sets the agenda sets the answerGovernance is a voting game
Sources
Arrow (1950). A Difficulty in the Concept of Social Welfare. Journal of Political Economy 58(4), 328-346. https://doi.org/10.1086/256963
Black (1948). On the Rationale of Group Decision-making. Journal of Political Economy 56(1), 23-34. https://doi.org/10.1086/256633
Austen-Smith, Banks (1996). Information Aggregation, Rationality, and the Condorcet Jury Theorem. American Political Science Review 90(1), 34-45. https://doi.org/10.2307/2082796
Romer, Rosenthal (1978). Political resource allocation, controlled agendas, and the status quo. Public Choice 33(4), 27-43. https://doi.org/10.1007/bf03187594