Is Lists “Voltage Effect” Flawed?

This post was co-written with Quality Management and Deming expert, Allen Scott who also used information gained in personal communication with quality management and statistical expert Don Wheeler, PhD.

I have tremendous respect for John List, PhD and many of his recommendations. I have read a lot of his work and heard him in presentations. I have also listened to him on Freakonomics. After hearing a review of his new book on Freakonomics, “Why Do Most Ideas Fail to Scale?” (the book is “The Voltage Effect: How to make good ideas great and great ideas scale“), I got a copy and read it.

Good Ideas

There are many great ideas contained, most specifically about why ventures may not scale. For instance, he emphasizes it is important not to be misled by false positives. This is when good results happen but it is not with a representative sample, thus falsely showing the idea may be successfully scaled. He documents how this happened with Nancy Reagan’s “Just Say No” anti-drug campaign, explaining why it was not as successful or even validated. He also explained the importance of knowing the intended audience. Here he cited an example where McDonalds had sampled devoted McDonald’s customers to test the new “Arch Deluxe” rather than testing it with the typical customer. He explained this was why the “Arch Deluxe” was not a successful new product and why it was important to use true customers, not devotees.

He also explains that when scaling it is important to know if it is the chef, the leader, or the ingredients that make the product successful. As he explains, it is easier to scale ingredients than people. Here he also explained the importance of spillover effects. As he made clear, small issues become more prominent when ideas are scaled. This can be documented from “General Equilibrium Effects” based on the theory. This theory explains how expectations can be disrupted because when one area changes, all areas change to adapt to that change. As an example he explained how when Uber raised driver salaries in hopes of helping them earn more take-home pay, more drivers then drove for Uber. The increase in salary increased the number of drivers and this led to less rides being given per driver. This meant the drivers did not earn the desired raise. This was a great example of General Equilibrium Effects.

He also explained that intervention spillovers can be positive or negative. As a positive example, he explained the spillover of Herd Immunity happens when many people in the community are vaccinated. He also warned that if costs were too high, it cannot scale. While many ideas were good, as I kept reading something was nagging at me and something seemed off. It was not until I got to Chapter 7, and when he began to focus on how to scale, that I realized what was nagging me.

Fatal Flaw?

What was nagging me was that the book was about improving the parts without accounting for the whole system and the dynamic interactions or “Systems Appreciation” in Deming’s Profound Knowledge. To compound his inattention to the whole system, he also failed to account for environmental impacts of any venture. It was as if he equated the impact on the environment at a cost of 0. The environment must be accounted for because the environment is an asset upon which every venture and all of us are dependent. Treating nature with no value encourages its misuse. Ventures should operate such that it supports regeneration because this can be the only way to ensure true value and improvement as it supports ongoing viability and profitability for everyone and everything.

System Improvement

As Dr. Ackoff explains, (see this powerful presentation titled, “If Russ Ackoff Had Given a TED Talk“) a system is not a sum of its parts but a product of its interactions. Further he explains if improvement of a system is done by improving the parts taken separately, you can absolutely be sure the performance of the whole will not be improved. This is what I believe is the fatal flaw in List’s book. He discusses methods to improve the parts without improving the system. In Dr. Deming’s terms, he does not have an “Appreciation for a System”.

One example he discussed in the book was about investing on marginal returns or the area that had the biggest return on the last dollar spent. This may work sometimes, however it mistakenly encourages management by results or managing by watching the scoreboard rather than continually improving the process. Managing by results will result in higher and higher variance, higher costs and lower profits. (see Red Bead Experiment) In the book Dr. List even relied on a faulty example, explaining hiring more people did not produce the same returns because the new group was not as productive. This mistakenly placed responsibility on the people, rather than the system from which results are generated.

Understanding Variation – Contribution from Allen Scott which also cited information obtained in personal communication with quality expert Don Wheeler, PhD

In The Science of Using Science: Towards an Understanding of the Threats to Scaling Experiments, John List, et. al. states, “Policymakers are increasingly turning to insights gained from the experimental method as a means of informing public policies.” They argue that knowing when evidence becomes actionable requires information about the population and the situation. Further they suggest this type of information is vital to knowing if scaling will work. 

Their writing seems to suggest more than experimental methods are necessary. Their concern, relying only on the scientific method can lead to a vast waste of resources, a missed opportunity to improve peoples lives, and a diminution in the public’s trust in the scientific method’s ability to contribute to policy making.

Actionable Evidence

Dr. Walter A. Shewhart in 1924 at Bell Labs developed process behavior charts to determine when evidence becomes actionable. These charts could identify appropriate statistical evidence by separating the noise from the signal. These charts provided an observational improvement method that plotted data over time. 

In Understanding Variation: The Key to Managing Chaos, by Dr. Donald J. Wheeler he documents that process behavior charts work and have been thoroughly proven. Further, it seems hard sciences can use the experimental methods and hold many variables constant, however social sciences must deal with unknown cause and effect relationships. These unknowns make the decision to scale problematic without more information. In such an environment, observational studies are needed rather than experiments.  If a test program is broad enough and predictable, reliable evidence will be gained about scaling. If however the evidence is localized and unpredictable, the evidence will be problematic. 

As explained earlier, List suggests this in his book, “Voltage Effect”, when he explains misleading evidence and false positives lead to misinformed choices to scale. As he explains, observational studies can be better than experiments when deciding to scale if they are representative enough to be predictable.

The problem as I see it is the assumption that we will know all of the important factors.  Experimentation cannot identify the unknown factors, only observation does this.

Don Wheeler, PhD

Real-Life Example

For example, despite experimental evidence about the value of Hormone Replacement Therapy (HRT), observation studies of over 8000 women over ten years showed that post-menopausal hormone replacement therapy changed the likelihood of heart-attack from 2% to 3%. This study made it clear that HRT benefits did not outweigh the risks.

For more on experimental studies versus observational studies, see the recent Quality Digest article by Dr. Wheeler: Different Approaches to Process Improvement Does your approach do what you need? We also recommend Dr. Greger’s review of this topic in his linked short Nutritional Facts.org July 4, 2022 vlog post, Observational Studies Show Similar Results to Randomized Trials.

Quitting Shouldn’t Exist…

Another concern I had with List’s book related to quitting. Dr. List emphasizes the need to get better at quitting and the need to quit. I am not sure why he chose to describe it as quitting. He was equating quitting with failure, but failure doesn’t exist (see Failure Doesn’t Exist…). The drive to succeed and do well for most entrepreneurs would stay the same, thus the aim would not be consistent with quitting. This is why the idea of quitting is a confusing reference. For instance, using his personal example, he explained that he chose not to make a difference by being a professional golfer, but as a professor in academics. Thus by his own admission, he did not quit wanting to make a difference, he just pivoted.

To me pivoting, a term used often by the NSF iCorps program and others, is a better way to encourage entrepreneurs and is a method to help them succeed and scale. A pivot should occur when an entrepreneur discovers, after researching the idea, the market and customers, that the idea is a no-go, or not a good idea to scale. After discovering the idea may be problematic, it is recommended they pivot to a variant or alternative. From my perspective, this is better terminology than quitting and allows the entrepreneur to carry forward the many assets and skills gained toward the pivoted aim of the venture.

While I do encourage you to read Lists book, “The Voltage Effect: How to make good ideas great and great ideas scale“, please keep in mind, any venture must contribute to systemic improvement. I am concerned List did not adequately account for the system’s impact, especially when parts are maximized as he suggests. This was a bit confusing since he did discuss “General Equilibrium Effects” and then ignored it throughout the text.

As Russ Ackoff makes clear, simply improving the parts cannot improve the system. As we all seek to make our contribution toward comprehensive improvements, it is recommended we focus on creating net-positive, pervasive, reciprocal, selfish, selfless, synergistic interactions so everyone and everything benefits. Please share how you practice paneugenesis.

BeWellr,

Craig M. Becker, PhD

Be selfish, selfless, & synergistic so everyone and everything benefits!

#SelfishSelflessSynergy

Please share your thoughts and questions below.

Contact me: BeWellr@bewellr

Sources: (added to keep me straight right now!) remove/change later

1. https://bfi.uchicago.edu/working-paper/the-science-of-using-science-towards-an-understanding-of-the-threats-to-scaling-experiments/
2. Wheeler, D. J. (1993). Understanding variation: The key to managing chaos. Knoxville, Tenn: SPC Press.