Where Do Great Ideas Come From?

The great business theorist Peter Drucker didn’t think all that much of ideas. “Ideas are cheap and abundant,” the management expert said, “What is of value is the effective placement of those ideas into situations that develop into action.” 

Drucker’s position is a common one. Across academia and industry, plenty of fine thinkers have made equivalent statements, arguing that real value resides in effective implementation, not ideation. The prolific executive brings most value to the world, this thesis goes, rather than the unproductive theorist. 

Though there’s wisdom in Drucker’s words, he is wrong in his assessment of ideas. Ideas are not cheap, not the valueless things the Austro-American characterizes them as. Certainly, ideas cannot impact society without “effective placement,” but there would be nothing to “place” without them. They are the seed of all progress, the beginning of every great invention. The fibers of our clothing, refrigeration of our food, design of our medicines, and architecture of our computer chips all began as ideas, or more accurately, a series of ideas stacked on top of each other, finely balanced. 

Nor are ideas abundant. Indeed, there is evidence they are growing rarer by the year. A Stanford University study titled “Are Ideas Getting Harder to Find?” detailed how “research effort is rising substantially while research productivity is declining sharply.” For example, maintaining Moore’s Law – which predicts the doubling of transistors on a computer chip every two years – has required massive increases in research efforts. Compared to the 1970s, 18x more researchers are needed to continue its trajectory, with productivity slipping. The study documents similar slumps in other industries like agriculture and healthcare. 

The decrement of ideas makes their study all the more important. Rather than diminishing them, we should seek to understand better how, where, and by who they are created. What incentives encourage originality? Who should you hire to boost an organization’s innovation? And how do you harness the abilities of a collective?

We surveyed more than 30 academic studies to answer these questions, selecting our seven favorite findings to distill. This piece is part of our “Big Questions” series. As with our previous edition, “Who Becomes an Entrepreneur?” it does not attempt to be a comprehensive literature review of the field, nor a monolithic one. Academia is always in conversation with itself; arguments, conflicts, and gradations should be expected. We approach these findings with curiosity, humility, and the hope that they spark new ideas and open fresh perspectives for us all. 

1. Good incentives

Incentives are the friend of the wise man and enemy of the fool. As the story goes, The British Government in India played the fool. In an attempt to rid Delhi of venomous snakes, the colonizers implemented a bounty program, rewarding anyone that brought in a dead cobra. 

Do you see what’s wrong with this scheme? While the snake population fell in the short term, enterprising locals quickly found a loophole in the British plan. Rather than killing the remaining snakes – an uncertain enterprise with diminishing returns – they could simply breed more snakes, delivering their bodies for a steady income. What began as an attempt to reduce the cobra population boosted it.

Companies that wish to avoid the dreaded “Cobra Effect” must be thoughtful about the incentives they implement. This is perhaps particularly true in the attempt to foster innovation. Creativity is a strange and often fragile flower that grows best in certain environments. 

“Incentives and Creativity,” published in 2009, probes the impact incentives have on creativity in the life sciences. The authors focus on “investigators” that received funding from two different groups: the National Institute of Health (NIH) and Howard Hughes Medical Institute (HHMI). 

Though both provide financing to research scientists, the conditions and expectations vary widely between the two entities. The NIH has short review cycles and a strong emphasis on concrete deliverables. Meanwhile, the HHMI is minded toward the long-term, with a higher tolerance for failure and a preference for experimentation. Unlike the NIH, it funds “people, not projects.”

Authors Azoulay, Zivin, and Manso sought to understand how the incentives of these different programs might impact innovation. To create a proxy for creative output, the researchers combined various factors, including the sum and distribution of citations, elections to the National Academy of Sciences, and the number of students trained in the subjects’ lab that won a major award.

Which funding source encouraged more creativity? It may not surprise you to hear that recipients of HHMI grants were found to be considerably more innovative. They published more, higher-quality papers over the course of their careers. Whereas just 4.1% of the NIH group was named to the National Academy of Sciences, roughly 33% of HHMI recipients got the nod. Meanwhile, HHMI grantees trained 83 winners of major awards (1.13 per scientist), compared to 90 among the control group (0.23 per scientist). 

“Tolerance for Failure and Corporate Innovation,” published the same year, illustrates that this phenomenon is not limited to research environments. Authors Tian and Wang found that IPO’ing companies backed by “failure-tolerant” venture capitalists are better innovators, producing more patents. 

To build a creative environment, incentivize long-term thinking and experimentation. Making room for failure opens the door to much greater success. 

2. Outsiders

Sometimes, the source of the best idea comes from the person that knows least about a given topic. The novice is often able to find solutions the expert is blind to. It’s a compelling theory that even Steve Jobs was fond of: “There’s a phrase in Buddhism, ‘beginner’s mind.’ It’s wonderful to have a beginner’s mind.”

Is it true? Do those with less knowledge about a subject area have an advantage when developing creative solutions? 

A 2014 study published in Management Science set out to examine such questions. In particular, authors Franke, Poetz, and Schreir sought to determine how outsiders in analogous fields might fare in generating ideas for a problem beyond their direct expertise. The researchers gathered 213 subjects across three functional areas: carpentry, roofing, and inline skating. These may sound like entirely disconnected spaces, but their use of protective equipment unites them. 

To test the novelty of their problem-solving, the Copenhagen Business School and Vienna University team asked these three groups of participants to come up with ideas to solve safety issues for all three areas, including their own. Carpenters, for example, were prompted with safety concerns in carpentry, roofing, and inline skating and asked to come up with potential solutions for each. Roofers and inline skaters went through the same process. All answers were then reviewed by market experts, who were asked to judge them blindly by their novelty and usefulness. 

The team found that those from “analogous markets” were significantly better at generating original ideas. Roofers had more original concepts for carpenters and inline skaters than for themselves, for example. Interestingly, the greater the distance the participant had from the target problem, the more novel their ideas. Inline skaters have less overlap with carpenters than roofers do, and yet they were the ones that provided the most original solutions.

Novelty does come at a cost, however. Though outsiders’ solutions might be more creative, experts found them less immediately useful. This makes sense – without sufficient context on the problem space, a participant from an analogous market may suggest infeasible solutions or those that require significant tweaking.

Nevertheless, there’s an interesting lesson in Franke’s study. When faced with a problem that requires novelty, don’t simply ask your peers or industry experts: find someone from an analogous market that brings a beginner’s mind to the problem. 

3. Collaboration

Renowned anthropologist Margaret Mead described her craft as the practice of “record[ing] in astonishment and wonder that which one would not have been able to guess.”

For the last thirty years, Kevin Dunbar has operated as an anthropologist of ideas. Across several studies, the University of Maryland professor has studied how scientific laboratories function and the processes that lead to breakthroughs. In the process, he has recorded a great deal that one would not have been able to guess. 

Dunbar’s work has surfaced fascinating discoveries. A 2009 Wired article, for example, details how the academic’s research illustrates the sheer unpredictability of the scientific process. Thomas Kuhn may have characterized science as a process in which “everything but the most esoteric detail of the result is known in advance,” but Dunbar found the opposite true. By his accounting, more than 50% of the data research labs collected was unexpected. 

Though significant, Dunbar’s contribution to the social conditions that foster innovation has made his name. Contrary to the stereotype that breakthroughs occur thanks to the spontaneous brilliance of a solitary genius, Dunbar found they happened during weekly lab meetings. During these sessions, scientists present data to their colleagues, receiving questions and critiques. This collaborative process was particularly effective in generating new, promising ideas. As Steven Johnson summarizes in his excellent book, Where Do Good Ideas Come From: “The ground zero of innovation was not at the microscope. It was at the conference table.”

Answering questions may be especially important to ideation. In his 1995 study, “How Scientists Really Reason,” Dunbar highlights how prompting a colleague to reframe their work at a different “level” can unlock new reasoning: 

For example, a scientist may be conducting a series of experiments aimed at discovering the mechanism by which a certain type of lymphocyte binds to a certain type of cell. The scientist is concerned with the experimental details and particular components of the mechanism. Other scientists may ask this scientist a question about how the lymphocyte got there in the first place, rather than how does it bind. This new question forces the scientist to reorganize his knowledge, and when he does this, his original question is also answered. 

In simple terms, finding ways to pull your intellectual counterparty out of a narrow viewpoint can help them reconfigure the information available. Ultimately, Dunbar’s work demonstrates the importance of collaboration to creativity. 

4. Superstars

Dunbar’s work demonstrates the importance of the group. But exceptional individuals still play a critical role in driving innovation. Pierre Azoulay’s study, “Superstar Extinction,” explores the connection between outstanding scientists and creative production. 

Published in 2008, the work examines the effect of a “superstar” scientist’s premature death on their collaborators’ productivity. To do so, Azoulay and co-authors Zivin and Wang filtered a sample of more than 10,349 elite scientists to focus on 112 that died unexpectedly between 1979 and 2003. Researchers then mapped each superstar’s professional relationships, creating more than 5,200 pairings between “collaborators” and “stars.”

To measure creative output, researchers focused on the number of articles collaborators co-authored after a superstar’s death, factoring in the impact of the journals in which they featured. The study found a “sizable and significant” decrease of 8.8% in output by this measure.

The decline was worse among those that intellectually overlapped with a superstar more closely: 12.2%. Those that worked with especially generative superstars (as measured by the number of citations their work received) also suffered steeper drops. Critically, this wasn’t the case for collaborators of superstars with extraordinary research funding. As the study remarks, “It is the star’s citation impact that matters in shaping collaborators’ post-extinction outcomes, rather than his/her control over a funding empire.” 

While collective reasoning is vital to ideation, exceptional individuals may play an outsized role. When an “irreplaceable source of ideas” dies, that loss is felt.

5. Brokers

Every organization is a network. People act as nodes, with information and capital zipping from one person to another, proliferating across departments. 

Where in that network do the best ideas come from? And where should pragmatic operators look to position themselves?

Ronald Burt’s 2004 paper, “Structural Holes and Good Ideas,” represents a fascinating contribution to the subject. The University of Chicago academic studied the managerial class at one of America’s largest electronics companies to understand where novel ideas come from within an organization.

Before discussing Burt’s findings, however, a little definition is needed. What is a “structural hole?” And who is a “broker?”

A structural hole is a gap in the network structure of an organization or market. When there’s little to no connectivity between the accounting and product departments or between an East Coast and West Coast division, they are separated by structural holes. Brokers are those individuals that serve as connectors between otherwise untethered subgroups. This person – whether inside or outside of an organization – acts as a go-between, a conduit. 

A diagram from the University of Chicago academic’s work illustrates both concepts. Figure 1 shows the position of two different organizational leaders: Robert and James. Who is better positioned? While James receives information from across Group B, Robert integrates perspectives from A, B, C, and D. He brokers the structural holes between these groups and, at least from a network perspective, should be better placed to deliver new perspectives to each group. In Burt’s parlance, Robert has a “vision advantage” – seeing more of an organization. 

To understand whether the Roberts of the world do, indeed, disproportionately contribute good ideas, Burt analyzed 673 managers at a large electronics business. As well as considering factors like gender, geographic location, race, age, and marital status, Burt used a cross-company questionnaire to determine “network constraint.” This measure captured the size, density, and hierarchy of a manager’s informational sources, proxying how much of a broker across structural holes they were. 

Participants were asked to provide ideas to improve the company’s supply chain, which senior management then scored. Burt found that managers with less constrained networks provided better quality ideas than those with constrained networks. This pattern holds even when looking at managers in the “top-ranks” – a high-level manager with a redundant informational network delivered weak ideas. Interestingly, those spanning structural holes were much more likely to discuss their ideas, allowing them to be debated, improved, and implemented. 

Finally, Burt found that those with lower network constraint scores were more likely to receive positive performance reviews, promotions, and higher salaries. Though traditional corporate cultures often overly encourage operators to focus on narrow execution, brokers are the ones that reap the rewards. If you want to increase the quality of your ideas, find a structural hole, and make it your own. 

6. Intrinsic motivation

Our motivation for taking a job may influence our performance. Published by the National Bureau of Economic Research (NBER), “What Makes Them Tick?” analyzes the impact of extrinsic and intrinsic motivations on effort and creative output in R&D scientists. 

The study was conducted in 2008 by Henry Sauermann and Wesley M. Cohen and is notable for its sample size. The authors relied on information from more than 11,000 R&D professionals across industries.

To assess the impact of particular motives on effort and creative performance, Sauermann and Cohen assessed a ranked list that subjects made, outlining why they chose their current job. Choices included extrinsic motivations like high salaries, special perks, and research funding. It also included intrinsic motivators like independence, responsibility, or the desire for an intellectual challenge. They subsequently analyzed this information alongside the number of hours worked per week (a proxy for effort) and the number of patents issued in the past five years (a proxy for creative production). 

The authors found that effort and innovative production were much more correlated with intrinsic motivators. R&D scientists who chose their positions because they provided more independence and fulfilled their curiosity worked more hours and had more patents to show for it. 

External motivations had mixed impacts. Scientists motivated by a high salary worked fewer hours, albeit only in one subgroup of the sample. Subjects motivated by job security tended to have lower creative output. 

If you want to build a team of high effort and innovativeness, offering a genuine intellectual challenge may be more important than upping the position’s salary. 

7. Elders

We started this piece by reporting that ideas are getting harder to find. But why? What makes innovation harder today than it was fifty years ago? 

Northwestern economist Benjamin Jones offers a compelling explanation in his paper, “The Burden of Knowledge and the ‘Death of the Renaissance Man.” To understand the “burden” Jones refers to, you must first agree with two simple claims: “First, innovators are not born at the frontier of knowledge; rather, they must initially undertake significant education. Second, the frontier of knowledge shifts over time.”

If you agree with these observations, the burden becomes clear. As the frontier of knowledge moves forward, it takes progressively more and more time for new generations to reach it. Simply put: no baby is born with an understanding of machine learning. If they wished to innovate at the frontier of artificial intelligence, they would have to spend a considerable portion of their life learning enough about math, electronics, computer science, statistics, neural networks, and more simply to get to the starting line. 

Jones explores one impact of the growing knowledge burden in a second work, “Age and Great Invention.” The Kellogg professor finds that the age at which notable innovations are achieved is significantly increasing. Over the 20th century, it has risen by 6 years, largely due to the extended education innovators now need to undertake. In 1900, the peak ability to generate a great invention arrived at approximately 30 years old; by 2000, it was nearly 40.

While that may not sound like much at first glance, it profoundly impacts civilization-level productivity. Because inventors don’t make up for their late start with greater productivity in middle age, their “life-cyclical innovation potential” has declined by 30%. Our greatest thinkers simply have less time to do their most meaningful work. 

While Jones’ findings are concerning from a societal perspective, they may offer some individual succor. If you have yet to develop your great invention, time may be on your side.