Data-Driven Innovation: What Is It?

        Innovation addresses evolving societal needs and drives business and economic growth. However, innovation is never easy or assured, because of the innately uncertain nature of the innovation process. The aim of innovation is to create new products, services and systems, new demands, and new markets. In turn, the creativeness and newness that define innovation also result in uncertainty in the process of pursuing innovation. It is often ambiguous what to innovate and how to innovate. Uncertainty surrounds the innovation process and challenges all innovators and the companies and governments that aspire to innovate.

        Meanwhile, the digitalization of our work and life have been generating growing data about technologies, processes, users, markets, etc., daily, hourly, and second-by-second. If we can properly mine, analyze and make sense of such data, particularly the naturally and often passively generated digital footprint data, we will be able to make innovators more inspired and innovation decisions more informed, and thus, reduce the uncertainty and increase the creativity in the innovation process. This is the “DDI” process.

        Herein, I define DDI as the process of innovating that draws information and inspiration from big data (of users, innovators, stakeholders, science and technologies, processes, environments, etc.). The data-driven information and inspiration can reduce uncertainty and enhance creativity in the innovation process (see how it works in Section III). Data science, machine learning, artificial intelligence (AI), and computing technologies further empower DDI.

        The term “DDI” has appeared in both lay press and a small number of reports and papers in the past few years but was given ambiguous or different meanings [1], [2]. The concept is often confused with the innovative products and services whose core features and value creation for users are based on data (which we will call “data-based innovation (DBI)”) [3]–[5], or with the operations, which mine and analyze data to optimize efficiency and accuracy in delivering predefined objectives (which we will call “data-driven optimization (DDO)”) [6]–[8]. Such ambiguity might limit or misguide the actions and efforts aimed to make the innovation processes more data-driven, informed and inspired.

        Herein, this article aims to crystally define DDI as a process and distinguish it from other data-related paradigms. I will first start with a few real-world examples of DDI and theoretical framing of what it entails and how it creates values. On this basis, I present a taxonomy of different data-driven approaches according to the values they create for different exploratory and creative actions in the innovation process. Finally, I discuss relevant technological, organizational, and policy considerations to embrace the future of innovation processes that are data-driven and AI-empowered.

        One example of DDI is when intuit created Quickbooks, an accounting management software for small businesses. The idea was conceived when the team analyzed the digital footprint data of users for their earlier product Quicken, a personal financial management software, and found many customers used Quicken in workplaces for business accounting. Then, the team developed a new product specifically for small business accounting, which was the Quickbooks [9]. In this case, the inspiration for innovation was obtained from rather simple user digital footprint analytics. In fact, DDI can be further empowered by advanced big data computation, machine learning, and artificial intelligence technologies.

        For instance, IBM conducted a large-scale semantic analysis of millions of public text documents about diverse materials to identify three natural materials, which have never been used in a battery and are unknown by battery engineers and combined them in a new battery design that outperforms state-of-the-art lithium-ion batteries [10]. In another example, atomwise, a San Francisco-based drug discovery startup, trained neural networks on large-scale prior experimental data to predict the performance of new drugs to accelerate drug innovation [11]. Since the outbreak of COVID-19, researchers around the world have been racing to develop medical cures and vaccines. Many have taken a data-driven approach to mine large databases of compounds, peptides, and epitopes to train machine learning models to discover, generate and evaluate vaccines, and candidate cure medicines [12].

        Enormous data that can be mined to aid innovators are publicly available and accessible everywhere in our digitalized life. For instance, e-commerce websites host massive publicly accessible data about products and their design specifications, users, and their opinions about products. Researchers have mined data from e-commerce websites to train neural networks that can learn the latent needs of consumers, based on their comments on consumer products and translate such into technical design requirements to inform product design [13]. In fact, mining and analyzing consumers’ behavior and opinion data from e-commerce sites, social media, and other online or digital spaces to inform new product development has been a common practice and well-studied in the marketing literature.

        In the meantime, the digital footprints of innovators and innovation activities may also inform and inspire innovators and companies for innovation. For example, the public patent database contains enormous unstructured multimodal descriptions of prior technologies across domains and digital footprints of inventors and companies regarding their innovation behaviors and performances. There have been efforts to develop patent data-driven expert systems to inform companies of latent innovation opportunities specifically for them [14] and retrieve and prompt design stimuli from the patent database to inspire innovators for generating new design ideas [15], as well as to train neural networks with patent data to predictively evaluate the value of new inventions [16].

        The US Chamber of Commerce Foundation [2] highlighted four types of open big data that can be explored to generate value across sectors. They include user-generated data (often online), industrial and sensor data (from GPS, mobiles, machines/equipment, and public infrastructure), business or enterprise data (such as inventories and transactions), and public data (curated or generated by government agencies, universities, or nonprofit organizations, such as patents, papers, reports). Such open and public data sources, as well as the proprietary data of companies and government agencies, may enable enormous opportunities for DDI.

3.1 Dissecting the Innovation Process

        These aforenoted examples demonstrate various ways or manners in which data-driven approaches can inspire, inform, and augment a variety of exploratory or creative but uncertain actions in the innovation process. Here, we further synthesize them to present a taxonomy of different data-driven approaches according to the different values they create for the complementary actions in typical innovation processes. To aid the synthesis, we focus on four core innovation process actions that shape the opportunity space and the design space essential for innovation (see Fig. 1).

        For opportunity discovery, the traditional approaches involve interviewing, surveying, or observing people to develop empathy and understand their needs, problems, frustrations, and challenges. By taking a data-driven approach, innovators may mine and analyze the users’ digital footprints of using or commenting on existing products or services to gain insights on the unmet needs (e.g., Quicken [9]). Innovators can also explore the digital footprints of their own in the innovation process to gain insights on the unexplored applications of their technologies as innovation opportunities (e.g., InnoGPS [14]). Opportunity discovery is an exploratory action by nature and thus best supported by unsupervised machine learning techniques. 

        For opportunity evaluation, innovators often rely on human expertise to translate the insights about people and needs gained from the discovery phase into specific design requirements that can direct design generation. In contrast, a data-driven approach can better cope with the opportunity uncertainty especially when the latent opportunity space is large and complex, by making innovators more informed when they evaluate and select opportunities for further pursuits [23]. For instance, innovators may utilize the digital footprint data of user preferences together with the design specifications of the products they purchased, used, or liked (e.g., from e-commerce sites) to train artificial neural networks that relate the former to the latter. Then such trained artificial neural networks can be used to automatically translate newly discovered user needs to specific design requirements [13]. Such actions are translational by nature and thus best supported by supervised machine learning techniques.

        For design generation, traditional processes rely on human expertise, gut feeling, or social processes such as brainstorming and crowdsourcing to tap on crowd human intelligence. By taking a data-driven approach, innovators may obtain inspirations from data repositories of prior designs and inspiration sources, e.g., asknature.org, moreinspiration.com and patent database that store the knowledge and concepts from millions of inventors, to stimulate creative ideation [15], [24]. Furthermore, computer algorithms can automatically generate new design concepts by mutating, combining, and recombining prior concepts from large knowledge databases [25]. For instance, genetic programming algorithms may start from a small set of initial designs to generate alternatives that better satisfy a predefined fitness function [26]. With the rapid advances of deep learning, the neural network-based generative adversarial networks and generative pretrained transformers have shown the ability to efficiently generate many and more novel designs that deviate from the training data [27], [28]. 

        For design evaluation, traditionally companies and innovation teams develop minimum viable products and engage external lead users for testing and feedbacks. Such processes are expensive, time-consuming, and serendipitous. By taking a data-driven approach, innovators can automatically evaluate and validate a very large quantity of diverse design concepts to accelerate the design evaluation, validation, and selection process. For instance, innovators can automatically evaluate and filter many new design concepts with a pretrained commonsense knowledge base [29]. One can also train deep neural networks with the data of prior experiments or successful/failed designs in the same context for automatically and predictively evaluating the performances and value of next design concepts (e.g., Atomwise [11], [16]).

        In summary, unsupervised learning of digital footprint data may help explore many more alternative opportunities and discover a bigger opportunity space than what humans could perceive on their own. The expanded opportunity space may cover more novel and more valuable candidate opportunities to choose and pursue and, thus, benefit creativity. Innovators drawing inspirations from large knowledge databases (e.g., patents, papers) and using data-driven generative algorithms may generate many design alternatives and create a bigger design space than what humans alone could conjecture up. The expanded design space may contain more novel and more useful candidate designs to choose and implement and, thus, benefit creativity. Meanwhile, data-trained models can automate the evaluation of many opportunities [23] and many design concepts [29] from the enlarged opportunity and design spaces, accelerate the convergent search and ensure the identification of the best innovation opportunity to design for and the best design for implementation. 

        Therefore, both the divergence-oriented actions (opportunity discovery and design generation) and the convergence-oriented actions (opportunity evaluation and design evaluation) can be augmented by different suitable data-driven approaches (see the taxonomy in Table I) to achieve greater creativity of the innovation process.

        The innovation processes have evolved constantly over human history. In the 19th century, innovations came from individual creative geniuses, such as Thomas Edison, Alexander Bell, and Nikola Tesla. In much of the 20th century, well-organized processes in formal R&D labs and centers in large companies such as IBM and AT&T championed innovation. Since the 1970s, innovation ecosystems like Silicon Valley, where frequent and inexpensive knowledge flows across organizational boundaries, have emerged as the powerhouse of innovation. 

        Today, companies seek innovative opportunities and ideas via various proactive open innovation programs. They collaborate with university-based researchers, organize corporate venture labs jointly with external venture capitalists and entrepreneurs, and run open innovation contests, idea crowdsourcing campaigns, and hackathons to tap into collective and crowd intelligence. To date, the innovation processes have mainly relied on human intelligence (of either individuals, human crowds, organizations, or ecosystems), with high serendipity. 

        As the society continually creates new knowledge and invent new technologies, we accumulate an expanding space of prior arts and knowledge, which can further benefit future efforts to innovate if they can be efficiently searched, retrieved, learned, and synthesized. Otherwise, the expanding space of precedents may instead appear as knowledge burdens on future innovators, who would be required to learn and know a great deal more precedents before they can design new and more useful products beyond the prior ones. 

        The growing knowledge burdens [30] will make the innovation process that solely relies on human intelligence less effective in the future. In contrast, data computation and rapidly advancing AI capabilities [31] can quickly and tirelessly learn an enormous knowledge from large unstructured datasets and transform or synthesize the learned knowledge to generate design concepts of new products and services and automatically evaluate them. Artificial intelligence fueled with big data is driving us into the new “decade of technology intelligence [32]”.

        Therefore, the key enabler for DDI down the road will be the AI for creative tasks, namely, “creative artificial intelligence (CAI)”. CAI requires not only machine learning (of what exist) but machine creation (of what are new and useful). As we elaborated earlier, opportunity discovery and evaluation and design evaluation can be augmented by machine learning (e.g., supervised, or unsupervised models), and design generation can be augmented by machine creation (e.g., generative algorithms). Fig. 2 summarizes the machine learning and machine creation methods to derive CAI in DDI processes. With CAI, even a novice engineer, designer or manager can work creatively on tasks that would otherwise require extensive knowledge acquisition, creative design skills, and gut feeling.

        The future of innovation processes will be more data-driven and AI-empowered, while creative geniuses, formal R&D labs, open innovation activities, and ecosystems will be continually important in the innovation process. Human innovators will play crucial roles as the developers and users of the DDI methods, tools, and expert systems, as well as the implementors of the resultant designs. In turn, these will make human innovators more informed and more inspired to discover more and better opportunities and generate more and better designs by themselves beyond what they could in purely intuitive or social processes. Fig. 3 summarizes the roles of human innovators in the DDI process.

        This article has made three major contributions. The first is the formal definition of DDI as an innovation process paradigm, distinguishing it from other data-related paradigms such as data-driven optimization and DBI. The second is the taxonomy, based on the double hump model, to synthesize different DDI approaches. On this basis, several strategic recommendations are made for future research, education, practice, and governance to foster DDI. The definition and taxonomy of DDI as well as the strategic recommendations may guide future investments, capability building and policy development efforts toward DDI.