Published on

Causal Data Science in Large U.S. Firms

We see more and more examples of companies starting to invest in causal inference capabilities. But how widespread is this phenomenon? And who are the industry players that are active in this space?

Authors
San Francisco Embarcadero, by Jason Barone on Unsplash
San Francisco Embarcadero, by Jason Barone on Unsplash

Causal data science applications seem to be on the rise, not just in academia but also in the private sector. Microsoft has a group working on machine learning and causal reasoning, and is investing in causal inference libraries such as DoWhy and EconML. Google is developing its own causal inference library CausalImpact, and Uber provides the causalml library for uplift modeling. While Amazon pursues research that measures causal interdependence of prices of products and explainable AI, experimentation and causality prove fundamental to the data science activities at Netflix or Lyft. But where does industry stand in terms of adoption? And what types of firms are driving this trend?

Measurement

To investigate these questions, we looked at a sample of the 5,000 largest publicly listed firms in the U.S. (according to market capitalization) and tested whether they disclose any causal data science activities on their websites. Specifically, we collected data on the occurrence of the key word “causal inference” on company domains using the Google Search API. For automatization, we relied on a proprietary service by Zenserp.com. After that, results were manually checked and adjusted if necessary. For retailers and companies with a direct online consumer interface, for example, we often refined search domains to a research (e.g., amazon.science) or career page (e.g., careers.upwork.com). That way, we made sure that we measure internal activities instead of third-party content. The approach is inspired by a growing scientific literature, which is using large-scale web mining and text analysis tools to infer firm innovation behavior (see, e.g., here and here). Finally, we merged this data with financial and other company-level information from Orbis to get a more detailed picture.

Results

We were able to identify 114 U.S. companies, or 2.3% of our sample, that are active in causal inference. These firms have a market capitalization of $64.5 billion and 41.4 thousand employees, on average, which makes them significantly larger than the rest of the sample. In terms of industry breakdown, we find most activity in the software industry (14.9%), but also a number of firms in pharma (7.9%), finance (5.3%), and architectural engineering (3.5%).

Map of causal inference activity by state.
Figure 1: Map of causal inference activity by state.
RankNAICSDescriptionFrequency
15112Software Publishers17
23345Navigational, Measuring, Electromedical, and Control Instruments Manufacturing9
33254Pharmaceutical and Medicine Manufacturing9
45415Computer Systems Design and Related Services7
55191Other Information Services6
65259Other Investment Pools and Funds6
75413Architectural, Engineering, and Related Services4
85182Data Processing, Hosting, and Related Services4
95417Scientific Research and Development Services4
102211Electric Power Generation, Transmission and Distribution3
Table 1: Industry breakdown of companies with causal inference activities.

When it comes to geographical distribution — unsurprisingly — California is taking the lead (35.1%), followed by Massachusetts (12.3%) and New York (11.4%). But there is also a degree of activity in Texas, Washington, and Illinois. While in California companies active in causal inference are spread across the Bay Area, they are concentrated on the East Coast, especially in Boston and New York City.

RankCityStateFrequency
1New YorkNew York12
2San FranciscoCalifornia6
3BostonMassachusetts4
4CambridgeMassachusetts4
5Mountain ViewCalifornia4
6San JoseCalifornia4
7ChicagoIllinois3
8Palo AltoCalifornia3
9Santa ClaraCalifornia3
10SeattleWashington3
Table 2: Location breakdown of companies with causal inference activities.

Moreover, we looked at the correlation between causal inference activity and financial indicators. While we recognize that correlation is not the same as causation, the cross-tabulation is still useful for getting a better sense of the types of companies that engage in causal data science. In addition to the clear association between causal inference activity and firm size, which we mentioned before, we find a strong correlation with net income (p < 0.001). There is also a positive relation with profitability, although the correlation is weaker and only marginally significant (p = 0.078).

Relationship between causal inference activity and net income.
Figure 2: Relationship between causal inference activity and net income.
Relationship between causal inference activity and profitability.
Figure 3: Relationship between causal inference activity and profitability.

Lastly, we explored the relationship between causal inference activity and R&D spending, which looks particularly interesting. Not all firms report R&D expenditures in their annual account statements. But for those that do, we find a clear positive relationship that is quickly approaching 100%. Thus, the largest R&D spenders, with annual budgets of $15 billion or more, seem to be almost all active in causal inference.

Relationship between causal inference activity and R&D spending.
Figure 4: Relationship between causal inference activity and R&D spending.

Summary

What we take away from this analysis is that causal data science still seems to be a niche topic in the industry. Only about 2.3% of the companies in our sample are active in this area, at least as measured by the presence of the term on websites. But put into perspective of a recent representative survey done in cooperation with the U.S. census bureau, which finds adoption rates of only 2.9% for machine learning technologies as a whole, this number does not appear so small anymore. Causal inference activity is strongly correlated with size and the most innovative and profitable firms are pushing this trend. Keep in mind though that our methodology is limited to large publicly traded corporations. We could therefore miss a particularly vibrant start-up scene — a research question which we leave for another day.

The timing seems to be just right for exploring investment opportunities in the causal inference space. Currently, we are still in an early enough stage for causal inference capabilities to be a true source of competitive advantage rather than just a means for catching up with technology leaders. And, as industry experts agree, the importance of causal data science for data-augmented business decisions will only grow in the future.


This post is part of a series that will explore the ecosystem of causal inference. Stay tuned and sign up for our Newsletter below to be updated on future posts.

Share on Twitter

Stay updated

Receive updates on new articles, research, and best practice.

Subscribe to News

Crafted with TailwindCSS and Next.js, running on Vercel. © 2021 causalscience.org.