‘Big Data’ in Animal Industries: Scale is Challenging, But Use is Wicked

September 20, 2021 | Letters

a researcher in mask and hairnet enters data collected from swine onto his tabletAuthor: Dr. Nicole Olynk Widmar, Associate Head and Professor, Purdue University Department of Agricultural Economics

Animal industries from livestock to pets to wildlife management have delved into data-driven decision making, wielding larger and more complex datasets; yet, it is not obvious what big data is, nor why so many industries are infatuated with it. The concept of big data is, by the nomenclature itself, focused on scale. But, form of data, its scope, and eventual use are arguably more complex problems. Understanding the complexities of challenges from scope, form and use of big data, in addition to the more commonly recognized scale, is necessary in order to facilitate valuable data-driven decision making broadly.

Individual households are generating big data sets, some aspects of which are being analyzed (i.e. what you do on your smart phone in your home being shared with app developers and/or online purchase data being used by retailers), and many of which are not; however, analysis does not necessarily suggest usefulness or value. Even when there is potential value in data analytics, the question of value for whom remains.

This Letter presents a viewpoint which incorporates big data, data analytics and data-driven decision making in animal industries into a larger discussion about the challenges surrounding data use. Ultimately, these challenges are outlined for the purpose of informing the discussion on scale, scope and form and to motivate the call to attention to the true wicked problem of big data, which I argue is ultimately use.

Wicked Problems

Wicked problems are “any complex issue which defies complete definition and for which there can be no final solution; such problems are not morally wicked, but diabolical in that they resist the usual attempts to resolve them,” (Brown, Harris, and Russell, 2010). Wicked problems are difficult or impossible to solve social or cultural problems for one or more of four reasons, including: incomplete or contradictory knowledge, numerous people or opinions involved, large economic burdens and the interconnected nature (entanglement) of the problem with other problems (Kolko, 2012).

Often revisited by researchers and scholars, Rittel and Webber (1973) identify ten characteristics of wicked problems (presented verbatim):

  1. There is no definitive formulation of a wicked problem.
  2. Wicked problems have no stopping rule.
  3. Solutions to wicked problems are not true-or-false, but good-or-bad.
  4. There is no immediate or ultimate test of a solution to a wicked problem.
  5. Every solution to a wicked problem is a “one-shot operation”; because there is no opportunity to learn by trial-and-error, every attempt counts significantly.
  6. Wicked problems do not have an enumerable (or an exhaustively describable) set of potential solutions, nor is there a well-described set of permissible operations that may be incorporated into the plan.
  7. Every wicked problem is essentially unique.
  8. Every wicked problem can be considered to be a symptom of another problem.
  9. The existence of a discrepancy representing a wicked problem can be explained in numerous ways. The choice of explanation determines the nature of the problem’s resolution.
  10. The planner has no right to be wrong.

In relation to the inquiry of this Letter — data in animal industries, with implications drawn for animal industries and instigating questions for societal questions surrounding data scale, form, scope and use — the critical aspect of wicked problems is less about the definition or what qualifies as wicked and more about what can be done.

Early discussions were interpreted somewhat bleakly in the sense that wicked problems are openly stated and commonly agreed to have no answers. Following the work of Heifetz (1994) and Roberts (2001), Head and Alford (2015) concur that problems are not simply tame or simply wicked. Instead, they argue that there are degrees of “wickedness” that can be understood, ultimately proposing a spectrum of problem types in an effort to illuminate differential features and intensities of different problems. Ultimately, they argue that while a solution may be elusive, partial or provisional courses of action in response to wicked problems can be formed, especially when a variety of approaches are considered and then applied judiciously as suited for the circumstances faced.

Waddell (2016) presents the concept of the societal change system as a framework to address the organizing challenge to allow multiple and diverse organizations to come together to address wicked problems (motivating the need through examples such as water, financial stability, climate change, food security and degradation of seas). Brown, Harris and Russell (2010) address wicked problems through transdisciplinary imagination, arguing for a collective understanding of an issue combined with imagination, presented as a central tenet of anyone involved in societal change. Indeed, the imagination to challenge existing cultural norms, combined with in-depth understanding of the problem and the conceptualization of knowledge as ever-changing and open to revision and improvement, provides a framework for inquiry into these sorts of challenges for which there are no actual answers.

Wicked Problems of Big Data

Traditional processes of identifying the issue at hand, gathering data, studying all options and choosing one strategy do not work with wicked problems (Camillius, 2008). As the study of wicked problems has progressed to include issues such as societal distrust (Vermaas & Pesch, 2020) and to address evolving and complex global issues of modern times, the general notion of insolvable, complex social and policy-relevant issues being wicked has remained largely unchanged.

Animal industries, agricultural industries, and many other facets of society today face challenges associated with big data. For example, livestock producers face challenges associated with Internet connectivity, storing and accessing large and complex datasets generated by their increasingly technologically advanced equipment, and have more market and economic data available from around the globe than could reasonably (or unreasonably) be employed in on-farm decision making.

Anecdotally, what we hear focused upon in animal industries meetings is the scale of the data. Like most other industries, we are enamored by the sheer scale while simultaneously confused about scope, form and use. We hear proposals for improved connectivity, data storage solutions and a multitude of solutions to supposedly tackle data privacy issues while agreeing to terms on apps and other data generating products without a second thought. The focus is generally on the data itself — what data is it, who owns it, where is it to be stored and/or accessed, when is it collected, and how will we protect it? Admittedly, these are technological challenges, but they are solved by more computing power, larger storage capacities or upgraded infrastructure. While these solutions may not exist yet, the problem itself has a foreseeable solution, even if it is technologically impossible today. Wicked problems are different and cannot be solved through a single invention or technological advance.

The cat is out of the bag, so to speak, when it comes to big data existing in animal industries, but how that data is analyzed, by whom, for what purpose and how much we participate in that process (knowingly or not) are all ongoing wicked problems of data use. If one subscribes to the continuum or degrees of wickedness conceptually (Morrison & Werf, 2016), then one may consider use alongside ownership and the potential for exploitation. One could argue, for example, that the ownership of data or privacy offered to data generators may be more difficult to tackle than relatively simplistic rules about what one can and cannot collect in the first place. After all, ownership or control of the data, if in conjunction with the capacity for use, feeds the potential for value generation or exploitation. In this way, one could envision privacy issues as wicked, but potential use remains even more wicked as ownership without the will and/or ability to use the data would not be so troubling.

In a world obsessed by data-driven decisions, the use of data to inform those decisions is the sticking point. In many animal industries and a multitude of other industries alike, it seems possible, and perhaps even likely, that while we are obsessed with big data in scale and perhaps scope and form, the wicked problem is actually use. In many ways, big data use was not a concern until we created big data, possibly unknowingly setting off a series of further challenges that were not foreseeable prior.

Implications for Animal Industries and Society

Given the breadth of the animal industries from companion animals to livestock animals to wildlife which capture the attention of the world’s human inhabitants in unique ways, the potential for data-driven decision making in animal industries is great. Data analytics related to OneHealth initiatives, especially given the global devastation in the ongoing COVID-19 pandemic-era, and the potential for big data to address global challenges is top of mind. The data at scale is a technological challenge, and the speed at which that data needs to be available to data scientists poses challenges, as do the multiple sources of data generation, storage, access, validation and quality controls. These are all significant technological challenges, economically costly to address and potentially impossible to solve today. All aspects associated with big data are hard, and perhaps even impossible given today’s technology or within bounds of cost or human capacity, but ultimately, data use is — and will remain — wicked.

The wicked problem surrounding big data is use in the form of data analytics to turn complex, large datasets into potentially actionable information. The inability to solve that problem should not deter industries from tackling the question, nor should the almost inevitable repeated failure to do so deter continued inquiry. Like any other wicked problem, these are not problems to be solved, but rather socio-economic, bio-physical, socially and politically charged problems requiring our collective and continuous attention.

Acknowledgments: Sincere thanks to my colleague Dr. Otto C. Doering III who first challenged me to consider social and public policy questions through the lens of wicked problems. I have expressed concern to Dr. Doering at my continued lack of understanding of inherently wicked problems, to which Dr. Doering has aptly responded, “that’s because they’re wicked,” which I feel embodies the nature of the discussion. Also, thank you to Mario Ortez and Taylor Thompson for assistance with reference materials and formatting.



Brown, V., Harris, J. & Russell, J. 2010. Tackling Wicked Problems : Through the Transdisciplinary Imagination, edited by John Harris, et al., Taylor & Francis Group: Milton, United Kingdom ProQuest Ebook Central

Camillius, J. C. 2008. Strategy as a Wicked Problem. Harvard Business Review. Available online: https://hbr.org/2008/05/strategy-as-a-wicked-problem. (accessed on 29 March 2021)

Head BW and Alford J. 2015. Wicked Problems: Implications for Public Policy and Management. Administ & Soci. 47(6):711-739. doi:10.1177/0095399713481601

Kolko, J. 2012. Wicked Problems: Problems Worth Solving: A Handbook & A Call to Action. AC4D: Austin, Texas, United States.

Morrison, K. & Werf, G. V. D. 2016. Large-scale data, “wicked problems”, and “what works” for educational policy making. Edu Rese and Evaluation. 22:5-6, 255-259, 10.1080/13803611.2016.1259789

Rittel, H.W.J., Webber, M.M. 1973. Dilemmas in a general theory of planning. Policy Sci. 4, 155–169. https://doi.org/10.1007/BF01405730

Vermaas, P. E. and U. 2020. Revisiting Rittel and Webber’s Dilemmas: Designerly Thinking Against the Background of New Societal Distrust. She Ji: The J of Desi, Econ, and Innov. 6(4): 530-545 https://doi.org/10.1016/j.sheji.2020.11.001

Waddell S. 2016. Societal Change Systems: A Framework to Address Wicked Problems. The J of Appl Behav Scie. 52(4):422-449. doi:10.1177/0021886316666374