This blog post describes some considerations around ethical use of artificial intelligence, or in most cases more accurately, stochastic parrots, in academia. I will start by describing use in scientific research, and then discuss use in teaching.
Guidelines for ethical work in academia
I will assume that ethical use of AI requires at the very least adherence to the following two sets of guidelines.
Open Science
Open Science principles have seen widespread adoption by the UN (see the UNESCO recommendation on Open Science) and pretty much every other influential actor and university. Therefore, they can be considered as wordwide parameters for academic work, both research and teaching.
Open Science principles cover many aspects of the scientific endeavour, with various perspectives. Three important ones are those of the UNESCO (see here), the EU (see here), and the Netherlands Open Science Program (see here).
UNESCO
UNESCO emphasizes that all infrastructure and products for the scientific endeavour must be open and free. Some aspects they describe are open source software, open hardware, and open science infrastructure. They also list core values and guiding principles. The listed core values are quality and integrity; collective benefit; equity and fairness; and diversity and inclusiveness. The listed core principles are transparency, scrutiny, critique and reproducibility; equality of opportunities; responsibility, respect, and accountability; collaboration, participation, and inclusion; flexibility; and sustainability.
EU
The EU lists a number of notable Open Science practices, including providing immediate and unrestricted open access to scientific publications, research data, models, algorithms, software […]. The EU Open Science Policy Platform also lists eight core areas, often called pillars, of Open Science: 1) rewards and incentives, 2) indicators & next-generation metrics, 3) future of scholarly communications, 3) European Open Science Cloud (EOSC), 5) FAIR data, 6) research integrity, 7) skills & education, 8) citizen science.
Dutch Open Science Program
The Dutch Open Science Program defines four core values: Quality and integrity; Collective benefit; Equity and fairness; and Diversity and inclusiveness. They translate these into the following guiding principles: Scientific knowledge is a public good and access to it is a universal right; Scientific outputs and processes must be as open as possible, but as restricted as necessary; Reproducibility and scrutiny are essential to safeguard the quality and integrity of scientific work; Diversity, equity, and inclusiveness are crucial for the success of Open Science; Academic and digital sovereignty must be safeguarded. In this last principle, they explicitly state:
To guarantee scientific knowledge as a public good for collective benefit, it is important to consider the sustainability, governance, and financial models of scholarly infrastructure, (retention of) copyright and open licensing of scientific work. The risks of becoming more and more dependent on commercial or foreign providers and their terms of use in all stages of the research life cycle asks for open (not-for-profit) alternatives for digital services and regulation.
The Netherlands Code of Conduct for Research Integrity
One thing these key stakeholders in the Open Science landscape have in common is that integrity is a core value. This shows how scientific integrity and open science are intertwined. This becomes even clearer when inspecting the Netherlands Code of Conduct for Research Integrity. This code of conduct defines five principles: Honesty, Scrupulousness, Transparency, Independence, and Responsibility.
This code of conduct is binding to all Dutch universities, universities of applied science, academic medical centers, funders, and related organizations. Other countries have similar codes of conduct, and the basic tenets will be the same.
For scientists, integrity is not an optional nice to have. At all times, scientists must be transparent, accountable, independent, scrupulousness and responsible.
Parameters for choosing scientific infrastructure
Both Open Science principles and scientific integrity principles mandate that scientific infrastructure is fully open and transparent (so, open source); that the applications and tools forming this infrastructure are developed in an integreous manner, responsibly and sustainably, and for the collective benefit of mankind; and that academic sovereignty is safeguarded.
The responsibility to consider these principles lies with both the individual scientist and at the institutional level. Institutions must survey their scientific infrastructures (their enterprise architecture choices, so to speak) and plan for phasing out closed (commercial, industry-owned) applications in favour of open science alternatives (open source and community-owned). Individual scientists must carefully evaluate the nature and provenance of any new tools they consider exploring or adopting.
Requirements for using AI in scientific applications
Applied to uses of AI (which I here use loosely to encompass large language models, other generative AI, and machine learning applications), the following requirements emerge (I use the word ‘tool’ as an umbrella term to encompass all aspects of any AI application, including the software and any underlying models):
- the tool is fully open source
- the provenance of the tool is fully transparent
- the tool was developed ethically
- the tool can be used sustainably
- the tool is controlled by academic or public organizations
Applying these requirements: ChatGPT and OpenAI
These requirements mean that, for example, the generative AI products produced by OpenAI can not be used for scientific work. OpenAI practices pretty much the opposite of openness (see this comparison by Mark Dingemanse). Their tools are not open source, but neither are the procedures they use(d) to develop the tools (i.e. the provenance).
However, investigative journalists revealed that they exploited Kenyan workers to train their large language models; very emotionally taxing, potentially traumatizing work, for excessively low salaries and without proper support (see here, here, here, or here. Without this training, their LLMs would spit out the racist, sexist, violent content that large swaths of the internet consist of.
Keep this crucial human training of a generative AI model in mind; we will come back to this later.
In addition, they knowingly imported works protected by intellectual property law, then refusing to be open about what they ingested ‘to avoid being sued’ (see here, here, or here.
In addition to these unacceptable practices, once constructed, the use of generative AI models is excessively costly in terms of the required energy (and so, harm to the planet; see for example here, here.
Using (free or paid) products that were developed with these methods cannot be commensurated with the principles in the scientific code of conduct and the open science principles.
This is true for any use of AI tools developed Big AI / Big Tech by scientists, including seemingly trivial use cases such as summarizing an email, meeting notes, or just for fun. Use of these products creates more value for this industry, an industry that clearly has no regard for the collective benefit of humandkind, planetary health, or intellectual property.
Epistemology and Sovereignty
However, specifically when applying such AI tools to the scientific endeavour (not just answering emails, but conceptualizing studies, collecting data, or conducting analyses), there is an important additional problem with using AI tools developed by industry organizations.
One of the reasons why academic sovereignty is so important to safeguard, as set out in the open science principles outlines above, is that scientists’ jobs are epistemic in nature. We produce knowledge, ideally ultimately to make the world a better place. The process of knowledge production is not neutral or objective; it is a subjective endeavour, rife with bias and influenced by politics and human convitions. The transparency and openness mandated by scientific integrity codes of conduct and open science principles partly serve to mitigate this unfortunate truth.
Any scientific enterprise consists of a plethora of decisions: which sampling strategy to employ, which analysis to use, which tools to use, the list goes on and on. Each of these decisions has epistemic implications. More concretely, the tools scientists use shape the way they produce knowledge in a variety of ways.
For example, the fact that the widely used quantitative analysis package SPSS only implemented Cohen’s d in version 27 (released in 2019). Cohen published his landmark book on power analysis and effect sizes in 1977, with a second edition published in 1988. In 1977, SPSS was on version 2; in 1993, version 5 was released, and in 1999, version 10. The fact that Cohen’s d was only included in SPSS in 2019 has considerably slowed down its widespread adoption in the social sciences, and as a consequence, adoption of sample size computation when planning studies.
Similarly, software for qualitative data analyses typically lets users label qualitative data fragments using codes organized hierarchically. This constrains how scientists construe the patterns in the data they analyze to hierarchical structures. For example, the analysis usually results in a code structure; the same code structure for all participants, requiring the assumption that the patterns of interest exist identically in all participants. In addition, Scientists typically do not specify what kind of hierarchical relationship exists between codes. Both characteristics of the used tools fundamentally change the knowledge that is created through those studies.
Both examples show how software fundamentally changes how knowledge is generated in the social sciences. Many things now considered “psychological knowledge” because of quantitative or qualitative research are in fact far from the truth, partly because software implicitly or explicitly favoured certain epistemic alternatives over others.
It is the responsibility of the researcher to be aware of their epistemic aims and keep these in mind when determining their ontological and epistemological stance, the theory to use, the methods to employ, as well as the operational procedures, including whichs tools to use. For example, it is common for researchers to select one analytical software package over another based on the analyses that are supported.
When using AI tools in scientific applications, whatever is produced is determined by the company developing the model. For example, the instructions provided to the Sama workers who trained OpenAI’s ChatGPT to stop spewing out sexual abuse, violence, and other problematic results shaped the responses the system will provide and which are trained out of the system. Any use of this system is constrained by the parameters provided by OpenAI.
In addition to this deliberate influence to determine the system’s output, the system’s input does not represent an unbiased sample of the human mind. This is because online activity differs per age group, country, and with other characteristics; and in addition, OpenAI made a lot of decisions when they decided which sources to ingest and which to ignore.
In other words, OpenAI controls what is returned by their generative AI models. This is the same for other Big AI models. Not only do they have full control over what the models produce, because the models and their development procedures are not fully open, biases cannot be spotted.
Applications of these AI tools in the scientific endeavour fulful an epistemic role; as such, using such tools delegates this responsibility from scientists and universities to exploited workers retraining AIs and their Big AI employers in Silicon Valley. Relocating this epistemic responsibility from academia to industry while simultaneously ‘academy washing’ the efforts seems quite inconsistent with the code of conduct for scientific integrity, as well as with open science principles such as protecting academic sovereignty.