This is a self-archived – parallel published version of this article in the publication archive of the University of Vaasa. It might differ from the original. A quick review of ethics, design thinking, gender, and AI development Author(s): Kronqvist, Aila; Rousi, Rebekah Ann Title: A quick review of ethics, design thinking, gender, and AI development Year: 2022 Version: Accepted manuscript Copyright ©2022 Taylor & Francis. This is an Accepted Manuscript of an article published by Taylor & Francis in International Journal of Design Creativity and Innovation on 02 Nov 2022, available online: http://www.tandfonline.com/10.1080/21650349.2022.2136762 Please cite the original version: Kronqvist, A. & Rousi, R. A. (2022). A quick review of ethics, design thinking, gender, and AI development. International Journal of Design Creativity and Innovation. https://doi.org/10.1080/21650349.2022.2136762 https://doi.org/10.1080/21650349.2022.2136762 https://doi.org/10.1080/21650349.2022.2136762 A QUICK REVIEW OF ETHICS, DESIGN THINKING, GENDER AND AI DEVELOPMENT Aila Kronqvist a* and Rebekah Rousib a University of Jyväskylä, Faculty of Information Technology, Jyväskylä, Finland b University of Vaasa, School of Marketing and Communication, Vaasa, Finland, 0000- 0001-5771-3528 *Aila Kronqvist, aila.j.kronqvist@student.jyu.fi, University of Jyväskylä, PO Box 35, FI-40014 University of Jyväskylä, Finland. Aila Kronqvist is PhD researcher in faculty of Information Technology, Cognitive Science in University of Jyväskylä. In her dissertation work, she is focusing on gender and ethics in design thinking. Rebekah Rousi is an Associate Professor of Communication and Digital Economy at the School of Marketing and Communication, University of Vaasa. Rousi holds a PhD in Cognitive Science. Her work has focused on understanding user and designer experience and emotions in a range of design settings from user interface design to human-robot interaction. mailto:aila.j.kronqvist@student.jyu.fi A QUICK REVIEW OF ETHICS, DESIGN THINKING, GENDER AND AI DEVELOPMENT Ethics, artificial intelligence (AI) and design thinking are current buzz words. The threat of bias-riddled machine learning algorithms is represented media- wide. Research and development initiatives are endeavouring to ‘translate’ ethics into processes and machine logic, and design thinking as a method is adopted to gauge the interests and values of a vast range of stakeholders. Gender, its framing, reflection, and critical evaluation in relation to design thinking as a means for developing ethical AI appears to be less represented in scholarly discourse. Against a background of critical theory and gender studies that describe and problematise gender its construction and norms in socio- technological discourse, the authors of this article aim to generate insight into the current state of gender in design thinking research focused on ethics and AI. A review of scholarly articles revealed trends in popularity of concepts and prominence in the application of design thinking in specific fields (i.e., educational research). Repetition characterises the more challenging topics or wicked problems. Provocation and investigation of gender from the perspectives of practitioners, creativity and its influence in design thinking seem all but visible. Keywords: Ethics, design thinking, artificial intelligence, gender, creativity Subject classification codes: include these here if the journal requires them Introduction ‘Design thinking’ has been a popular term for the past few decades. It has been used to refer broadly to thinking and cognition within design – designer’s thinking, cognition in design its processes and formations, problem-solving and creativity (Buchanan, 1992; Cross, 2011, 2018). It has also been used to describe a co-creational methodology that engages multiple fields and disciplines to effectively problem-solve, imagine and realize products and services from a wider stakeholder perspective (Kimbell, 2011; Leifer & Meinel, 2018). The scope includes user-centrism, human-computer interaction, business, design, engineering and more. Design thinking as a methodological approach is known for its multidisciplinary engagement in creative problem-solving and design realization. Stanford d.school’s design thinking process model is perhaps the most well-known artefact relating to practical application of design thinking (Thoring & Müller, 2011). It was adapted from the consultancy company IDEO that had focused on developing systematic design methods for tackling wicked problems (Buchanan, 2011; Kelley & Littman, 2001). In international health emergencies (pandemics), global warming, economic instability and resource challenging climate, designers are faced with bountiful wicked problems. None-the-least, the race towards a smart society inherent with highly functioning artificially intelligent (AI) systems has introduced a new wave of challenges – ethical issues such as privacy and security concerns (Fischer-Hübner & Martucci, 2014; Timmers, 2019), autonomy (Laitinen & Sahlgren, 2021), inequality and growing imbalance in wealth distribution (Bootle, 2019), human role-negotiation, employment and dignity (Boon, 2021), energy issues (Peter, 2018) and bias within algorithm logic (Livingston, 2020). All these issues are currently represented technology design and ethics discourse. Ethics, in its raw sense draws from the idea of ensuring that one treats others in the way they would themselves want to be treated. This assumes a mutuality and reciprocity that enacts distinctions between what is right and fair in certain circumstances, and what is unjust and evil (Singer, 2011). Current design practice often includes the implementation or integration of AI systems – ethical discussions are rife in this area (Chaudhury, 2021; Treiber, 2021). These AI-related wicked problems and ethics have been the impetus for the authors of this article, and for what is present in this text as a quick review of ethics, design thinking, gender and AI development. Considerations for this article began with the authors discussing their experiences as females in industry, design, research and software development. Through highlighting the commonalities of their experiences such as the likelihood of younger (trendy, white) males receiving exciting and highly visible projects, in addition to freedom of choice, time, combined with higher levels of freedom and creativity. While females were issued the ‘must do’ assignments with a reluctance to say ‘no’ due to time missed during maternity leaves and caring for sick family members. Soon the discussion eventuated at the relatively homogenous sample of designers and developers in charge of designing our lives and our algorithms (Klugman, 2021; Lohr, 2018). Thus, the discussion soon moved towards the lack of diverse representation (in this case gender diversity) in design which also replicates in designs and algorithmic bias (Houser, 2019). Bias, and particularly AI (or algorithmic) bias, represents one of the pressing issues in current ethical design and AI development discourse (Ferrer et al., 2021; Roselli et al., 2019). From this point of departure we understand design thinking in two ways: 1) as a user-centered design (UCD) method that may be significantly influenced by the person undertaking and facilitating the UCD process; and 2) the design thinking behind the algorithmic logic (white male bias). Thus, the authors began to look beyond data and towards the developers themselves. What does algorithmic bias (Danks & London, 2017), and ‘designed bias’ (Perez, 2019) in general say about the frames within which designers and developers operate, and the demographic bases of the development teams in general? And, how has this been addressed in research? Bias describes prejudice against or inclination for certain individuals or groups that are often considered as unfair or imbalanced (March, 1972; Oxford Languages, 2021). From this understanding, there is an assumption that bias can be seen in several circumstances regarding groups and individuals that vary from a range of perspectives (culture, ethnicity, gender, class, age etc.). Here, we focus on gender due to the first author’s experience as a female practitioner working in information, software, and enterprise architecture. Based on this, the authors were keen to probe deeper into what gender and its potential bias mean from the perspective of design thinking and its ethical application. Not only are there ethical repercussions for matters such as gender bias in design and design thinking processes, but the ways in which these biases impact creativity – what it means, how it is interpreted (read or understood) in processes and products, and how it limits available information, and approaches. In Finland, the context of this study, there are long traditions of gender-based professional sorting (women to handicrafts and men to design). This includes common beliefs regarding who has the potential for higher levels of creativity (men) than others (women) (see e.g., Riska, 2008; Svinhufvud, 1998; Utriainen, Salmesvuori & Kupari, 2014). In fact, in the fields of art and design, men were often assumed to adopt the role of designer or architect, while women were relegated by default to repetitive, craft-like fields that emphasized mechanical skills (Svinhufvud, 1998). Research about creativity in relation to gender has not found any significant measurable differences regarding the variable within and between creative fields. According to studies, there are no significant differences in creativity between genders in childhood and adolescence (Baer & Kaufman, 2008; Stortzfus et al., 2011). Studies have shown no significant difference between levels of creativity and gender in children and adolescents (Baer & Kaufman, 2008). There were conflicting results shown regarding gender and creativity in adults. At the heart of the arguments explaining why differences exist rests a myriad of theories. These range from different prioritizations in the sensory-cognition system (Runco, Cramond & Pagnani, 2010) to “blanket environmental” ones (Baer & Kaufman, 2008). Despite the lack of difference between actual levels of creativity, there is less representation of women in creative fields in adulthood, especially in technical professions, possibly due to differences in gender expectations and environments during childhood (Baer & Kaufman, 2008; Stortzfus et al., 2011). The trajectory of this quick review and its broader research study is to ascertain what is already understood of the ways in which these categorizations live on in a design-oriented AI-aspiring society. This is coupled by further examination of how ethics is linked to discussions in design thinking, and if this in turn strongly connects to gender considerations of the practitioners in the field. The study utilizes literature searches conducted through Google Scholar to show result quantities of key search terms, how studies connect with one another, and how ethics, in relation to gender and AI development are being represented in contemporary design thinking research published discourse. In order to establish a common language for the paper, we begin by a concise account of ethics, followed by briefly describing design thinking as an IDEO and d.school derived methodology, including the conception of creativity in this process model from the perspective of collective cognition. Acknowledging the richness and diversity of ethics as a philosophical and sociological, yet focusing on brevity, we touch upon the origins and some classifications, then move into greater detail from the perspective of modern design and design thinking practice in AI development. The literature search results are presented, which is followed by a reflective discussion interpreting what these results represent in terms of conditions related to gender, ethics and design thinking at the dawn of a smart society. Ethics Ethics is an area of scholarship and philosophy that spans back through human history (see e.g., Kant, 2001; Rivers, 2000). Not only is the field of ethics vast, but as with all deeply philosophical and by nature, controversial scholarly disciplines, the character of ethics, what they involve, and how they may be defined are still much contested (Braunack-Mayer, 2001). During times of pandemic, economic instability, evolving workforce and intelligent technological transformation, ethical questions move increasingly to the fore of public consciousness (Chandler, 2014; Hoffmann & Hahn, 2020; Mascaro, Korb, Nicholson & Woodberry, 2015). There is no ‘one size fits all’ approach to understanding ethics, their philosophical and theological roots, and the factors that influence their construction within cultures and various cultural settings. Ethics, and the field of moral philosophy, entail the organization, standardization, recommendation and defending of behavior that could be considered as either right (just) or wrong (unjust) (Fieser, 2020). Ethics is commonly divided into three main approaches: 1) metaethics - the origins of ethics; 2) normative ethics - where moral principles are related to behavior and how they are permeated through socio-cultural systems, both discursively and systemically (Gewirth, 1960; Pellegrino, 1995); and 3) applied ethics - ethics put into practice through design, systems and behavior (Fieser, 2020). Applied ethics go beyond utilitarian ethics (a part of normative ethics) in actionable endeavors for maximum benefit for many with minimum harm (see e.g., Mill, 2010) through the acceptance of complexity. Applied ethics attempts to analyse phenomena and apply principles, standards, and guidelines to generate the greatest possible positive outcomes considering controversy and complexity. There is always knowledge of trade-offs and potential opposition, as the issues applied ethics deals with will never be fully and acceptably resolved on a societal level or otherwise (Fieser, 2020). These may be understood as moral dilemmas (Nichols & Mallon, 2006) or the wicked problems of the moral world. The historical emergence of The Golden Rule (The Ten Commandments of the Bible and the Code of Hammurabi) to treat others the way you would like to be treated (Singer, 1963) – is an interesting topic for applied ethics. As it also falls into the category of both ruler/influencer -influenced invention, and the necessity to provide moral structure to an otherwise chaotic society (see, e.g., Prince, 1904; Saariluoma & Rousi, 2020). Virtue ethics, as studied by the Ancient Greeks (e.g., Aristotle etc.) relates more specifically to individual actors in connection to moral character and virtues held and represented by individuals or groups (Hursthouse & Pettigrove, 2018). Virtue ethics have proven extremely popular within the modern corporate landscape, particularly in relation to technology-related issues, brand management and corporate reputation (see e.g., Audi, 2012). This is due to the social and cultural power of identifying values (i.e., equity, fairness, sustainability etc.), aligning them with the business message and offerings, which in turn aligns potential customers (Murphy, Laczniak & Wood, 2007). Clear, consistent, and repetitive communication of these values in public discourse via websites, social media, and other marketing media, develops strong semantic associations between particular characteristics of an organisation and recognition of what is deemed as ‘good’ or socially responsible (Van de Ven, 2008; Williams & Murphy, 1990). Thus, from a marketing and communications perspective, virtue ethics is about creating strong associations between what is good, just, and responsible, while at the same time othering and defining what is not (Ailon, 2015). From a design perspective, design teams and practitioners have the task of probing into the heart of the organisation and its people’s (multiple stakeholders’) and embodying these through output (communication, form, logic, business models etc.) (Mortati, Villari & Maffei, 2014). The harnessing of values, or value-creation is not simply projected in the look and feel of products and services (Amit & Zott, 2001). Rather, the way they operate and are encountered, their existence and relationality within communities and societies (freedom of choice, voluntariness, satisfaction, benefits etc.) additionally radiate a company’s ethical stance (Rousi, 2021). This is where the mechanisms and qualities of design thinking as a cognitive method of inclusion – users, stakeholders, multidisciplinary – enter the picture, particularly in the era of intelligent digital transformation (Magistretti, Pham & Dell’Era, 2021). Design thinking Design thinking in its various forms, has been a popular concept in industry and academia for several decades (Verhulsdonck, Howard & Tham, 2021; von Thienen, Clancey & Meinel, 2021). Depending on one’s scope, design thinking can be thought of as a movement to combine multiple disciplines and perspectives in design, development and business processes (i.e., as seen in the d.school model; Kimbell, 2011), or can be traced further back through cognitive scientific studies on design cognition, design logic and even problem-solving (see e.g., Cross, Dorst & Roozenburg, 1992; Dorst, 2011; Newell & Simon, 1972; Rowe, 1991). If design refers to planning, process and shaping phenomena through intentionality (Crilly, 2011; Dennett, 1987; Parsons, 2015), then design thinking can be seen as the explicit act of expressing and communicating thought and values through design. It can also be understood as a means of extending and embodying the human mind through designed objects, services and systems (Saariluoma, Alanen & Rousi, 2021). This idea of human augmentation comes into play when engaging in design thinking as a method. For through multi-disciplinary teams, stakeholder inclusion and then manifestation of the design project the method can be seen as an actionable process of collective and extended cognition (Murty, Paulini & Maher, 2010). This is particularly pertinent when considering that a great deal of design projects involve the development of AI – cognitive technology – that carries the logic of its human creators. Over or under representation of particular cultural groups or types of designers instills a narrow logic within the systems that excludes diversity and the potential for broader considerations and alternatives. The practical six-step design thinking process that was developed and promoted through Stanford’s d.school can be understood along the lines of strategic design-focused communication that combines minds (knowledge) and articulate values from multiple viewpoints to create and sustain more effective design outcomes (Kimbell, 2011). Moreover, through engaging with experts across disciplinary and professional borders there is an expectation not only that the outcomes will be more effective and affective, but sustainable (economic, socially responsible, and possibly even environmental) and ethical (doing right and being good in relation to broader fields of stakeholders) (Birkeland, 2012). In fact, this ethical standpoint of attempting to maximise benefits for as many stakeholders as possible while increasing positive societal impact, has been a cornerstone of design thinking discourse over recent years (Andrews, 2015). This has been manifested in activities such as manifestos, workshops, reports, and toolkits (see for instance, IDEO’s DesignKit1 or Sohail, 2017). Levels of ethical thinking, or moral positioning can also be seen within current corporate strategies such as those that promote sustainability (environmental, economic, corporate, humane) and social responsibility (i.e., Google, IDEO, Gofore etc.). This form of approach is also key to understanding many of the current development practices such as Agile, Lean, and other human-centered design and development initiatives. By nature, design thinking aims at offering the opportunity to influence the direction of design from a human (humane) and stakeholder-centric perspective (Kolko, 2015). This is achieved through increasing the possibilities of positive impact through engagement, interaction and communication, which in turn may be understood as an ethical and democratic approach to design (Saward, 2021). The more intensive the design thinking process, the more possibilities there are for opening the gates to ethical considerations (see for instance, Dorst, 2011; Panke & Harth, 2019). The Ethical Design Thinking Toolkit (Sohail, 2017) for example, combines key tools in human-centric design from 1 https://www.designkit.org/ https://www.designkit.org/ IDEO and considerations for personal values by Joe Edelman (co-founder of Time Well Spent, 2018). This combination provides a strategic approach for aligning personal and human values with tangible product design (product values, see e.g., Friedman & Kahn, 2003). In reflection of the virtue ethics description on value-alignment with business (image, vision, marketing) and offerings mentioned above, this ensemble allows for value-based embodiment within the organisation’s design. Questions such as, “What is the problem I am trying to solve?” or “Who am I trying to solve these problems for, and why?”, as well as, broader questions including, “Does my challenge drive toward ultimate impact?” are as much the foundational considerations of design thinking as they are for ethical thinking (Saariluoma & Rousi, 2020). Thus, design thinking from a practical design methodological perspective, entails deeper levels of human understanding in light of design, shifting the emphasis from, “Will someone use it?” to “Why and how will they use it, and what does this mean in the context of their lived experience?” (Dorst, 2011; Leikas et al., 2012). This deeper understanding is reinforced by multi-professional, or multidisciplinary, examination of the design problems in question in order to gain a holistic view for design and development. To build knowledge on design problems through multiple disciplines also entails that ethical questions arise that may vary in nature from one field to the next (Dwivedi et al., 2021; Norwood & Paterson, 2002). For innovation processes and practices in the space of complex data-intensive emerging systems such as AI, this characteristic is paramount (Dwivedi et al., 2021). For, in the case of AI, ethical considerations are not only implicated in the technology itself but amasses questions via the borders it spans and fields it entails (Leikas, Koivisto & Gotcheva, 2019). For this reason, the authors have sought to gain an enhanced understanding of the current state of ethics in design thinking scholarship and discourse. Numerous perspectives are relevant to this topic (i.e., privacy, equality, economics, ethnicity, culture, linguistic etc.) the matter of gender in relation to ethics and design thinking was chosen due to its relevance and resonance with the authors’ professional experience. Through taking the aspect of gender in design thinking and ethics, this quick review serves as a basis for extended empirical research in the field. Materials and method A literature review was performed to generate an understanding of how ethics have been studied and represented in design thinking research from the angle of gender. The literature searches progressed in a structured manner (Swanson & Santamaria, 2021) that began with the basic terms of “ethics and design thinking” and accumulatively progressed through the themes: “gender and design thinking”; “gender and design thinking and ethics”; and “AI and ethics”. The authors utilized online search using Google Scholar. To ensure integrity, two researchers performed the same search actions in the above- mentioned databases (Kitchenham, 2004). The search results were ordered by relevance. Data that arose from the search term “gender and ethics” is excluded, because the resulting articles rested outside the scope of this review (they were not related to design thinking or AI discourse). To maintain a manageable sample, enabling a ‘quick review’ of the current state of research and discourse, the first two pages from search results of each category were documented in an excel. These were then analyzed based on the article title, abstract, focus of the article and year of publication. For the purposes of this quick review, the year or period of publications is important as it gauges the rate of popularity in represented themes, topics, and issues during certain windows of time. To ascertain active and current discourse of research studies from the respective periods presented (based on publication date), possible books were rejected from results. Books are fruitful to examine, yet often possess a broader scope, theoretical contribution and discursive framework while presenting thought that has developed and matured over a longer period (Tilburge University, 2021). The overall search results are represented in Table 1. Table 1: Google Scholar search results per search terms Search terms Hits Date Year (range) "Ethics" and "design thinking" 555,000 11.3.2021 2011- 2021 "gender" and "design thinking" 20,100 19.3.2021 2001- 2020 "gender" and "design thinking" and "ethics" 2,280 24.3.2021 2001- 2021 "AI" and "ethics" 1,980,000 19.3.2021 2003- 2020 “AI” and “ethics” generated 1,980,000 hits (publication years ranged from 2003 to 2020). This is currently a popular field of research as researchers, technologists and businesses alike strive to instill ethical principles and account for ethical issues within their AI development (Vakkuri et al., 2020). This is broader than strictly design thinking research, yet the researchers considered it important to ascertain the level of inclusion of human insight through so-called human-centered approaches to emerging technologies (AI ethics). AI ethics is kept in the results as opposed to gender and ethics, due to the prominence of the topic within the discourse of design thinking related literature. The search words “ethics and design thinking” generated 555,000 hits (between 2011 to 2021). This shows that ethics and design thinking are relatively recent topics. “Gender and design thinking” attracted 20,100 hits (ranging from 2001 to 2020) demonstrating a longer, yet perhaps not so popular tradition of study in design thinking research. Finally, “gender and design thinking and “ethics” gained 2,280 hits (dated 2001 to 2021). The time range is like the gender and design thinking category, no doubt with overlaps, yet the inclusion of “ethics” meant a dramatic decrease in hits (see Figure 1). Figure 1: Search and selection process The details (citation details, title, abstract, keywords and focus of study) of the first two pages of search hits from each category were entered into an Excel spreadsheet. The authors then reviewed the document and coded the data according to topic and focus. The authors aimed to ascertain the scope of focus areas that have been covered within the sample of literature to understand current discourse and trends in each category. In total, 69 articles were included in the spreadsheet for closer review. The abstracts were read and analyzed, and subsequently 12 full texts (3 from each search term) with the highest number of citations were assessed in terms of deriving themes. Results Here, the findings of the excel-based data analysis are discussed in relation to the main categories and trends arising in the literature. The results of each search term are discussed sequentially. These results can be seen in the supplementary material (Appendices tables) for the article. The findings were analyzed in terms of keywords, focus, and how design thinking was approached. In addition to gaining insight on the most popular topics and range of perspectives represented, the authors also reflect on the citation amount to ascertain the level of influence the publications have had on other research. "Ethics" and "design thinking" Fourteen publications out of the results emerging on the first two pages of the search on “ethics” and “design thinking” were chosen for this review (see Appendix 1). These publications were selected due to the direct inclusion of “ethics” and “design” and/or “design thinking” being included in their title and their relevance – as evaluated from their abstracts – in relation to the design of emerging (AI) technological systems in practice and education. In total, at the time that this quick review was performed, these publications were cited 664 times (2011 to 2021). Zwijsen et al.’s (2011), “Ethics of using assistive technology in the care for community-dwelling elderly people” was cited 263 times being seemingly the most influential paper out of the 14. The article focused on the development of assistive technology and ethics in context of aged care. Their study revealed three prevailing themes: personal living environment – issues of privacy, autonomy, and obtrusiveness; the outside world – stigma and human contact; and the design of assistive technology devices – individual approach, affordability, and safety. Zwijsen and colleagues discussed how while the above-mentioned concepts are vital for ethical approaches to design thinking grave problems rest in their application. The emphasize the term “thick concepts” to denote constructs that lose the power to signify through careless overuse (see e.g, Laclau, 2017). Their research showed tendencies for assumptions such as those pertaining to the ethical value of human autonomy and self- determination, without gaining full insight in the types of case-specific and contextual factors that impede possibilities for autonomy (i.e., cognitive decline, mobility issues etc.). Peter Dalsgaard’s (2014) article, “Pragmatism and design thinking”, was the next most cited (165). Interestingly, this takes a philosophical pragmatist perspective to understanding design thinking through attaching experience to ethics. In this article, Dalsgaard focuses on the relationship between pragmatism, or the position that individuals experience and know the world through agency (doing), and design thinking. Dalsgaard draws on John Dewey (2004) to cite the similarities between pragmatism and design thinking. In his article, he elevates the pragmatist principles of psychology, logic, experience, art, democracy, morals, and ethics. No doubt based on this inclusion of morals and ethics; the paper emerged in the search results. Dalsgaard compares design thinking and pragmatism through learning and emergence – understanding that the world is never a complete picture or fully finished product. Thus, pragmatism, like design thinking, is about understanding constantly evolving processes. This theoretical and anecdotal paper2 interweaves philosophy with practice, and as seen by the citation amount, the theory- meets practice way of uniting design thinking to ethical questions has been popular. This article promotes an understanding that everything happens in interaction. Design thinking is posed as a site of interaction between disciplines and stakeholders. Shapira, Katchie and Nehe’s (2017), “The integration of design thinking and strategic sustainable development” (88 citations) taps into the burning topic of sustainability. While the citation rate is still relatively small considering work on other topics, this article attempts to establish a framework for strategic sustainable development. This can be considered an applied ethics approach to instilling sustainability within human-centred design and innovation practice through design thinking. The study presented in this article 2 The paper documents empirical examples in which Deweyan theoretical concepts (peepholes and inquisitive use) were translated to practice. was influenced by IDEO’s understanding and promotion of design thinking – very much from the perspective of promoting understanding among educators. While the article differs in approach to Dalsgaard’s article, it also emphasizes the importance of learning and education. Here, the authors argue that ethics (and sustainability) are conditions within an ongoing process, for which we need the tools to learn. Similarly, some papers addressed education explicitly. Kirkman, Fu and Lee’s (2017), “Teaching Ethics as Design” observes the factors influencing learning and skill acquisition (including thinking skills) in the context of design and engineering education. Kirkman et al’s (2017) article focuses on understandings of ethical values in design. This paper presents a study in which problem-based learning is used as a vehicle to systematically identify and analyze ethical and design values in every decision made within a design process. Ethical thinking skills and moral imagination are examined via Latent Semantic Analysis of short answers to ethical design questions. In the same vein, Spiekerman and Winkler’s (2020) much less cited article (six citations) “Value-based engineering for ethics by design” focuses on values in the design process and how these should guide the deliberation of priorities. A key notion within the paper is emphasis on respect for users from design to implementation. Jensen and Vistisen’s (2013) article, “The ethics of user experience design discussed by the terms of apathy, sympathy and empathy” (one citation) explores the constructs of empathy in relation to sympathy and apathy. Empathy and sympathy are often confused with one another, and the notion of apathy is used to expand understandings of ethics towards social responsibility and accountability for the designs that have been produced. The remaining articles in the list by Madson (2021), Ericson (2021) and a book by Rousi, Leikas and Saariluoma (Eds. 2020) have not yet been cited according to the search date. Key issues within these publications include creativity, design thinking, innovation, education (medical and design/engineering), entrepreneurship, emotions, and culture. Several key themes emerged through this literature search category. Firstly, emotions and empathy, is an area that is recognized as contributing to our understanding of ethics, as well as the incentive to design with ethics in mind. In other words, as designers (engineers, developers) we should aim to include consideration for the users and potentially the users themselves within the design process to ascertain the best possible solution, supporting physical, social, economic, and psychological needs to promote wellbeing (on many levels) and minimise negative impacts. Secondly, design thinking, learning (problem-based) and education, are common categories in which design thinking is instrumental for both the development of ethical design processes and products, as well as probing ethical aspects of design. Thirdly, values are important. The distinction between design and ethical values is often made, and the importance of aligning design, designer and user and ethical values is integral. Interestingly, while articles representing sustainability do not arise as much within this search category, the citation amounts show that it is a topical subject in design thinking ethics. This stands true also for the theme of aging and returning to the roots of understanding what ethics are considering philosophy and design (pragmatism). "Gender" and "design thinking" The act of searching articles with the search term, “gender” and “design thinking” was fascinating and the results can be seen in Appendix 2. Altogether 17 publications are reviewed from this category (see Appendix 2). The overall citation amount at the time of retrieval was 6’109 from 2001 to 2020. At the time of the literature search, no articles emerging in Google Scholar had specifically focused on gender in design thinking processes. Gender instead seemed to be the result and bi-product of data analyses in which gender was just one of the independent variables. The article with seemingly the most influence in terms of citation quantity (2924) was Dym et al.’s (2005) “Engineering design thinking, teaching, and learning.” The article focuses on the role of design thinking in engineering education, how it has been utilized and how its role should be emphasized. The starting point of article was that the goal of engineering education is to produce engineers who can design. Here, design thinking is treated as a complex field. The paper briefly reviews the history of design thinking and its role in engineering education from different perspectives. Gender was mentioned in their review of related work, then included within their description of collected background information. The article mentioned previous studies regarding gender issues in HCI, in addition to research regarding females in engineering, yet gender was not otherwise dealt with in this article. The second most cited article (1797 citations) was “Design thinking for social innovation” (Brown & Wyatt, 2010). This article focused on how to utilize design thinking in social innovation projects. The authors present an example of innovation work on a local water supply in Ghana. Three pages in this article focus on women, their aspirations, priorities, and design initiatives that were intended to support access to resources and foster family wellbeing. This can be seen in the authors’ references to a free fishing nets initiative for pregnant women and mothers of small children in Ghana, as well as some text discussing internships. While females were prominent within the article, they are framed s receivers of the initiatives of the project. Gender issues per se in terms of the influence of designer gender, or prevailing gender-based ideologies, were not addressed in this piece. The third most cited (928) article, “Design thinking: past, present and possible futures” (Johansson‐Sköldberg et al., 2013) focuses on two design thinking perspectives: managerial and design. The article critically examines the discourse related to design thinking and its different context-related meanings. In managerial area, design thinking is seen as an effective means for instilling creativity and innovation. Regarding design, the paper identifies various discourses. The authors state that different views should not compete against each other, instead they should be developed in parallel. In this article, gender is mentioned in terms of students’ changing views towards gender, and as a personal research interest. Once more, the analysis does not delve deeper in relation to how gender may be read within design thinking practice. Most of the articles focused on either “design thinking” (five articles) or “education and learning” (five articles) (see Appendix 2). Articles in the group “design thinking” highlighted different applications of design thinking, for example team-based design thinking, childhood design thinking, and history of design thinking. Articles related to “education and learning” focus on various applications of design thinking in education and learning, similarly to what is stated above. Two articles focus on design “studying design thinking”. One focuses on studying different approaches of design thinking, and another on measuring design thinking ability. One publication focuses on gender equality and diversity (Christensen, Mahler & Teilmann, 2020) with zero citations. This paper was written within the scope of critical management studies and discusses a GenderLAB workshop. Within the rationale of the article the authors draw on theories of performativity to illustrate the roles of women and men is various design settings. The idea was to accentuate bias and fixations within ‘taken-for-granted’ roles and actions deliberated within cultural and societal norms. These search terms also returned single hits in the following areas: co-creation - co- creation in design thinking; computer programming - utilizing design thinking in computer programming; ethics in design thinking - gender and ethics in design thinking; philosophical view - differences between dualistic and multiplistic thinkers; and social innovation - how to use design thinking in social innovation. "Gender" and "design thinking" and “ethics” Altogether twenty publications are reviewed from this category (see Appendix 3). The overall citation amount at the time of retrieval was 5’087 from 2001 to March 24, 2021. The article with seemingly the most influence in terms of citation quantity (2924) was once again Dym et al. (2005). Dym et al’s main tenet was that design is a complex process that should be harnessed also by engineers, particularly considering the systemic technological future that is emerging. Design thinking is treated as multi-faceted, far beyond any understanding of a five-step method. Besides posing some open questions regarding design thinking, the article does not delve deeply into the nuances of considering design thinking from a critical gender perspective. Brown and Wyatt (2010) was the most cited in this search category. It is seen in this article that issues of gender in design thinking are tightly intertwined with the ethical considerations of wellbeing and equality. The third most cited article (102 citations), “Combining critical reflection and design thinking to develop integrative learners,” by Welsh et al. (2013) focuses on the managerial education domain. The paper argues that combining design thinking with managerial education affords students the possibility to learn additional skills, such as critical reflection and knowledge transformation. The combination of managerial and design thinking education is beneficial for learners. Once more, gender does not arise as a key factor of concern within the study. Once more, education and learning are in the spotlight of papers in this category. Educational fields include entrepreneurship, medical education, reforming education, and education. Two articles focus on design thinking in healthcare. One concentrates on gender-affirming design thinking within medicine protocols and another on caring design as an application of design thinking. Two more articles focus on ethical design thinking. One centers on design thinking and diversity, and another on design thinking and racialized factors. Additionally, these search terms returned single hits in the following areas: design thinking and organization – examining design thinking in organizational changes; design thinking and place - studying the relationship between design thinking and place; design thinking and robotics - using design thinking in robotic design; design thinking and social concept - design thinking on social innovation; design thinking and well-being - design thinking and community well-being; philosophical view - theoretical issues behind the practical approach; and design thinking in workplace - utilizing design thinking in workplaces. "AI" and “ethics” Altogether eighteen publications are reviewed from this category (see Appendix 5). The increase in popularity of intelligent technologies has brought with it an ever present need to consider and address ethical issues (see e.g., Vakkuri et al., 2020; 2021). As seen in user-centered design and development embedded in design thinking, ethical questions are present from the ground up in techniques such as user stories (Halme et al., 2021). The article with seemingly the most influence in terms of citations (414) was Jobin et al.’s (2019), “The global landscape of AI ethics guidelines.” During the past few years various organizations, private, research and public, have been defining ethical principles and guidelines for AI development. The debate is ongoing around what ethical AI means and what requirements, standards and practical issues are required for AI usage. In this research, the authors mapped global agreements and understandings of the area. They discovered that five ethical principles are globally discussed: transparency, justice and fairness, non-maleficence, responsibility, and privacy. Their work revealed that the research and development of this area is very important. Hagendorff (2020, cited 163 times) focused on analyzing 22 sets of AI guidelines and principles. This article echoed the sentiments of Jobin et al’s paper in terms of the necessity of translating the principles into practice, yet additionally suggests ways in which to apply them practically in development. Similarly, to the other two articles, in his much earlier article McLaren (2003, cited 85 times) described the difficulties in concretizing ethical principles into action. Yet, what is significant about this paper is the fact that they explicitly stress the necessity of gathering information about past cases and examples to isolate details about the implicated situations, their contexts and then about the causality of ethical factors during moments of operationalization. McLaren argued that we learn from previous experience to anticipate the future. This follows in a discussion by Musschenga (2005) who discusses the relatively and context-sensitivity of ethics and moral frameworks. Principles and guidelines dominate most of the articles (eight articles) (see Appendix 4) with the main concern of how to apply these in practice. The articles cover topics concerning existing AI related guidelines and principles, their flaws, usage in practice and application. One article is related to ethics in addition to where and how to use AI for doing good instead of harming anyone or anything. The second largest group of articles (4) concentrates on ethics documentation. Articles are related to AI ethics overview, importance of ethics and international cooperation. These search terms returned also single hits in the following areas: automated AI oversight - ethical issues in AI surveillance; concerns - common concerns of AI ethics; and ethical design - the importance of ethical training of AI practitioners. Discussion The results of the literature search performed in this quick review were surprising not in terms of what arose, but what did not arise. The key themes of the review can be seen in Figure 2. There were strong patterns in the themes that arose particularly in terms of design thinking research being connected to educational methods and approaches (Welsh et al., 2013). From the AI ethics perspective, the greatest emphasis was placed on the transition of ethics from principles to practice (Hagendorff, 2020; Halme et al., 2021; Jobin et al., 2019; McLaren, 2003). Then, there were also the papers that represented the philosophical and pragmatic traditions implicated in design thinking processes that are linked to embodied and emotional experience and empathy (Dalsgaard, 2014; Kirkman et al., 2017; Jensen & Vistisen, 2013; Spiekerman & Winkler, 2020). Dalsgaard’s approach through John Dewey’s pragmatic theorization connects with the rationale of design thinking as an effective learning tool. Values are also tied into the emotional weighting of ethics (Spiekerman & Winkler, 2020). Where gender did arise within the results was in relation to equality and social innovation (Christensen, Mahler & Theilman, 2020; Dym et al., 2005). Figure 2: Key themes of review What did not arise was research into distinctions between genders in design thinking processes and attributions or critique of creativity and creative framing of practitioners who engaged in design thinking. One could stop to wonder whether or not findings such as Zwijsen et al.’s (2011) discovery of the overuse and misuse of constructs and ethical assumptions such as the need for human autonomy and self-determinism could have introduced a form of reflectionism that incorporates critical theory and gender at its core for deciphering from where these assumptions arise and what they conceptually denote. Moreover, Jobin et al’s findings of the five pillars of AI ethics - transparency, justice and fairness, non-maleficence, responsibility, and privacy – could be challenged in relation to various readings of these factors, as well as how and by whom they were established. Does gender matter when approaching an ethical design problem through design thinking in terms of possible readings, interpretations, and actor (designer, professional) - subject (consumer, civilian) relationships? One also needs to wonder as to whether the key to effectively developing AI systems within a design thinking process is not simply a matter of concretely (literally) translating theory to action, but rather creatively (laterally) attributing qualities to specific traits and functions. There are several limitations to this research, one such being the reliance on Google Scholar as the main search engine – in order to gain an accurate view on the types of representation, application and framing of design thinking and ethics in research, a larger systematic study of the alternate digital libraries (EBSCO, Science direct, ACM, IEEE etc.) would be beneficial also to avoid biases in the Google algorithms. The addition of researchers and thorough review of all results would additionally give a more rigorous view on the state of ethics in design thinking research. This would also assist in the exclusion of grey literature from the data. In future research it would be interesting to investigate how women have experienced their level of influence in design decisions and the roles they adopt in design thinking processes (i.e., design lead, mediator, facilitator, creator, mentor etc.). Future research will concentrate on empirical interview studies that also gauge how professionals experience potential changes in dynamics that are contingent on gender, role in relation to creativity, customer/user and types of technology. A technological review, and review of AI systems biases may enlighten the situation on how these dynamics affect aspects such as algorithmic logic. Conclusion In this article the authors described a quick review of literature under the terms of “design thinking”, “gender”, “ethics” and “AI” in order to capture a glance of research both the in relation to the connection between design thinking as a method, in relation to: the demographics of those implementing it (designers and gender); the design and technological space that many projects currently focus on and that draw the most ethical attention (AI systems); as well as the ethics in logic behind those systems (e.g., bias caused by lack of diversity in design teams). This mode of approach connects design thinking as a method, to a more cognitive scientific understanding of design thinking as cognition through design – meaning that the thought in and behind the processes informs the design outcome, and a multidisciplinary design team represents an assemblage of collective cognition. Based on the main research interest and professional experience of the authors, “gender” and “ethics” in relation to design thinking was particularly in focus. The authors were interested in understanding how the gender of designers and developers engaged in design thinking methodological processes affect the ethical outcomes of predominant contemporary design projects (these days often linked to AI system development). For this particular study, they conducted a quick review of literature to understand the current state of these considerations in published research. While there were common themes and focuses within the areas of design thinking as method, design thinking and ethics, and even AI ethics, such as design thinking’s application as a method for education and learning in various disciplines – medicine and health care, engineering and organizations. While the results for a straight search in gender and ethics were extremely broad and indirectly related, what was interesting to observe was that at the time of keyword searches, results with these terms (3,3 million) in Google Scholar only ranged between 1996 to 2017. Hits directly on gender per se dropped off, and were overtaken by those more focused on ethics in general. This was coupled by a rise in AI ethics related literature. The authors ponder over whether considerations for gender in the power of design have decreased, or whether consideration for ethics overall automatically incorporates an understanding that gender is one variable among many that need to be accounted for. A curve in popularity of the topic may be attributed to a range of factors, none-the-least current emphasis on predominant seemingly non-gendered (uni-sex) AI ethics discourse. This may represent an algorithmic bias in the field’s development, or potentially embodies a non-binary approach to AI ethics discourse. On this note, “AI” and “ethics” received a significant amount (1,98 million) of hits. The timeframe within two first pages is from 2018 to 2021. This reveals that indeed, research on ethical considerations and applied ethics within the scope of technology development has shifted focus towards intelligent technological development. “Ethics” and “design thinking” revealed a current trend in the field of AI to consider ethics as an important part of development. The search results show that relatively little scholarly attention has been placed on ethics, design thinking and particularly the matter of creativity – as a praxis and discourse – through the lens of gender. Despite projects aiming directly at equality and increased quality of life as seen in for instance, by Brown and Wyatt (2010) and Lopes et al., (2020), ethical issues pertaining to gender representation among designers and developers seems relatively unexplored. In fact, the impact of the young [white] trendy designer/entrepreneur/developer/architect stereotype, myth or fact in predominant software practice and business still appears to be overlooked when discussing the relationship between ethics and AI logic. Some talk has occurred regarding over-representation of this category of developer among Silicon Valley businesses (see e.g., Rangarajan, 2018; Watson, 2016). It would be highly worth applying a cognitive design perspective towards investigating the impact of the relationship between developer/designer demographics and bias within ML and AI system logic (Storey, Fracchia, Müller, 1999). "Gender" and "design thinking" and "ethics" received the lowest number of hits (2280 hits). In most selected articles, gender was a background variable rather than being the focus. This in itself, the authors of this current article feel is interesting, as it underplays the significance of gender difference and gendered logic (no matter how technological or constructed, e.g., De Lauretis, 1987) within the framework of AI systems and how they operate in societal formations and socio-cultural structures. The same applies to the category “gender” and “design thinking”. This seems to be a comparative basis for analyzing results, yet does not actively test how the gender of for instance, designers and/or developers affects the ways in which: a) they engage in design thinking processes – who do practitioners empathize with stakeholders, and does gender matter?; b) how do they see themselves within the process in light of socio-cultural and technological positioning (reflective/reflexive practice and praxis); and c) how do cognitive-affective and reflective processes impact the process outcome – can normative biases be detected, analyzed and somehow resolved? Then, to combine the strong representation of education in design thinking literature with AI ethics research investigating bias, one could pose the question: who are training future designers and developers and how does this reflect in decision-making and design logic? Funding details This research was supported by the Faculty of Information Technology, University of Jyväskylä, Finland, and the Communication Studies research group at the School of Marketing and Communication, University of Vaasa, Finland. Data availability statement The authors confirm that the data supporting the findings of this study are available within the article and its supplementary materials. Declaration of interest statement We declare that this article does not pose any conflicts of interest. References Ailon, G. (2015). From superstars to devils: The ethical discourse on managerial figures involved in a corporate scandal. Organization, 22(1), 78-99. Amit, R., & Zott, C. (2001). Value creation in e‐business. Strategic management journal, 22(6‐7), 493-520. Andrews, D. (2015). The circular economy, design thinking and education for sustainability. Local Economy, 30(3), 305-315. Androutsos, A., & Brinia, V. (2019). Developing and piloting a pedagogy for teaching innovation, collaboration, and co-creation in secondary education based on design thinking, digital transformation, and entrepreneurship. Education Sciences, 9(2), 113. Antikainen, J., Agbese, M., Alanen, H-K., Halme, E., Isomäki, H., Kemmell, K-K., Rousi, R., Vainio-Pekka, H., & Vakkuri, V. (2021). Aligned AI implementation model ECCOLA. In the Proceedings of REST2021. Springer, Cham Audi, R. (2012). Virtue ethics as a resource in business. Business Ethics Quarterly, 22(2), 273-291. Baer, J., & Kaufman, J. C. (2008). Gender differences in creativity. The Journal of Creative Behavior, 42(2), 75-105. Balboni, P., & Macenaite, M. (2013). Privacy by design and anonymisation techniques in action: case study of Ma3tch technology. Computer law & security review, 29(4), 330-340. Beckman, S. L., & Barry, M. (2007). Innovation as a learning process: Embedding design thinking. California management review, 50(1), 25-56. Birkeland, J. (2012). Design blindness in sustainable development: from closed to open systems design thinking. Journal of urban design, 17(2), 163-187. Blizzard, J., Klotz, L., Potvin, G., Hazari, Z., Cribbs, J., & Godwin, A. (2015). Using survey questions to identify and learn more about those who exhibit design thinking traits. Design Studies, 38, 92-110. Boon, M. (2021). AI in Society, and the issue of human dignity. https://research.utwente.nl/en/publications/ai-in-society-and-the-issue-of-human- dignity Bootle, R. (2019). The AI economy: work, wealth and welfare in the robot age. Hachette UK. Boydston, J. A. (Ed.). (1981-1990). Later works of John Dewey: 1925-1953 (Vols. 1- 16). Carbondale, IL: Southern Illinois University Press. Braunack-Mayer, A. J. (2001). What makes a problem an ethical problem? An empirical perspective on the nature of ethical problems in general practice. Journal of Medical Ethics, 27(2), 98-103. Brown, T., & Wyatt, J. (2010). Design thinking for social innovation. Development Outreach, 12(1), 29-43. Buchanan, R. (1992). Wicked problems in design thinking. Design issues, 8(2), 5-21. Burton, E., Goldsmith, J., & Mattei, N. (2015, January). Teaching AI Ethics Using Science Fiction. In Aaai workshop: Ai and ethics. Carmel-Gilfilen, C., & Portillo, M. (2010). Developmental trajectories in design thinking: an examination of criteria. Design studies, 31(1), 74-91. Chandler, D. (2014). Beyond good and evil: Ethics in a world of complexity. International Politics, 51(4), 441-457. Cross, N. (2011). Design thinking: Understanding how designers think and work. Berg. Cross, N. (2018). A brief history of the design thinking research symposium series. Design Studies, 57, 160-164. Cross, N., Dorst, K., & Roozenburg, N. (1992). Research in design thinking. Delft: Delft University Press. Chaudhury, D. (2021). Why The Ethics Of AI Are Complicated. https://www.forbes.com/sites/forbestechcouncil/2021/08/30/why-the-ethics-of- ai-are-complicated/?sh=51ed699643a7 Christensen, J. F., Mahler, R., & Teilmann-Lock, S. (2020). GenderLAB: Norm-critical design thinking for gender equality and diversity. Organization, 1350508420961528. Coleman, E., Shealy, T., Grohs, J., & Godwin, A. (2020). Design thinking among first‐ year and senior engineering students: A cross‐sectional, national study measuring perceived ability. Journal of Engineering Education, 109(1), 72-87. Crilly, N. (2011). The design stance in user-system interaction. Design Issues, 27(4), 16-29. https://www.forbes.com/sites/forbestechcouncil/2021/08/30/why-the-ethics-of-ai-are-complicated/?sh=51ed699643a7 https://www.forbes.com/sites/forbestechcouncil/2021/08/30/why-the-ethics-of-ai-are-complicated/?sh=51ed699643a7 Cutumisu, M., Schwartz, D. L., & Lou, N. M. (2020). The relation between academic achievement and the spontaneous use of design-thinking strategies. Computers & Education, 149, 103806. Dalsgaard, P. (2014). Pragmatism and Design Thinking. International Journal of design, 8(1). Danks, D., & London, A. J. (2017, August). Algorithmic Bias in Autonomous Systems. In IJCAI (Vol. 17, pp. 4691-4697). De Lauretis, T. (1987). Technologies of gender: Essays on theory, film, and fiction. Indiana University Press. Dennett, D. C. (1987). The intentional stance. MIT press. Dewey, J. (2004). Democracy and education. New Delhi, India: Cosmo Publications. (Original work published 1916) Dorie, B. L., Cardella, M., & Svarovsky, G. N. (2014). Capturing the design thinking of young children interacting with a parent. Dorst, K. (2011). The core of ‘design thinking’and its application. Design studies, 32(6), 521-532. Dwivedi, Y. K., Hughes, L., Ismagilova, E., Aarts, G., Coombs, C., Crick, T., ... & Williams, M. D. (2021). Artificial Intelligence (AI): Multidisciplinary perspectives on emerging challenges, opportunities, and agenda for research, practice and policy. International Journal of Information Management, 57, 101994. Dym, C. L., Agogino, A. M., Eris, O., Frey, D. D., & Leifer, L. J. (2005). Engineering design thinking, teaching, and learning. Journal of engineering education, 94(1), 103-120. Eddington, S. M., Corple, D., Buzzanell, P. M., Zoltowski, C., & Brightman, A. (2020). Addressing Organizational Cultural Conflicts in Engineering with Design Thinking. Negotiation and Conflict Management Research, 13(3), 263-284. Eicher, B., Polepeddi, L., & Goel, A. (2018, December). Jill Watson doesn’t care if you’re pregnant: Grounding AI ethics in empirical studies. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society (pp. 88-94). Ericson, J. D. (2021). Mapping the Relationship Between Critical Thinking and Design Thinking. Journal of the Knowledge Economy, 1-24. Etzioni, A., & Etzioni, O. (2016). AI assisted ethics. Ethics and Information Technology, 18(2), 149-156. Feinberg, M. (2007). Hidden bias to responsible bias: an approach to information systems based on Haraway’s situated knowledges. Information Research, 12(4), 12-4. Ferrer, X., van Nuenen, T., Such, J. M., Coté, M., & Criado, N. (2021). Bias and Discrimination in AI: a cross-disciplinary perspective. IEEE Technology and Society Magazine, 40(2), 72-80. Fieser, A. (2020). Ethics. The Internet Encyclopedia of Philosophy. [December 15, 2020] https://iep.utm.edu/ethics/ Fischer-Hübner, S., & Martucci, L. A. (2014). Privacy in social collective intelligence systems. In Social collective intelligence (pp. 105-124). Springer, Cham. Friedman, B., & Kahn Jr, P. H. (2003). Human values, ethics, and design. The human- computer interaction handbook, 1177-1201. Gero, J. S., & Milovanovic, J. (2020). A framework for studying design thinking through measuring designers’ minds, bodies and brains. Design Science, 6. Gewirth, A. (1960). Meta-ethics and normative ethics. Mind, 69(274), 187-205. Giddens, A. (1984). The Constitution of Society: Outline of the Theory of Structuration. Cambridge: Polity Press. Goldschmidt, G., & Rodgers, P. A. (2013). The design thinking approaches of three different groups of designers based on self-reports. Design Studies, 34(4), 454- 471. Goldsmith, J., & Burton, E. (2017, February). Why teaching ethics to AI practitioners is important. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 31, No. 1). Greenwood, M., & Freeman, R. E. (2018). Deepening ethical analysis in business ethics. Journal of Business Ethics, 147(1), 1-4. Grodzinsky, F. S., Miller, K. W., & Wolf, M. J. (2008). The ethics of designing artificial agents. Ethics and Information Technology, 10(2-3), 115-121. Hagendorff, T. (2020). The ethics of AI ethics: An evaluation of guidelines. Minds and Machines, 30(1), 99-120. Halme, E., Vakkuri, V., Kultanen, J., Jantunen, M., Kemell, K. K., Rousi, R., & Abrahamsson, P. (2021, June). How to Write Ethical User Stories? Impacts of the ECCOLA Method. In International Conference on Agile Software Development (pp. 36-52). Springer, Cham. https://iep.utm.edu/ethics/ Hammersley, M., & Gomm, R. (1997). Bias in social research. Sociological research online, 2(1), 7-19. Hamington, M. (2019). Integrating care ethics and design thinking. Journal of Business Ethics, 155(1), 91-103. Haraway, D. (1988). Situated knowledges: the science question in feminism and the privilege of partial perspective. Feminist Studies, 14(3), 575–599. Haraway, D. (2013). Simians, cyborgs, and women: The reinvention of nature. Routledge. Hayes, E. R., & Games, I. A. (2008). Making computer games and design thinking: A review of current software and strategies. Games and Culture, 3(3-4), 309-332. Hickok, M. (2021). Lessons learned from AI ethics principles for future actions. AI and Ethics, 1(1), 41-47. Hoffmann, C. H., & Hahn, B. (2020). Decentered ethics in the machine era and guidance for AI regulation. AI & SOCIETY, 35(3), 635-644. Hoffman, M. L. (1984). Interaction of affect and cognition in empathy. Emotions, cognition, and behavior, 103-131. Hursthouse, R. & Pettigrove, G. (2018). "Virtue Ethics", The Stanford Encyclopedia of Philosophy (Winter 2018 Edition), Edward N. Zalta (ed.). Accessed August 27, 2021, from: https://plato.stanford.edu/archives/win2018/entries/ethics-virtue/. IGI Global: Ethical Stance. Dictionary. https://www.igi-global.com/dictionary/ethical- stance/37711#:~:text=1.,to%20be%20right%20and%20true, last accessed 2021/01/18. Irani, L. (2018). “Design thinking”: Defending Silicon Valley at the apex of global labor hierarchies. Catalyst: Feminism, Theory, Technoscience, 4(1). Jensen, T., & Vistisen, P. (2013). The ethics of user experience design discussed by the terms of apathy, sympathy, and empathy. The possibilities of ethical ICT, 242. Jobin, A., Ienca, M., & Vayena, E. (2019). The global landscape of AI ethics guidelines. Nature Machine Intelligence, 1(9), 389-399. Johansson‐Sköldberg, U., Woodilla, J., & Çetinkaya, M. (2013). Design thinking: past, present and possible futures. Creativity and innovation management, 22(2), 121- 146. Kant, I. (2001). Lectures on ethics (Vol. 2). Cambridge University Press. Kazim, E., & Koshiyama, A. (2020). A high-level overview of AI ethics. Available at SSRN. https://plato.stanford.edu/archives/win2018/entries/ethics-virtue/ https://www.igi-global.com/dictionary/ethical-stance/37711#:~:text=1.,to%20be%20right%20and%20true https://www.igi-global.com/dictionary/ethical-stance/37711#:~:text=1.,to%20be%20right%20and%20true Kimbell, L. (2011). Rethinking design thinking: Part I. Design and culture, 3(3), 285- 306. Kirkman, R., Fu, K., & Lee, B. (2017). Teaching Ethics as Design. Advances in Engineering Education, 6(2), n2. Kitchenham B.A. (2004). Procedures for Undertaking Systematic Reviews, Joint Technical Report, Computer Science Department, Keele University (TR/SE- 0401) and National ICT Australia Ltd. ( 0400011T.1). Klugman, C. M. (2021). Black boxes and bias in AI challenge autonomy. The American Journal of Bioethics, 21(7), 33-35. Kolko, J. (2015). Design thinking comes of age. Harvard Business Review. https://cdn.fedweb.org/fed-42/2892/design_thinking_comes_of_age.pdf Kurtmollaiev, S., Pedersen, P. E., Fjuk, A., & Kvale, K. (2018). Developing managerial dynamic capabilities: A quasi-experimental field study of the effects of design thinking training. Academy of Management Learning & Education, 17(2), 184- 202. Laitinen, A., & Sahlgren, O. (2021). AI systems and respect for human autonomy. Frontiers in Artificial Intelligence, 151. Laclau, E. (2017). Why do empty signifiers matter in politics?. In Deconstruction (pp. 405-413). Routledge. Leifer, L., & Meinel, C. (2018). Introduction: Reflections on Working Together— Through and Beyond Design Thinking. In Design Thinking Research (pp. 1-12). Springer, Cham. Leikas, J., Koivisto, R., & Gotcheva, N. (2019). Ethical framework for designing autonomous intelligent systems. Journal of Open Innovation: Technology, Market, and Complexity, 5(1), 18. Leikas, J., Saariluoma, P., Rousi, R. A., Kuisma, E., & Vilpponen, H. (2012). Life- based design against loneliness among older people. The Journal of Community Informatics, 8(1). Livingston, M. (2020). Preventing Racial Bias in Federal AI. JSPG, 16. Lohr, S. (2018). Facial recognition is accurate, if you're a white guy. In Ethics of Data and Analytics (pp. 143-147). Auerbach Publications. Lopes, M. R., & Vogel, C. (2020, September). Gender Effects in Mobile Application Development. In 2020 IEEE International Conference on Human-Machine Systems (ICHMS) (pp. 1-6). IEEE. MacKinnon, K. R., Ross, L. E., Gualdron, D. R., & Ng, S. L. (2020). Teaching health professionals how to tailor gender-affirming medicine protocols: A design thinking project. Perspectives on medical education, 9(5), 324-328. Madson, M. J. (2021). Making sense of design thinking: A primer for medical teachers. Medical Teacher, 1-14. Magistretti, S., Pham, C. T. A., & Dell'Era, C. (2021). Enlightening the dynamic capabilities of design thinking in fostering digital transformation. Industrial Marketing Management, 97, 59-70. Mai, J.-E. (2016). Three Models of Privacy. New Perspectives on Informational Privacy. Nordicom Review, 37(special issue): 171-175. doi:10.1515/nor-2016- 0031 March, J.G. (1972). Model bias in social action. Review of Educational Research, 42(4), 413-429. Mascaro, S., Korb, K., Nicholson, A., & Woodberry, O. (2015). Evolving ethics: the new science of good and evil. Andrews UK Limited. McGinn, C., Bourke, E., Murtagh, A., Donovan, C., Lynch, P., Cullinan, M. F., & Kelly, K. (2020). Meet Stevie: A Socially Assistive Robot Developed Through Application of a ‘Design-Thinking’Approach. Journal of Intelligent & Robotic Systems, 98(1), 39-58. McLaren, B. M. (2003). Extensionally defining principles and cases in ethics: An AI model. Artificial Intelligence, 150(1-2), 145-181. McLuhan, M., & Fiore, Q. (1967). The medium is the message. New York, 123, 126- 128. Melles, G., de Vere, I., & Misic, V. (2011). Socially responsible design: thinking beyond the triple bottom line to socially responsive and sustainable product design. CoDesign, 7(3-4), 143-154. Mentzer, N. (2014). Team Based Engineering Design Thinking. Journal of Technology Education, 25(2), 52-72. Mill, J. S. (2010). The basic writings of John Stuart Mill: On liberty, the subjection of women and utilitarianism. Modern Library. Mortati, M., Villari, B., & Maffei, S. (2014). Design Capability for value creation. In 9th DMI: Academic Design Management Conference (pp. 2490-2512). Mulvenna, M., Boger, J., & Bond, R. (2017, September). Ethical by design: A manifesto. In Proceedings of the European Conference on Cognitive Ergonomics 2017 (pp. 51-54). Murphy, P. E., Laczniak, G. R., & Wood, G. (2007). An ethical basis for relationship marketing: a virtue ethics perspective. European journal of marketing. Murty, P., Paulini, M., & Maher, M. L. (2010). Collective intelligence and design thinking. In DTRS’10: Design Thinking Research Symposium (pp. 309-315). Musschenga, A. W. (2005). Empirical ethics, context-sensitivity, and contextualism. The Journal of medicine and philosophy, 30(5), 467-490. Newell, A., & Simon, H. A. (1972). Human problem solving (Vol. 104, No. 9). Englewood Cliffs, NJ: Prentice-hall. Nichols, S., & Mallon, R. (2006). Moral dilemmas and moral rules. Cognition, 100(3), 530-542. Nilsson, Å. W., & Jahnke, M. (2018). Tactics for norm-creative innovation. She Ji: The Journal of Design, Economics, and Innovation, 4(4), 375-391. Norman, J. (2001). Design as a framework for innovative thinking and learning: how can design thinking reform education?. Design and technology educational research and curriculum development: The emerging international research agenda, 90. Norwood, J. M., & Paterson, A. (2002). PROBLEM-SOLVING INA MULTIDISCIPLINARY ENVIRONMENT? MUST ETHICS GET IN THE WAY OF HOLISTIC SERVICES?. Clinical Law Review, 9, 337. ÓhÉigeartaigh, S. S., Whittlestone, J., Liu, Y., Zeng, Y., & Liu, Z. (2020). Overcoming barriers to cross-cultural cooperation in AI ethics and governance. Philosophy & Technology, 33(4), 571-593. Oxford Languages. (2021). Bias. https://www.google.com/search?q=bias&rlz=1C1GCEB_enFI966FI966&oq=bia s&aqs=chrome..69i57j0i512l5j69i60l2.711j0j4&sourceid=chrome&ie=UTF-8 Overmyer, T., & Carlson, E. B. (2019). Literature review: Design thinking and place. Journal of Business and Technical Communication, 33(4), 431-436. Panke, S., & Harth, T. (2019). Design thinking for inclusive community design:(How) does it work?. Journal of Interactive Learning Research, 30(2), 195-214. Parsons, G. (2015). The philosophy of design. John Wiley & Sons. https://www.google.com/search?q=bias&rlz=1C1GCEB_enFI966FI966&oq=bias&aqs=chrome..69i57j0i512l5j69i60l2.711j0j4&sourceid=chrome&ie=UTF-8 https://www.google.com/search?q=bias&rlz=1C1GCEB_enFI966FI966&oq=bias&aqs=chrome..69i57j0i512l5j69i60l2.711j0j4&sourceid=chrome&ie=UTF-8 Pellegrino, E. D. (1995). Toward a virtue-based normative ethics for the health professions. Kennedy Institute of Ethics Journal, 5(3), 253-277. Peter, S. C. (2018). Reduction of CO2 to chemicals and fuels: a solution to global warming and energy crisis. ACS Energy Letters, 3(7), 1557-1561. Prince, J. D. (1904). The code of Hammurabi. The American Journal of Theology, 8(3), pp. 601-609. Retrieved May 5, 2020, from: https://www.journals.uchicago.edu/doi/pdfplus/10.1086/478479 Rangarajan, S. (2018). Inequality, Silicon Valley diversity – here’s the clearest picture of Silicon Valley’s diversity yet: It’s bad. But some companies are doing less bad. Reveal. Accessed August 2, 2021, from: https://revealnews.org/article/heres-the-clearest-picture-of-silicon-valleys- diversity-yet/ Reniers, R. L., Corcoran, R., Drake, R., Shryane, N. M., & Völlm, B. A. (2011). The QCAE: A questionnaire of cognitive and affective empathy. Journal of personality assessment, 93(1), 84-95. Rességuier, A., & Rodrigues, R. (2020). AI ethics should not remain toothless! A call to bring back the teeth of ethics. Big Data & Society, 7(2), 2053951720942541. Riska, E. (2008). The feminization thesis: Discourses on gender and medicine. NORA— Nordic Journal of Feminist and Gender Research, 16(1), 3-18. Rivers, I. (2000). Reason, Grace, and Sentiment: Volume 2, Shaftesbury to Hume: A Study of the Language of Religion and Ethics in England, 1660–1780 (Vol. 37). Cambridge University Press. Roselli, D., Matthews, J., & Talagala, N. (2019, May). Managing bias in AI. In Companion Proceedings of The 2019 World Wide Web Conference (pp. 539- 544). Rousi, R. (2021). Ethical Stance and Evolving Technosexual Culture – a case for human-computer interaction . Invited Paper. HCII 2021. Springer, Cham Rousi, R., Leikas, J., & Saariluoma, P. (Eds.). (2020). Emotions in Technology Design: From Experience to Ethics. Springer Nature. Runco, M. A., Cramond, B., & Pagnani, A. R. (2010). Gender and creativity. In Handbook of Gender Research in Osychology (pp. 343-357). New York, NY: Springer. Rowe, P. G. (1991). Design thinking. Cambridge, MA: MIT press. https://www.journals.uchicago.edu/doi/pdfplus/10.1086/478479 https://revealnews.org/article/heres-the-clearest-picture-of-silicon-valleys-diversity-yet/ https://revealnews.org/article/heres-the-clearest-picture-of-silicon-valleys-diversity-yet/ Ryan, T. (2009). Teaching and Technology: Issues, Caution and Concerns. Handbook of Research on New Media Literacy at the K-12 Level: Issues and Challenges. Saward, M. (2021). Democratic Design. Oxford University Press. Saariluoma, P., Alanen, H. K., & Rousi, R. (2021). Fashion Technology: What Are the Limits of Emerging Technological Design Thinking?. In Advances in Intelligent Systems and Computing. Springer. Saariluoma, P., & Rousi, R. (2020). Emotions and Technoethics. In Emotions in Technology Design: From Experience to Ethics (pp. 167-168). Springer, Cham. Shapira, H., Ketchie, A., & Nehe, M. (2017). The integration of design thinking and strategic sustainable development. Journal of Cleaner Production, 140, 277-287. Singer, M. G. (1963). The golden rule. Philosophy, 38(146), 293-314. Singer, P. (2011). Practical ethics. Cambridge university press. Sohail, M. (2017). How to practice ethical design thinking – a methodology to encourage designing for human values. Accessed August 27, 2021, from: https://medium.muz.li/how-to-practice-ethical-design-d8a6a8dcf4b0 Spiekermann, S., & Winkler, T. (2020). Value-based engineering for ethics by design. arXiv preprint arXiv:2004.13676. Stefaniak, J. (2020). The utility of design thinking to promote systemic instructional design practices in the workplace. TechTrends, 64(2), 202-210. Steen, M. (2015). Upon opening the black box and finding it full: Exploring the ethics in design practices. Science, Technology, & Human Values, 40(3), 389-420. Stoltzfus, G., Nibbelink, B. L., Vredenburg, D., & Hyrum, E. (2011). Gender, gender role, and creativity. Social Behavior and Personality: an international journal, 39(3), 425-432. Storey, M. A., Fracchia, F. D., & Müller, H. A. (1999). Cognitive design elements to support the construction of a mental model during software exploration. Journal of Systems and Software, 44(3), 171-185. Svinhufvud, L. (1998). Finnish textiles en route to modernity. In Finnish Modern Design: Utopian Ideals and Everyday Realities 1930–97 (pp. 181-207). Yale University Press. Swanson, D., & Santamaria, L. (2021). Pandemic Supply Chain Research: A Structured Literature Review and Bibliometric Network Analysis. Logistics, 5(1), 7. Tan, C., & Wong, Y. L. (2012). Promoting spiritual ideals through design thinking in public schools. International Journal of Children's Spirituality, 17(1), 25-37. https://medium.muz.li/how-to-practice-ethical-design-d8a6a8dcf4b0 Tavani, H. T. (2013). Ethics and technology. Wiley. Thoring, K., & Müller, R. M. (2011). Understanding design thinking: A process model based on method engineering. In DS 69: Proceedings of E&PDE 2011, the 13th International Conference on Engineering and Product Design Education, London, UK, 08.-09.09. 2011 (pp. 493-498). Tilburge University (2021). Scholarly books vs. scholarly articles. https://libguides.uvt.nl/humanities/books-vs-articles Timmers, P. (2019). Ethics of AI and cybersecurity when sovereignty is at stake. Minds and Machines, 29(4), 635-645. Tom, K., Jonathan, L., & Tom, P. (2001). The art of innovation: lessons in creativity from IDEO, America’s leading design firm. Treiber, M. (2021). We invited an AI to debate its own ethics in the Oxford Union – what it said was startling. https://www.ikangai.com/we-invited-an-ai-to-debate- its-own-ethics-in-the-oxford-union-what-it-said-was-startling/ Tsai, M. J., & Wang, C. Y. (2020). Assessing Young Students’ Design Thinking Disposition and Its Relationship With Computer Programming Self- Efficacy. Journal of Educational Computing Research, 0735633120967326. Tu, J. C., Liu, L. X., & Wu, K. Y. (2018). Study on the learning effectiveness of Stanford design thinking in integrated design education. Sustainability, 10(8), 2649. Tuan, N. T., & Shaw, C. (2016). Consideration of ethics in systemic thinking. Systemic Practice and Action Research, 29(1), 51-60. Utriainen, T., Salmesvuori, P., & Kupari, H. (2014). Introduction: critical and creative turns. In Finnish Women Making Religion (pp. 1-17). Palgrave Macmillan, New York. Vakkuri, V., & Abrahamsson, P. (2018, June). The key concepts of ethics of artificial intelligence. In 2018 IEEE International Conference on Engineering, Technology and Innovation (ICE/ITMC) (pp. 1-6). IEEE. Vakkuri, V., Jantunen, M., Halme, E., Kemell, K. K., Nguyen-Duc, A., Mikkonen, T., & Abrahamsson, P. (2021). Time for AI (Ethics) Maturity Model Is Now. arXiv preprint arXiv:2101.12701. Vakkuri, V., Kemell, K. K., Kultanen, J., & Abrahamsson, P. (2020). The current state of industrial practice in artificial intelligence ethics. IEEE Software, 37(4), 50- 57. https://libguides.uvt.nl/humanities/books-vs-articles Val, E., Gonzalez, I., Iriarte, I., Beitia, A., Lasa, G., & Elkoro, M. (2017). A design thinking approach to introduce entrepreneurship education in European school curricula. The Design Journal, 20(sup1), S754-S766. Van de Ven, B. (2008). An ethical framework for the marketing of corporate social responsibility. Journal of business ethics, 82(2), 339-352. van der Westhuizen, D., Conrad, N., Douglas, T. S., & Mutsvangwa, T. (2020). Engaging communities on health innovation: experiences in implementing design thinking. International quarterly of community health education, 41(1), 101-114. Van Langenhove, L. (2017). Varieties of moral orders and the dual structure of society: A perspective from positioning theory. Frontiers in Sociology, 2, 9. Verbeek, P. P. (2017). Designing the morality of things: The ethics of behaviourguiding technology. Designing in ethics, 78. von Thienen, J., Clancey, W. J., & Meinel, C. (2021). Theoretical Foundations of Design Thinking. Part III: Robert H. McKim’s Visual Thinking Theories. Design Thinking Research: Interrogating the Doing, 9. Verhulsdonck, G., Howard, T., & Tham, J. (2021). Investigating the impact of design thinking, content strategy, and artificial intelligence: A “streams” approach for technical communication and user experience. Journal of Technical Writing and Communication, 51(4), 468-492. Wajcman, J. (2000). Reflections on gender and technology studies: in what state is the art?. Social studies of science, 30(3), 447-464. Wajcman, J. (2007). From women and technology to gendered technoscience. Information, Community and Society, 10(3), 287-298. Warnecke, T. (2016). Capabilities, human development, and design thinking: a framework for gender-sensitive entrepreneurship programs. Review of Social Economy, 74(4), 420-430. Watson, R. (2016). In Silicon Valley, young white males are stealing the future from everyone else. The Guardian. Accessed August 2, 2021, from: https://www.theguardian.com/commentisfree/2016/may/27/in-silicon-valley- young-white-males-are-stealing-the-future-from-everyone-else Weisman, L. (1994). Discrimination by design: A feminist critique of the man-made environment. University of Illinois Press. https://www.theguardian.com/commentisfree/2016/may/27/in-silicon-valley-young-white-males-are-stealing-the-future-from-everyone-else https://www.theguardian.com/commentisfree/2016/may/27/in-silicon-valley-young-white-males-are-stealing-the-future-from-everyone-else Welsh, M. A., & Dehler, G. E. (2013). Combining critical reflection and design thinking to develop integrative learners. Journal of Management Education, 37(6), 771- 802. Whittlestone, J., Nyrup, R., Alexandrova, A., & Cave, S. (2019, January). The role and limits of principles in AI ethics: towards a focus on tensions. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society (pp. 195-200). Williams, O. F., & Murphy, P. E. (1990). The ethics of virtue: A moral theory for marketing. Journal of Macromarketing, 10(1), 19-29. Zoltowski, C. B., Buzzanell, P. M., Brightman, A. O., Torres, D., & Eddington, S. M. (2017, June). Understanding the professional formation of engineers through the lens of design thinking: Unpacking thewicked problem of diversity and inclusion. In ASEE annual conference & exposition proceedings. Zwijsen, S. A., Niemeijer, A. R., & Hertogh, C. M. (2011). Ethics of using assistive technology in the care for community-dwelling elderly people: An overview of the literature. Aging & mental health, 15(4), 419-427.