Mpp through recording has started I that their local ahead and introduce your first So good afternoon, everyone, and thank you for joining us today. Our speaker is there some massive in, and she's a sociotechnical researcher whose work has been at the intersection of various should be important in design problems and in the context of the global south. And today we're going to be listening to her talk about re-imagining AI fundamentals to work or go resource communities. And there has been visiting us off and on through the years since she graduated. A few years ago afterwards. Well, she she's a Georgia Tech alum. She got her masters and then went on to do her PhD at UC Irvine. And that worked at Google for many years and has recently quit and is proud me unaffiliated and is happy to. Hey, Questions about that if you'd like. After the talk. Of course, if you have any questions about that, feel free at any clarification questions that it might make sense to pause in the middle for. We're happy to do that and you can put your questions in the chat. Otherwise, I'll just hold on to the questions until the end of the talk. And I think we might end there. You might end close to maybe five or 10 minutes before our time is upgrade. So if it takes a few extra minutes, we're happy to to extend the time for that. But obviously, if you have class, if you have to go, then that's also fine. So yeah, they excited to have net there she is. And she's an old friend and colleague and really an amazing research. And doing really bachelor work has been for many, many years. So always great to have you here. And with that knit that landed over to you. Thank you for that generous introduction. It's always a pleasure to be back here at Tech and to see many old faces in the new ones as well. So let me get started with sharing my screen first. Okay, so my talk today is about the myopia of model centrism guy. And my talk is pretty substantial. It runs for a while. So I appreciate any clarification questions during the talk and any big questions to to be held off until the end. And the ideas and arguments in the talk represent my worldview and results from my research and do not represent Google's views. So in this talk, I present research insights and ask fundamental questions of how AI is developed and impacts the humans involved, particularly the most vulnerable. I start with these words from Jiu question Muti, the Indian philosopher. What is important is to put forth a fundamental question, not to find an answer. And if we are capable of looking at the fundamental question without seeking an answer, then the very observation of the fundamental brings about. So I hope to leave us all with several open-ended questions throughout the stock. Many scholars today argue that AI research and practice places novel model development. That is, the mathematical algorithms at the center. New model architectures and bigger and better state of the art performances are celebrated. Criticism has been raised about how the overt emphasis on models has reached a point of ridicule with even members of the AI community starting to call it alchemy. Empirical challenges to being one, incremental and leaderboards. While the AI animal communities appear to lionized models, dataset work is rejected in leading conferences illustrating to us what is considered science and what is not, what is valued and what is not. Ai models are also measured up against clean and large datasets without noise. Primarily to write papers and to report performance of that are high. But the real world is dynamic and noisy. Early progress in AI, which is the low-hanging fruit of quantity alone, no longer applies to harder and more subjective problems and edge cases like cancer predictions and my, my talk today will focus on high-stakes domains. Of this nature. And AI is increasingly applied to the high-stakes areas which have critical safety impacts on living beings, including humans, wildlife plans, and the environment. Several of these high-stakes AI projects seek to intervene in low resource contexts in fragile, complex, and economically poor domains. For example, vision statement from one of our research participants and AI researcher wants to diagnose tuberculosis from x-rays in 10 seconds instead of 15 days. And in regions with one doctor put 30 thousand people. Now, noteworthy as this may sound, several of these high-stakes air projects are high modernist, like this one above which demonstrate strong confidence in the potential for scientific and technological progress as a means to reorder the natural world. Now it is against this backdrop where I situate my research. Resource constraints, make certain aspects legible. That will humans are enrolled in the service of the machine and low-resource areas in the form of domain experts, annotators, feel partners, and even users. Now the downstream impacts of these projects can be huge regardless of the success or failure. Specifically, I've been interested in answering the questions of how does the machine made, what does this process include or exclude? What are the practices of developers building AI systems in high-stakes areas? How are the crucial non-modal aspects like data steam by AI researchers and developers in their system development. All the various human contributors perceived and engaged with by the developers. And what do these practices tell us about whether AI is even a good fit for high-stakes areas. In order to answer these questions, I had been studying up the AI practitioners and researchers building cutting edge high-stakes AI systems in low resource contexts from around the world. Studying up, which some of you may be familiar with is a technique from anthropology of observing the practices of professionals. But also quite literally here, moving up the pipeline of machine learning. So in today's talk, I'm going to focus on a few projects, a qualitative study with AI researchers in the US, India, and various Sub-Saharan African countries. As well as a study which scholars, activists working on social justice in India on their concerns about responsible AI. My vantage point is that of a, those geotechnical researcher having worked with marginalized communities in the Global South in order to build technologies for the past 15 years, I discuss both the technical aspects of how these systems are made, as well as a sociopolitical aspects of how humans are in Ward and impacted by these systems in the global south. So there four parts to my talk. Starting with data cascades and then domain expertise and the de-skilling around it. And how we can re-imagine algorithmic fairness when these centered from the West. And what does, what are some practical implications or AI as well as allied communities like HCI, ICT, and many others. So starting with data cascades, an AI, this was joint work with several Google researchers. This is awarded a best of chi last year. The data is the critical infrastructure necessary to be a systems. It can determine performance, robustness, scalability, safety of, and fairness of AI systems. Paradoxically, for AI researchers, data work is often the least incentivized aspect. It is viewed as operations relative to the glamorous work of building models. Data is often viewed as a databases, legal or licensing issue, but we believe that human computer interaction has an important role to play in elevating data and data work as first-class citizens of AI system building. Though in our research, we observe and define data cascades as compounding events that cause negative downstream effects from data issues that result in technical debt over time. In our study with researchers and developers, 92% reported experiencing at least one leader cascade in their system development. 45 percent experienced two or more cascades. Data cascades are complex, long-term, occur frequently and persistently. There are big in the diagnosis and manifestation with no clear indicators, tools and metrics to detect and measure the effects. In fact, several AI practitioners tend to use system level metrics like F1 score, AUC, and not any metrics that put into data itself. Data cascades are triggered when conventional practices are applied in high-stakes domains like viewing data as operations, moving fast, hacking model performance without consideration for data quality or undervaluing domain expertise and labor involved in the production of the system itself. Data cascades can have negative impacts on the AI development and deployment, including burnout of relationship harm to beneficiary communities, discarding entire datasets and performing costly iteration. But cascades are often avoidable by stepwise an early interventions houses. So we observe four types of cascades. I will discuss two for brevity here, they're discuss their, their detailed and the paper. To starting with interacting with physical world brittleness. High-stakes AI systems are typically brittle. They closely interact with previously on digitized environments like air quality sensing, ocean sensing, ultrasound scanning, leading to even more reasons for a model to break. Like limited training data, complex underlying phenomena, volatile domains and changes to regulations. Data cascades of this nature can appear in the form of hardware, environmental and human knowledge drips. So as an example, even the slightest movement of a traffic camera you would wind can result in a model failure in detecting traffic patterns and accidents and violations on highways. Artifacts like fingerprints, shadows, dust on the lens. Improper lighting and pen markings can affect predictions. Rain and wind and environmental conditions can impact image sensors in the wild, leading to incorrect model results. Conventional AI Practice of training on pristine data but deployed in the messy real world, ability to trigger these cascades. And data cascades of this type took the longest to manifest, taking up to two to three years to emerge, almost always in deployment. Now again, this all ties back to the, the, the development process and the metrics that are used where caskets can go uncashed and undetected for a very long time and result in these disastrous effects in some cases. Another type of data cascade will set off by lack of documentation across various cross organizational relationships within the organization with field partners, with data collectors and external sources. So there are several cases where practitioners inherit, inherited datasets that lack critical details. And missing metadata can lead practitioners to make assumptions, ultimately leading to discarding their datasets or recollecting data. For example, in the case of medical robotics, a lack of metadata and collaborators changing schema can lead to very precious medical data being just discarded entirely. As high-stakes data tend to be niche and specific with various underlying standards and conventions. Data set collection, even minute changes can render a dataset unusable. No conventional AI Practice of neglecting the value of data documentation and field partners not being made aware of achieving good quality in the datasets appear to set these off. No cascades here became visible through manual reviews, often by chance. And the impacts of these cascades can include wasted time and effort being blocked on building models and discarding entire datasets. So our results point to those sobering prevalence of messy protracted an opaque data cascades troublingly in high-stakes two means. Cascades point to the contours of a larger problem of residue will conventions and perceptions and AI and ML drawn from the world of big data, of abundant digital resources, of Model valorisation, of moving fast to proof of concept and viewing data as grunt work in the workflow, we need to move towards a proactive focus on the values, practices, and politics of humans in the data pipeline. We need to move from the current approach of goodness of fit towards model to goodness of data as well. From doing more and collecting more data to doing better with data. We need to innovate on structural incentives to recognize and reward data work in conference tracks, organizational recognition of data glue work, greater collaboration with data collectors and domain experts, which will be the next part of my talk. And shared benefits for ML developers as well as field partners. Data ethics and practical data work. Oversight boards like the IRB and ethics standards should be a part of ai, education and praxis. And finally, we need greater data equity in the global South where our research points to how developers have access to open and pre-trained models. And even the education is, is fairly accessible and democratized. It is with data and compute where we see the greatest disparity is in models that are being built for low-resource context versus high resource contexts. So this won't let you had a bunch of practical impact. The pair team released a guidebook with a entirely rewritten data chapter, which is, which is the state of the art guidelines around working with data in the machine learning development pipeline. New lips introduced a dataset track for the first time, a recommendation that we make in the paper calling for larger structural changes in AI. And it appears to be motivated by our work. And we're opening up a new research agenda on interfaces, processes, and policy for data excellence in AI. And this work has also been implemented in 10 products at Google internally. So with that, I move to the second part of my talk on de-skilling of them, domain expertise in eye development. This is joint work with a faculty member, largish, weird argument at Georgetown. Now, field workers are necessary for AI projects in areas with limited infrastructure where there are limited existing datasets that are available. So data often needs to be collected from scratch and it's hard to do it digitally. And so humans are often enrolled and these humans are typically field workers. It is standard procedure in building AI and low-resource areas to partner with field organizations. And they workers like community health workers, agricultural extension workers, microfinance, NGO workers in order to bootstrap as well as in maintaining and deploying these models. Now these four field workers are known to be underpaid and overworked. For example, an asha worker in India, a community health worker, can perform a mindboggling array of 720 tasks. An EIA data collection is often additional to their primary, primary responsibilities and want to call attention to as Russ Miles work here on dissertation work you are on the experiences and perceptions of health workers in India. Now the disregard for local domain expertise in model building is fairly well understood in AI. The sample court, every time I fire a linguist, the performance of the speech recognizer goes up by an OB researcher, Frederick genic, demonstrates to us how the domain expertise is viewed. Given that model seek to emulate an aggregate local expertise of field workers. To what extent is the worker's expertise acknowledged, engaged weight, accredited by the AI developer in building the model. This was our research question. We find that despite the mastery and tacit knowledge of field workers that in fact takes decades to build. And that which the AI model seeks to emulate. Developers, reduce field workers to data collectors for the Uber export models, we find that most developers and researchers do not have any first-hand contact with, provide training or compensate field workers for their data label. With the absence of direct interactions with domain experts as we call them. In most cases, the work of the field worker was primarily read through dataset artifacts and glitches in the dataset to be precise, AI developers attributed poor data quality to poor what practices of field walkers and be present for conceptions of field workers as held by developers. Build workers as corrupt, lazy, non-compliant, and as dataset itself, which is where the researchers do not have an understanding that there are actually humans enrolled in creating or labeling the datasets. Even though it's just one hop away. Field workers were perceived to come in the way of the worthwhile model development efforts of developers. I forgot to mention that this is a chi 2020 to paper. So this will print, will, you will see this paper in greater detail at the conference. So in despite the limited engagement and understanding of field workers, developers reported creating several disciplinary interventions in order to influence data, to influence field workers to collect better quality data. In the form of surveillance, gamification, cros verification, and pre-processing fixes. Now, our work does not concern itself with verifying the veracity of the judgments of AI developers. Rather, our focus is on reporting the very existence and the prevalence of these views and biases and the subsequent enactment of disciplinary and punitive interventions on field workers by developers. Despite their seminal contributions to the AI systems. Often for free, albeit limited consent and transparency. To borrow from James Ferguson, the political scientists and anthropologists data collection in AI operates as an anti-politics machine, making political decisions about what constitutes expertise and creating technical solutions to technical problems through management interventions to influence field workers, to connect good quality data. We argue that field workers should be accurately viewed as domain experts with special knowledge and mastery in their local contexts. The AI development apparatus in low resource contexts, these skills and invisible isis domain experts in low-resource in these areas. Even though the models in our study sought to emulate and improve over an aggregate over the expertise or field workers. The expertise itself was treated as non essential knowledge by the developers. The workhorses, knowledge, networks and capabilities were viewed as assets were dataset collection rather than necessary expertise for system building. Now, AI developers who are experts in the technical fields, but not in the application domains. For example, or develop a product or creating of cancer prediction model will inevitably leave gaps when domain expertise is excluded from the model that seeks to learn the expert's knowledge in the form of data. The data quality issue as perceived by AI developers is only an issue if we think of domain experts in these limited ways, that is, as data collectors. But if we wanted to re-imagine domain expertise as an essential partnership throughout the AI pipeline, we can see new possibilities for collecting, modelling, and scaling knowledge. The domain experts can contribute to critical questions that can impact model behaviors. What exactly are the modelling? What assumptions are appropriate? What features should be included in the model? What are we trying to predict? How will we know in the partnership and data and model building with the domain experts can help identify and alleviate algorithmic bias. Data cascades an unintended outcomes that remained typically latent and opaque for long time periods. To recall for viewing domain experts as full range experts in their communities. And to find participatory ways to partner with domain experts in the ML lifecycle from problem formulation to measuring impact subsystems. Good problems for models are not always good problems for the world. In, in a few cases. In our research we found that motivated by this premise of field workers are collecting poor quality data. Developers and researchers actually went on to build AI models that detected outliers in the domain experts in, in, in the data set that was collected by the workers. And we're passing on this information of outliers or corrupt field workers to supervisors in power who could then act upon them. Now we argue that it is possible to interpret the perceptions of worker corruption and sabotage as actually acts of worker resistance to exploitative conditions of basically unpaid labor that is being done to create datasets for AI. Modifying James C. Scott, memorable phrase, way, but these are the weapons of the data collected. Now instead of motivating and overworked health worker to do more work for AI data collection. One might ask how to help them achieve their goals better, how to prioritize their numerous visits. Better capture of the institute knowledge, allocate medical and limited medical resources better and more appropriately, and build visibility into their contributions. Indeed, this is one way to realize the shared goals of both the AI researcher and developer and the domain expert on human development in low-resource areas. Behavioral tricks for improving data set collection like surveillance and gamification, will likely only cause long-term damage and resentment among workers. We need better recognition and attribution of domain experts contributions. And I'll run this up towards the end of the dock on further applications. So moving to the third part of my talk. Now, this was a paper, at fact, last year on re-imagining algorithmic fairness in India and beyond. Again, joint work with several collaborators at Google Research. Now, conventional algorithmic fairness make several assumptions that are based on Western institutions and infrastructures. For example, a successful outcome from fairness research, which is bands on facial recognition. For example, in Auckland, New Jersey, San Francisco, among many other places, comes from a concerted effort across decades of scientific research on western populations. Public datasets, APIs and Information Acts for researchers to analyze model outcomes and ML industry and research. Community, including academia that's fairly responsive to bias reports, government representatives that are glued into technology policy, and an active media that systematically scrutinizes and reports on AI impacts in the real world. Now we argue that the assumption is about the relevance and efficacy of these factors can fail and even be harmful in much of the world. Fairness research remains rooted in Western concerns and histories. Subgroups like race and gender. Datasets like ImageNet and WordNet. Measurement scales like the Fitzpatrick scale, and legal tenets that sometimes come from civil law. However, conventional algorithmic fairness is increasingly copy pasted an espoused in AI policies in countries like Tunisia, Uruguay, India, and more without paying attention to the local contexts of deployment. Now without engagement, with the conditions, values, politics, and histories of the non-risk, algorithmic fairness can be a tokenism at best, pernicious, at worst for the low resource communities and the non-Western communities. If algorithmic fairness is to serve as the ethical compass of AI. And indeed this has become the locus in which all these important conversations and developments are happening in the AI community, it is imperative that the field recognize its own default, biases and blind spots in order to avoid exacerbating historical hands that it purports to mitigate. We must take pains not to develop a general theory of algorithmic fairness based on the study of western populations alone. Though the questions we ask are, could fairness have structurally different meanings in the Northwest? Could phase frameworks that rely on Western infrastructures be counter productive elsewhere? How do social, economic, and infrastructural factors influence fair machine learning? We contribute an analysis of end-to-end algorithmic power in India. And we provide a starting roadmap or algorithmic fairness when centered in India, a non-western country. So as I mentioned, we focus on India, the world's largest democracy. Increasingly on paper, a multiethnic, multilingual, post-colonial nation, and the world's second largest Internet using population. Despite the rise of high-stakes AI deployments in India in areas like predictive policing and welfare disbursements. There is no substantial responsible AI tech policy or a substandard body of research in the country. To one of our key arguments is that the distance between ai models and disempowered communities in India who, who, whom the models hope to serve. The distance is actually quite large because of technical distance, social distance, ethical distance, temporal distance, and physical distance. And I'll explain this a bit more. But a myopic focus on simply localizing or contextualizing the output of a model for a particular contexts can perhaps backfire on the communities as well as the researchers. The three themes that challenge existing assumptions around ML fairness when centered in India, starting with data and model distortions, biases and models are often attributed to bias datasets. And there's a notion of fixing the data to fix the bias. But data itself can be missing in the form of digital divide where half of India does not have access to the Internet. And this was really pronounced during the time of COVID, the initial phase of COVID, where the government introduced a contact tracing app on which only half of the country could get online. So what was the efficacy of contact tracing? Excluded half is primarily women, rural communities, and Artemis sees who are the indigenous tribes of South Asia. Now my prior body of work on gender equity online was motivated by the fact that two-thirds of countries worldwide have more men than women online. And in India at the time of the research, only 29 percent of internet users were women. And when we see these forms of disparities in other subgroups as well. Data can also be missing due to class, gender, and cost inequities in accessing and creating online content. And as a result, any digital data is a representation of the privileged of data. Practices are common in India, which can happen outside of a network. They through phone calls which may not count as data as measured by these systems. Data can also be missing due to artful use of practices to manipulate algorithms. Motivated by concerns like privacy, abuse and reputation. Socioeconomic and demographic data sets at the national, state, and municipal levels are frequently unavailable to the public. Especially datasets featuring migration, unemployment, incarceration, or education divided by different subgroups. And this is a major impediment to doing any kind of ML fairness research in the country. Moving on to proxies which are commonly used in in, in fairness research. Proxies can be similar to the west, but have different implementation nuances due to the fact that India is a pluralistic country. For example, a name is the most semantically meaningful proxy which can communicate cost, gender, religion, class, and ethnicity in India, zip codes are rather heterogeneous. Unlike some portions of the west, which have a history of red lining. With slum housing and Porsche housing often are butting each other in urban areas of the country, mobility, which is another commonly used proxy, has been reported to be much lower for women due to safety concerns and for people with disabilities due to limited infrastructure like ramps. Traditional occupations may correspond to costal religion. Proxies may not even generalize within the country, like asset ownership. Owning a two-wheeler scooter motorcycle in the city of Mumbai presents something very different from owning the same motorcycle in a rural part of Maharashtra, the state within which Mumbai is located. Ai systems. And this is an important point I wish to make in India, remain under analyzed for biases, mirroring the limited public discourse on oppression. Now this is in stark contrast to the public discourse in countries like the US, Canada, and even parts of the EU where we see anti-racism and indigenous concerns arising to the fore. And that creates a greater accountability and pressure for AI researchers and developers to pay attention to these issues. And finally, we introduce axes of discrimination that are specific to the Indian context, such as cost, religion, gender identity, and more, which can be a starting point for fairness exams. Moving on to the second theme, which is double standards and distance from ML makers. Now, users are perceived to be the bottom billion data subjects and given poor recourse in, in systems, effectively limiting their agency. Whereas industry and research is somewhat responsive to use a buyer's reports in the West. Because an agency of Indian users, especially those disempowered by caste, religion, or gender, is poorly addressed in AI systems. Now these systems appear to remove street-level bureaucrats, administrative officers, frontline workers, which I'm referencing the second part of my talk. These other human infrastructures that play a crucial role in providing recourse to marginalize Indian communities. The hi-tech, a legibility of AI renders recalls out of reach for groups marginalized by literacy, education, or legal capital. And Muslim bodies are used as test subjects for AI surveillance. Now this picture on the right here, it comes from a story on how human efficiency trackers, which you can see here, the image of the wearable, are deployed among Dalits, sanitation workers, not Dalits are the most oppressed in the caste system and even considered to be outside of the caste system in India. And sanitation workers and in this case come from these communities. Now these, these trackers are deployed onto the workers and they are equipped with microphone's, GPS cameras and a SIM card, which has led to women workers avoiding restaurants for fear of camera capture, or walkers waiting for the tracker to die before going home. While Indians are part of the AI and machine learning workforce and, and Research UK, me. A majority of, of Indians work in AI and ML services like data integration and business process outsourcing. And the minority engineers often come from privileged class and caste backgrounds. Now this limits remediation of distances and representation of oppressed communities and the work of AI developers and researchers. My final theme is that of unquestioning AI aspiration area is aspirational and readily adopted in high-stakes domains, often too early in India. This, this may be true of many parts of the global south as well. For example, the facial recognition service used by the Delhi police has been known to have 20 percent accuracy and cannot distinguish between boy and girl children. And it was used to arrest 1100 protesters against the citizenship build in January of last year. Now, for reference, a standard threshold of 80 to 95% is recommended for law enforcement AI contrast with 2%. The statement from the government was, this is a software. It does not see faith, it does not see clothes. It's the only sees the face and the face the person is caught. While algorithms may not be trained on subgroup identification. Proxies can correspond to Dalits are the bosses and Muslims just proportionately. Under trials and prisons are largely from these communities and non-literate members. Now in, again, in contrast to the, the focus of research and implementation on explainability, transparency, and interpretability. We find that AI solutions in India had end-to-end opacity with unknown data are non-modal behavior and unknown inferences. And there's a lack of an effective ecosystem of tools, policies, and processes that can help stakeholders interrogate these AI solutions, which is again a major impediment in ensuring meaningful fairness. I want to bring attention to a paper by Shivani NOT IN and reducing minor on data used in New Delhi, predictive policing, where the technique used for data collection was to physically hangout in call centers to observe what went into dataset entry and what biases and perceptions of the of the call center operators when in how they influenced the data. It's a remarkable paper, but this is a non, it's an unscalable method for various questions along the ML pipeline and effectively eliminates ways of inquiring into different aspects of AI fairness in India. Take journalism in India appears to couple app launches and take business deals and do not scrutinize algorithms and they impacts sufficiently. This is again in stark contrast to the contribution of journalistic entities like the Guardian, New York Times, ProPublica, and several others that have raised the created this course and raised awareness around these issues and brought public attention and allow the public to demand accountability around the issue. So a fair ML strategy for India needs to reflect it's deeply plural, complex and contradictory nature and needs to go beyond model fairness. You to model and data distortions, we must combine datasets with an understanding of context. Marginalized communities need to be empowered in identifying problems, specifying fairness expectations, and designing systems to avoid top-down fairness. In guess heterogeneity means that a fair ML researchers commitment should go beyond the model output into creating systems that are accessible to the community. Like the fatal Union Carbide gas leak, which is one of the world's worst industrial accidents. 1984. Unequal standards, inadequate safeguards and dubious applications of AI in the non-Western can lead to catastrophic effects. For fair ML research to be impactful and sustainable, it is important to create a critically conscious ecosystem through solidarity with the various policy makers, civil society and journalists. Context matters. We must take care not to copy paste the Western normative ML fairness everywhere. Borrowing from Arturo Escobar, we must evolve blue reverses of ethics to flourish. This work went on to create a new agenda around responsible AI in the global south. And there's some follow-up projects, the results of which you will see soon, hopefully on detecting bias season models that are centered in India. So I'm going to move on to the implications and allow a bit of time for questions. So we find that empirically find that there is an enormous focus on model development among AI researchers and developers due to various incentives like publication prestige, how AI residencies are evaluated, competitive differentiation in this tight market and so on. Yeah, education is now accessible, but remains largely focused on bodybuilding. Not addressing the real world challenges of collecting data or deploying and measuring systems, which practitioners increasingly have to do so the gap between what is taught in, in the curriculum and what an air practitioner ends up doing is increasingly large and in fact quite dangerous as, as developers and researchers start to take on high, highest stakes and problematic subjects. Some practitioners report hacking model performance to look good to funders, clients and conferences with less regard for the underlying safety or quality of the systems. This is all what focus on models often comes at the cost of ignoring important questions around the role of disempowered communities drafted into building or using these systems. This is problematic as AI systems increasingly seek to intervene in domains by governments. The whole society and policymakers have historically struggled to respond. Now this view comes from the conventional ML Pipeline, which typically begins with an available, perhaps unclean dataset and ends with a model evaluation or deployment, typically in a sanitized system or a laboratory setting. Now it is clear that this scope of what is considered to be AI must evolve. Metrics are currently driven by concerns of machine needs, human efficiency, cost, and outperforming the industry standard, which often relies on the expertise of developers, rather than appropriate metrics to evaluate claims relevant to affected communities as decided by the domain experts and by the communities themselves. Non-modal aspects like data, domain expertise and meaningful safeguards are considered to be outside of the scope of AI. Relegated to operations. Where's the elite status of AI system building is restricted to the high priest of developers, leaders of partner organizations, celebrity scientist, bureaucrats. And disturbingly, the machine intelligence itself. Data work is undervalued despite its importance in AI systems, essential experts with rich history are largely seemed, seemed to be automated, reduced or punished, or dataset collection to build Uber export models which may eventually displace them in the long run. The emphasis on algorithmic fairness, which, which is, which hinges upon model fairness largely offers a veneer of credibility. But when examined closely, these frameworks can be dangerously symbolic and possibly harmful to non-western communities in which they were not conceived them. More work is needed in all of these areas. We need to move from model at the center to ecosystem thinking. While there has been much needed progress on including impact statements in AI and ML papers, introducing the new datasets track and saw a lot more has to evolve to reflect the on the ground practices in AI data what needs greater recognition and inclusion. Our expertise is defined. Who is considered to be an expert? How expertise fed into models is acknowledged and credited are all important questions to answer. When we press upon AI developers and researchers need to embrace more participatory stances with domain experts and humans in general, and not rely upon control and punishment, which is what seems to be the norm. Unregulated surplus label from the various humans. Contributing to AI poses new questions. There are long chains of human beings that are involved in enabling model development in AI and low resource areas. Should attribution, transparency, provenance, and compensation evolved to include this surplus and often free labor. Economic measurements of AI, such as the opening vision statement on detecting tuberculosis in 15 seconds instead of 10 days. Measure the before and after effects of algorithms that are dropped into social settings. However, they miss out on measuring how domain experts and workers were involved in, in one and drafted into hidden label. Whether and how communities and expertise was displaced or redistributed as a result of the model. In the short and long term. We need to expand the parameters of what gets measured and seen in AI analyses in order to fully comprehend the multi-dimensional effects of these systems. Ultimately, AI systems need to be seen as interventions or existing infrastructures, especially in loading source context and not as sanitize deployments. I want to end with this quote from David Attenborough, the renowned conservationist at COP 26. Is this how our tail is to end a tale of the smartest species doomed by the all-too-human characteristic of failing to see the bigger picture. Pursuit of short-term goals encode whether it is climate change, industrial capitalism, or it's modern offspring of AI models. Can we start to see the bigger picture before it's too late? I hope so. I have a lot of people to thank. Happy to hear any comments or take any questions at this point. Thank you. That was a great talk and not think about it. There are two questions in the chat right now. I will invite others to add their questions. We are getting close to the end of time here as if people need to be. We understand. But let's keep going pronounced. So there's a question from Amy Chen who says, thank you for your wide-ranging engaging an important presentation with all the different considerations urease for AI development and use. Do you see any differences and implications for private and public sector actors and applications? Would you mind repeating the question, the last Patricia? Yeah. Yeah. I mean, it's basically saying that with all the consideration that you're raising around developing and using AI, are there differences and implications for private and public sector actors and applications? That's an interesting question. I had been privy to private sector having been in the system for awhile. My view is that I mean, there's the what should be done. But the current that if events are such that public sector seems to be mirroring the private sector. In fact, I want to go back to my point about the aspirational use of AI in the global South. Where nation-states are embracing AI for the technological efficiency and modernity and the potential impact to the economy without sufficient scrutiny. And are in fact embracing the same narratives that are created by private sector on, on why this is also high food and it needs to be. There's the fear of missing out, though, here on the ground it seems. And again, this is an off-the-cuff reaction. Or it seems like there isn't much of a difference between the actions of private sector and public sector. Because of the various incentives that everybody has. I think of course, the public sector has in terms of implications and what can be done. The public sector has far greater, I guess, freedom and risk scope for responsibility if they choose to consider that in, in, in creating these accountability structures like in civil society, the journalism, the, the open datasets that can allow us to scrutinize these models and so on. There's a number of different ways in which in which progress can be achieved. I think primarily when driven by the public sector and will not probably come from the public private sector, which is motivated by capitalism. Thank you, Nick, that key says this is brilliant work with, yeah, yeah, and that has long needed a critical look at both algorithmic fairness in data work. And asks on the first part of your talk, I was struck by the dean of the model creators, but the data workers. Where do you think this originates from? One PRE I have is CS in quotes, has historically devalued what seen as applications work. Basically anything that's not purely algorithmic or has a human, our domain element to it. What are your thoughts? That's a great point, Keith. And I appreciate your point about how the historical development of the field of computer science. And I think that that is certainly eat reflects in, in our interviews and our research on how AI, AI itself is considered. I hope, I hope to have shown an arc of how this all how these perceptions seem too actually. I have effect on the ground in terms of the domain experts in the communities and so on. But academically and in terms of the, the science of doing AI research, It's definitely seems like there's a, there's a strong bend towards reporting model performance and hence that, that motivates practitioners in the field to, to game it in order to be legitimate and accepted and be competitive and so on. And on the other hand, I think also in practice. And especially having research on a very human-driven part of AI system building, which is in low-resource areas where a lot of people are recruited. It definitely seems like AI researchers perceive this to not be their responsibility. And 2, and for this to be someone else's responsibility, it's not clear who that someone else's. Even when the data in human beings are literally one step away and the data that is collected by them has enormous impacts on the behavior and the performance of the AI system itself. Though there's, I think structurally the field is designed such that models are king. As well as these various other incentives keep, keep these roles and agreements simply about what is considered AI, what is not AI? Who should be credited and so on. Okay, And then one final question. So from TLA, people responsible for adopting a new AI in a high-risk situation, assess whether data cascades are occurring and what are some preventative measures avoid these cascades. Yeah, that's another good question. We discuss in detail in the, in the discussion section of the people and data cascades on how cascades can be prevented. And there were a few exceptions in our interviews with cascades were indeed prevented. It just requires intentional practice. And in firstly, just recognizing that data is more than just more than just a vehicle to get to model, model performance. In high-stakes domains. There's a, I think of course it starts off with where's the data coming from? Who has contributed to the data? What's the relationship with them? What's the engagement with them? You know, can can they be training and there'd be transparency on the downstream use cases, on the potential applications. Can they be composition on what metrics can be used in these systems? And then there are step-wise system building actions that developers can take two, rather than doing system wide, doing stepwise text to ensure that data is well documented, it is fair, it is reliable, it is comprehensive. And performing various checks in in that in the pipeline itself. And then again, I think towards the end, rather than being surprised by cascades, I think they can be very proactive set of measures that can be taken to anticipate cascades, to be able to create the instrumentation to measure them. And if all the steps were performed correctly until then I think there's a lower probability, but one should anticipate. One should no longer be surprised that data can cause disasters, they think. And set it up for the long-term. But there's more in the people. Okay. So that thing at the end of the questions and we're outside of pain a few minutes at a time. And there are others who see. Thank you. Thanks Nick, there until the next day and then we'll put forward to what you do now post school girl. Okay. Okay. Bye, everybody. Thank you, everyone.