Understanding Potential Sources of Harm throughout the Machine Learning Life Cycle
Published on:
My reaction to a case study of how South Asian cultural representations can be generalized by generative AI to an outsiders western gaze from limited training with South Asian culture.
Case Study:
Understanding Potential Sources of Harm throughout the Machine Learning Life Cycle
Summary
The case study discusses how South Asian cultural representations can be generalized to an outsiders gaze from limited training in the lens of western ideals. It mentions specific AI models that produce images that are either not up to date or just inaccurate representations of countries and cultures like Bangladeshi and Pakistani.
Discussion Questions
The following are my answers to various questions asked at the end of the case study.
1.What does cultural representation mean to you, and how might this definition and your experience with past representation impact how you would evaluate representation of your identity in generative AI models? What aspects of your identity do you think you would center when evaluating representations in AI model output?
I grew up with a lot of family members being adopted, like my dad. There were adoptions from two colombian kids from my grandparents, so this brought a whole new culture into our family because we were white. It is important to educate, have accurate and diverse cultural representation so there are no harmful stereotypes that can fester into real life situations like getting a job. AI can be bad with holding biases that score less with minorites. If I wanted generative AI to have an accuratae representation of myself I would say it would need to get my look right, skin color, face shape, body. It would also need to get down my beliefs, manner, and morals.
2.What do you think is the role of small-scale qualitative evaluations for building more ethical generative AI models? How do they compare to larger, quantitative, benchmark-style evaluations?
Small scale qualitative evaluations should be used in context specific evaluations for figuring out if something is unethical. Interviews can also be used to get more diverse opinions on it. “Ultimately, participant aspirations for generative AI focused heavily on restoring agency and community control over the terms of representation” (https://mit-serc.pubpub.org/pub/bfw5tscj#nyx1jxwa3mc). However, the smaller evaluations do are not broad and are specific issues. Larger, quantitative, benchmark-style evaluations are good for seeing how much biases occur in broader areas, and/or over time. They can be more efficent if done correctly. I think its good to use a combination of the two, to ensure accurate biases.
3.Participants in this study shared “aspirations” for generative AI to be more inclusive, but also noted the tensions around making it more inclusive. Do you think AI can be made more globally inclusive?
Yes, I think if done right, it can be. There are a lot of problems and issues to look out for. “Left unchecked, algorithmic systems can scale existing regimes of representation through patterns of over- and underrepresentation in algorithmic systems” (https://mit-serc.pubpub.org/pub/bfw5tscj#nzxy9iyi0ua). There needs to be more development of AI reasoning in the context of other cultures, this was mentioned by one of the participants in the case study. “why can’t we imagine a more authentic world that our communities can build ourselves?” (P32, Pakistan). Maybe each country can contribute to a globally inclusive AI. This way, it will get most if not all different cultures.
4.What mitigations do you think developers could explore to respond to some of the concerns raised by participants in this study?
Like I mentioned above there could be leaders from different communites, cultures, and countries that are involved in the design and evaluation of generative AI models. Developers could also use regional institutions to adapt base models to their own cultural contexts, and use cultural evaluation frameworks that are both qualitative and quantitative. Also, representation is dynamic and socially constructed, inclusivity must be treated as an ongoing process, and these AI models should be regularly retrained and reviewed as culture evolves.
5.As mentioned in this case study, representation is not static; it changes over time, is culturally situated, and varies greatly from person to person. How can we “encode” this contextual and changing nature of representation into models, datasets, and algorithms? Is the idea of encoding at odds with the dynamism and fluidity of representation?
The AI design would have to allow for iterative implementation. Datasets could be treated as living archives that change through community reviews, and documentation of when and how data were collected. Models can use feedback systems that allow users to flag outdated or harmful representations, prompting regular retraining that accounts for shifting social norms and values.
6.How can we learn from the history of technology and media to build more responsible and representative AI models?
It would be wise to spend more time on training and prep work before releasing the AI into the public. On two separate occasions Twitter has released AI chat bots that act like users, both times they had spout racist, and sexist remarks. Developers of AI should also implement cultural technologies, meaning using historical awareness with the AI pipeline, with questioning who creates datasets, and which cultural narratives are amplified or erased.
A New Question
Will there ever be a point in which AI has a 100% accuracy of biases? Can we, ourselves, ever achieve complete accurate biases?
I ask this question because I was thinking about how AI will always have a margin of error. With that being said we ourselves are far from perfect. I am curious how combining our inaccurate biases with inaccurate AIs can furthur exacerbate these problems.
Reflection
Reading this case study made me think about how little I think about problems like misrepresentation. This is probably due to me being apart of western culture. I am also white so that furthur pushes this thought. I don’t get directly effected by it, because in a way I am apart of the benefiters of these AI biases.