I’m vas Bednar and I’m the host of lately at Globe and Mail podcast and I’m Katrina onstead the executive producer of lately and it’s Pride this month in Canada happy Pride Katrina happy Pride to you vas from one strike girl to another yes in Toronto this is a very big event right lately is on the business beat so let’s talk business last year 3 million attendees came to the city and pride generated a combined tax revenues of 231 plus million dollars also maybe as many good parties so there are more parades and events to come in Vancouver Calgary all across the country this summer pride is really an economic Powerhouse absolutely and we’re actually going to take a look at a slightly different but adjacent economy everybody in the lately crew was talking about this wired article by the writer reys Rogers it’s called here’s how generative AI depicts queer people and it takes a look at how tools like open AI Sora responded two prompts so a prompt could be something like show me a queer person and we looked at the answers and they were pretty sanitized there was a lot of airbrushed kind of plasticky looking people with fluffy colored hair that tended to be purple and honestly when you take a look at these images you realize like they’re pretty ridiculous yeah but you know then again AI fails aren’t exactly a shocker right AI Imaging is chronically dissatisfying to most of us and it’s that unrealness that you just mentioned that plasticity that’s always the tell that something is AI right there’s always either too many fingers or not enough fingers like something with the digits is always wrong I think of Kate Middleton’s daughter’s strange thumbless Stumpy hand I don’t know if that was Photoshop or AI the jury is still out but what the wired article was pointing out is how the stakes are very different for minority communities and we were curious what is AI still getting wrong or right when it shows us lgbtq plus people and why does it matter we’re talking to Dr zabina vber they’re a computer scientist and an organizer with queer in AI which is a global group of lgbtq researchers and scientists that advocates for better queer representation in AI zabina explains how we got here the fascinating history of these images how AI is only as good as the data gobbles up and in the early days it was gobbling up a lot of hypersexualized and negative depictions of gay life from porn or violent news reports but there was another thing that happened when we were working on this episode there was a lot of online chatter amongst Tech workers and leaders about a so-called shift away from an insti Embrace of diversity equity and inclusion programs to kind of replacing them with Merit excellence and intelligence standards so from Dei to Mei Alexander Wang who’s the founder of scale AI announced that he’s formalized his company’s quote Mei hiring policy and other people seem to be signing on to this and this news just bumped up against the conversation that we were having in and outside of slack about how to make computer models more reprentative and mindful and it kind of raised this new question which is can Tech leaders embed thoughtful representation in their models if they’re moving away from the same principle in their workplace yeah it’s a shift to keep an eye on definitely but on a more positive AI note zabina did mention a favorite online AI art project called the zzy show this is a deep fake drag Cabaret it leans into the creepiness of AI and is extremely fun and addictive we recommend yes we will put the link in our show notes as our present to you our guest is Dr a a VOR and this is lately so alongside your many academic achievements you also dabble in standup comedy can you tell me like an AI joke or or something in your repertoire oh um let me think about it for a second um I did a Show recently like two or 3 days ago and my opener there was like AI gets a pretty bad reputation these days people expect AI to take their jobs make love to their wife and reprogram their electric toothbrush to commit election fraud I think that’s the best I can do what I like about that joke is it it punctures that fantasy of AI yeah but for lgbtq plus people what’s the the reality how do today’s AI systems Miss for the queer Community yes I mean I recently wrote a blog post about a little experiment I run where I had chat GPT um generates stories about straight and queer characters okay so my prompts were very simple I gave either the prompt tell me a story about Thomas who’s straight or tell me a story about Thomas who’s queer and for each of these prompts I just had chat GPT generate a handful of answers like 10 to 20 answers and what was really interesting is that the stories that were generated about Thomas who is straight were really stories where his sexual orientation played no role whatsoever like the word heterosexual a stay didn’t even feature in these stories they were just stories about okay some guy who lives either in a village or in a big city who sails a boat adopts the dog learns to play the guitar you know goes fishing or something yeah exactly like a wide variety of things that are like stereotypical stories they were like stories you could read in a children’s book right okay but Thomas was queer only ever received one kind of narrative he was always born and raised in a small town figures out that he’s queer like by dreaming of knights in shining armor or something like again very stereotypical his community initially rejects him but then they come around to accept him as he is and he always ends up with some sort of romantic partner and they’re happy ever after okay so yeah you can see that it’s really just reproducing one stereotype of what a queer person story is like and it totally ignores what other things might be going on for Thomas and it points to more like a fundamental problem that Ai and queerness has and where they Clash let’s talk about that fundamental problem right and maybe how it’s reinforced so I think there are two puzzle pieces here one is that the fundamental thing about how machine Learning Works is that it learns statistical patterns from large amounts of data right and the more data you have the better your performance will be imagine you have a big collection of music and you want to classify is this a pop song is this A rock song is this a jazz song right the more tagged data you have the better your tagging system will eventually become at distinguishing a pop song for a rock song for example the second part of the puzzle is queer people are a minority right depending on what statistics we use 10 to 20% of people fit under this umbrella somewhere so queer people will always be under represented in data because they’re under represented in societies they will always be just a small part of the data set and this is kind of the problem where Ai and cre identities Collide because these AI systems learn well what they see many examples for but when you only see very few or very stereotypical or negatively biased examples of queer representation in your data then this is what the model will learn and the model will even amplify and replicate these things can you walk me through some of the history with these systems how did today’s representations or Reflections or conjurings of queer life evolve from the very first days or earlier days of AI because it strikes me that it’s not a new problem we sort of keep seeing it in different ways so in the earlier days when people query or interact with models and specifically trigger them on queer terms they would get very negative depictions of queer people like a colleague of mine at the University of Edinburgh did a great paper on how trans people are depicted by image ation systems and that was two or three years ago before the iterations that we see now and there the case was that if you prompted these systems with image of trans woman or image of trans man you would get very dehumanized and sexualized pictures and the reason for that was that lots of these image data sets were scraped from the internet without much oversight or filtering and lots of the images came from porn s sites where things were attacked as trans man trans woman but those were pornographic pictures of trans people and so this was biasing the system and that led to quite a sexualized dehumanized end result then when we look at the early iterations of chat GPT for example or other big chat Bots when they were prompted with queer terms like explain to me what gay means or what is a gay man these systems would often answer with things like I can’t talk about this I am just a chat bot right so we see there that this is a fil that had been put in deliberately by the people who released these systems to a wider audience because they knew that if you prompted these systems bad things would come out because bad things were in there so instead of letting that happen they had this filter but the filter again was harmful in its way because if you query hey what a straight you would get a definition for that you would have discourse about that it was just kind of putting the queer phobia at a different level and where are we now I’m thinking of mid Journey a generative AI service that creates images and artwork from natural or Simple Text prompts at the moment we are kind of at the third step like the next iteration of this so now when we query mid Journey we get these really stereotypical depictions of queer people so we now can assume that all the harmful and dehumanizing narratives have been erased but in instead they were kind of supplanted with this really polished and stereotypical Narrative of queerness so to say let’s talk about that stereotypical images of queerness what does that tend to look like what is this so-called course correction yes so what we found is that the depictions are basically models so you you have imagery that is really reminding one of a pride ad from a car company the people are all like skinny Tan in their mid 20ies with wonderful flowing hair and they are holding rainbow flags and and that kind of thing or um purple hair I noticed exactly the purple hair is a really interesting thing because there is this one haircut that gets reproduced over and over again by the image systems which is a person with kind of shaved short sides and the curly floofy top and it’s always is purple or pink right so from a computational perspective is it possible or maybe philosophically worthwhile to even try to capture the fluidity of human sexuality and gender through algorithmic systems like what could satisfying representation look like or is this something that is kind of impossible for us to get to I think from a computational perspective perspective these models have something that I consider to be a fundamental flaw and that is they have a hard time distinguishing what features are visible or which features are not or what features are influential in say a narrative about a person or not because generally we are all aware about stereotypes that exist like what a gay man looks like what a lesbian woman looks like and so on but also at some point Our Lives we learn that you can’t see a person’s sexuality by looking at them we know that a politician my grandpa my primary school teachers these people could be gay or trans I wouldn’t be any wiser right because these are things that you can’t see in somebody’s face yeah whereas an image generation system only has pictures and tags and if you tag a person’s depiction with queer then the model would just associate this despite the fact that we kind of philosophically believe that queerness isn’t something that is visible on the outside so I think the next satisfying step would be to train models that are actually able to distinguish these things the way that we humans are able to distinguish these things that like if I want to generate a picture of a pride parade it will be a picture of people and I know okay Pride parades will probably have rainbow flags and identity Flags but that’s the point and lot lot of the models that we build are built on assumptions that are not necessarily true like one assumption is for example that you can only have two genders like male and female this gender is determined at Birth and remains immutable throughout your life another thing is that your name should be immutable throughout life these are things that are baked into the systems that we interact on a day-to-day basis and that lead to lots of harm when you are somebody for whom that’s not true like we know that trans and non-binary and intersex people exist and they generally fall through the cracks of these systems which leads to them being treated differently or being treated worse so the dominant kind of stereotypical image of the tech world is the straight white guy in the hoodie or maybe a vest right even vest for summer I want to talk about the presence of queer people in directly developing and informing AI systems so the tech Hub of the world unfortunately it’s not in Canada arguably in San Francisco so a lot of gay people there Sam Altman the CEO of open aai is openly gay as is Peter teal former PayPal current paler and there are lots of lgbtq plus programmers and developers in the field have queer people made an impact on the technologies that try to capture their experiences and likeness I think first of all a diverse Workforce is is absolutely essential to building good products because if we want to build a product that serves a wide range of people we need to be able to anticipate their needs on the other hand just being queer and a programmer is not like a Magic Bullet that solves all diversity problems but there’s a limit to what we can do and also what an organizational structure would allow because again despite there being many queer people in San Francisco I’m sure there are still minority I’m still sure that you will not find a company where 80% of people who work there are queer or maybe you had like I would love to work for them you know what never say never yeah we are getting to speak to each other during pride month happy Pride by the way and for a while pre pandemic at least we saw large technology firms putting significant amounts of money into what they refer to as trust and safety and a lot of these programs have been starting to be shut down or really kind of Contracting for instance Twitter or X I have trouble calling it that cut more than a third of its trust and safety team is the awareness is the in-house awareness and attention to queer representation in technology firms slowly downshifting do you feel it and do you think it’s going to change technology I must must say personally I haven’t seen it as much because I’ve been booked for like two Pride events this pride month at companies that I consider to be quite large but again those things can start slowly like maybe next time we ask for funding for like a conference event people will say ah like you know our priorities have shifted I think it points more to a different thing it points to cooling of the hype because you can put money into things that are considered frivilous or nice to have as queer representation has always been it was always business first and then we want to put sprinkles on it to make it look nice it never has been for any company I’m sure the core of their business principle to be nice to queer people I very much doubt it so I think it points to the fact that there is a fatigue setting in from the AI and large language model Hypes and that companies are becoming aware of that and cutting where it is easiest for them okay so easy to reduce those teams at a time where maybe people are demanding or looking for better representation or inclusivity could you tell me a story about how the application of or engagement with a facial recognition system has played out in the life of an lgbtq plus person so facial recognition is a really tough topic to begin with because I think there are many reasons to say that as a technology and in its application it’s just something we shouldn’t do a trans friend of mine said that as soon as she started hormone replacement therapy the passport gates at the airport stopped working for her and this is like a thing where navigating international travel as a trans person is already an absolute mindfield where you have body scanners that will Mis categorize your body and be like oh you have a female gender marker but like what is this let’s do like a very invasive surge of your private parts and if you have yet another thing where like an automated system will hand you over to suspicious law enforcement who think that your gender representation is actually an act of Disguise or an act of somehow doing something nefarious I think what is important is just what are the consequences if your face is elgible to a facial recognition system what happens if your face is more likely to to be misrecognized or to trigger some insecurity in the model okay that happens when like these models are trained on image data sets that are not really representative of minorities and that is the problem you will have a system that probably accurately recognizes a white man but that will see for example a trans person and say like okay something is weird here either they get misrecognized as someone else or they they get recognized as in danger alert something’s weird right and that is when those systems trigger intervention and more surveillance or background checks or search okay so I want to understand more because you said we shouldn’t engage with facial recognition technology at all why not what’s so bad about it well I mean ontologically there is nothing inherently wrong about recognizing a face that’s something that we as humans do all the time it becomes Authority issue when we see the massive harmful potential that this technology has and like this technology Works especially bad for women of color because they are underrepresented in these data sets and knowing what we know about the marginalization of women of color being misread or misrecognized by these systems puts them at a greater danger of like surveillance and um negative interactions with law enforcement for example or if these systems are implemented in healthcare situations or border crossing situations do you think that these algorithmic programs these systems will eventually catch up and adapt and be improved as machines learn from more users more bodies more faces more people who may be transitioning or or have transitioned are things moving fast enough or should they just stop I mean while being a queer Advocate I’m also a researchers and I know that things shouldn’t stop we should try build better system and research systems better but I think we should really interrogate when facial recognition is necessary and if it’s necessary because it is just a very powerful tool for Mass surveillance that I personally think is very concerning especially in a political climate that we’re experiencing where like the most recent election in Germany for example has had quite a right shift and there are far-right parties coming into power and having a database of people’s names faces movement profiles could have been benign in the hands of like a democratic government but far-right parties can become owners of these databases and we know that they don’t have anything good in mind for queer people or for immigrants people of color and so on so I think these kinds of massive repositories of personal data should not exist in the hands of anybody neither commercial actors nor for governments okay you wrote in a blog post for queer and AI that data sets haven’t moved beyond the word gay being an insult and that queer is sort of coded as a bad word in scrape data sets why is that and when are we going to change those tags so when we collect all of this data and we just put it in a pile and me using gay to mean like awesome and cool gets thrown in the same pile as a Nazi using gay to mean despicable and terrible this is all the same pile that these models learn from so when we live in a world where the majority of usages of the word gay are negative usages then this will be learned by the models if in our training data gay is used as an insult it will categorize it as an insult if it’s used as an insult in 80% of times and 20% of times it’s used as a good thing it will go with the majority representation there is another level where people build these systems and there are curse word filters and those still contain queer identity terms specifically so like sometimes those words will not even show up big media Outlets will be the ones that provide the text websites for example that have only few incoming links like my blog for example or other queer media they will not even be in these data sets they will not be scraped in the first place I wanted to touch on audio for a second I wondered if you were familiar with Q the gender neutral voice I want to play it for you is that okay okay yeah do it okay here we go hi I’m Q the world’s first genderless voice assistant think of me like Siri or Alexa but neither male nor female I’m created for a future where we are no longer defined by gender but rather how we Define ourselves so this product which is a collaboration between several veral different companies has existed for about 5 years the hope is that it’ll be adopted as one of the potential defaults on our voice activated assistance what do you make of it I think it’s really cool I I was not familiar with this I think the first level is why are all of our voice assistants female right because it is the role that a voice assistant hat is basically like The Helpful little secretary that is just nonthreatening and helps you right mine’s an Australian man that’s what I set it to when I need directions anyway yeah so see like whatever feels comfortable for you to get directed from but these companies probably imagine male audiences that might be threatened by a male voice giving them orders but that will not be threatened by just a cute female voice being nice and helpful this is just reflecting the roles that these companies Envision another thing is the idea that voices have gender is I think also something that’s not exactly straightforward because especially for Trans people voice is a big issue like I for example think my voice is wrong oh W I’m rolling with that right like I wish it was different but it would take lots of voice training or like hormonal instruction and so on this is a thing that people actively live with because I know that my voice really influences how I’m read we are really still stuck with this very binary view of gender and the whole idea of being non binary or being transgender is kind of that these categories are social categories they’re not biological they’re not pre-ordained and generally I think the whole idea that oh this is a male face this is a male voice this is a female face this is a female voice is really just like enforcing gender stereotypes and what if instead we could expand our notion of what a man or woman looks like or what a non-binary person looks like to that end what does an AI future that’s more lgbtq plus inclusive look or sound like well first of all it should be not owned by large companies but it should be owned by the people who are most impacted by it in my kind of very Rosy unicorn happy future view or what I want to work towards as a researcher is that we make tools that we can give communities who want to solve a problem my favorite examples for that that I like to cite all the time is this AI drag show that drag artists in London started creating during the covid pandemic because they couldn’t do shows anymore and so like a large chunk of their income was impacted by that yeah so they took the recordings of their drag performances and created this AI drag queen that people could interact with online oh wow and so the whole process like from the Inception from the data that was used to train from who interacts with it all that was in the hand of of the impacted community and this is my vision because I’m kind of tired of being dependent on large companies considering queer people as a worthwhile audience or consumer that be like okay let’s make our voice assistant attractive for queer people by doing this thing yeah because again if the money runs out if the hype runs dry this is the first thing to be cut and this is not surprising to anybody of course I mean I love efforts towards queer people being considered people and considered customers and considered worthwhile right it’s great when companies do that but I think there’s an equal amount of effort to be put into open- Source initiatives into Research into Grassroots organizations because we can rely on ourselves to take care of ourselves it is I think safer and more solid than waiting for big companies to like give us breadcrumbs I’m going to chase your digital breadcrumbs to that AI drag show it sounds pretty rad Dr zabina vber thank you so much thank you was a pleasure to be here you’ve been listening to lately a glob and mail podcast our executive producer is Katrina onstad the show is produced by Andrea varsen and our sound designer is Cameron mver I’m your host best Bednar and in our show notes you can subscribe to the lately newsletter where we unpack a little more of the latest in business and technology a new episode of lately comes out every Friday wherever you get your podcast
Everyone loves an AI fail, like a few extra fingers on a generated image. But what happens when the flaws of this nascent technology are much more serious? For the LGBTQ+ community, the stakes are high: Machine-learning models and AI-based tech like facial recognition can promote outdated stereotypes and public discrimination.
Our guest, Dr. Sabine Weber ( , is a computer scientist and an organizer with Queer in AI ( , a global group of LGBTQ+ researchers and scientists whose mission is to raise awareness of queer issues in artificial intelligence. Weber explains how we got here, how AI is only as good as the data it gobbles up, and the real-world consequences of misrepresentation.
Also, Vass and Katrina discuss how AI tech bros are making the switch from DEI to MEI – and what that might mean for equity in Silicon Valley.
Check out The Zizi Show, ( a deepfake drag cabaret act created by drag queens when the COVID lockdowns prevented them from performing live. Recommended by Dr. Sabine Weber!
This is Lately. Every week, we take a deep dive into the big, defining trends in business and tech that are reshaping our every day.
Our executive producer is Katrina Onstad. The show is produced by Andrea Varsany. Our sound designer is Cameron McIver.
Subscribe to the Lately newsletter ( , where we unpack more of the latest in business and technology.
Find the transcript of today’s episode here. (
We’d love to hear from you. Send your comments, questions or ideas to [email protected].
