Research Summary

For non-academics

In my PhD I have been working on the media conducting both applied and methodological research.

My main research project is concerned with how consumers have been reacting to major sexual violence accusations in the French music market. Using detailed market data of streaming consumption in France for 8 years, I am using causal inference methods to identify whether consumers turn away from artists following major scandals. To retrieve major scandals and control for consumers awareness of the latter, I have developed a bottom-up approach retrieving artists' presence on the internet and combining these information with automated information retrieval methods.

I have also been working on the French national press, looking at how media biases have evolved over the past 20 years. In this project I am using natural language processing methods and language models to develop a fine-grained measure of news slanting.

Finally, I am very much interested in recent advances in natural language processing and language models. I am conducting a thorough scientific watch both on the methodological evolutions and the use of these methods in social sciences. I have already written a short comment on the latter (see below) and am co-organizing the CREST NLP & Social Sciences seminar.

Publication

• The dangers of using proprietary LLMs for research (with Ana Macanovic, Etienne Ollion, Rubing Shen), Nature Machine Intelligence (2024)
  abstract

  In this comment we critically discuss the usage of ChatGPT and the likes to conduct social science research. We put forth 3 matter of concerns, namely the lack of reproducibility of the results obtained, the issue of potentially sharing sensitive data to the companies running the models and finally the bias towards English that such models reinforce.

Work in progress

• Consumers’ Response to Sexual Violence Accusations: Evidence from the French Music Market

• Using Word Embeddings to Compare the Prevalence of Gender Stereotypes in Major Music Genres from 1958 to 2022 (with Cameron Rhys Herbert, Roxana Hofmann, Donia Kamel, Maël Lecoursonnais), Submitted
  abstract

  This paper presents a content analysis of gender stereotypes in popular song lyrics using word embeddings. We begin by explaining how we curated a novel data set comprising lyrics from popular songs in the US over the past 70 years. We then explain word embeddings, detailing their nature and application to our lyric corpus. Subsequently, we apply embeddings to explore the prevalence of gender stereotypes across major music genres and to test the frequently voiced belief that certain genres, specifically hip hop, make significantly greater use of gender stereotypes than other genres. Our findings showed that while all genres exhibited stereotyping of men and women, the specific content of these stereotypes varied significantly by genre, often in surprising ways, such as that gender stereotypes in hip hop, often perceived as being distinctly sexist, were rarely stronger in hip hop than in other genres. Finally, we reflect on the strengths and limitations of using word embeddings to study music lyrics and provide suggestions for their best application to social science questions.

• Media Slant as Political Refraction: Measuring the Ideological Diversity of the French Media Landscape (with Felix Lennert, Etienne Ollion, Rubing Shen)

Other

• ChatGPT for Text Annotation? Mind the Hype! (with Ana Macanovic, Etienne Ollion, Rubing Shen), SocArXiv, (2023)
  abstract

  In the past months, researchers have enthusiastically discussed the relevance of zero- or few-shot classifiers like ChatGPT for text annotation. Should these models prove to be performant, they would open up new continents for research, and beyond. To assess the merits and limits of this approach, we conducted a systematic literature review. Reading all the articles doing zero or few-shot text annotation in the human and social sciences, we found that these few- shot learners offer enticing, yet mixed results on text annotation tasks. The performance scores can vary widely, with some being average and some being very low. Besides, zero or few-shot models are often outperformed by models fine-tuned with human annotations. Our findings thus suggest that, to date, the evidence about their effectiveness remains partial, but also that their use raises several important questions about the reproducibility of results, about privacy and copyright issues, and about the primacy of the English language. While we definitely believe that there are numerous ways to harness this powerful technology productively, we also need to harness it without falling for the hype.