On September 19-21, 2024, the 19th Congress of the Polish Association of Social Psychology (PSPS) took place at SWPS University in Katowice.
During the congress, employees of the Digital Social Sciences Lab – Hubert Plisiecki and Maria Flakus – gave talks presenting their research.
On the second day of the conference, Hubert Plisiecki gave a talk entitled: “How to assess scientific integrity? Estimating replicability of Polish psychology project”, co-authored by Paweł Lenartowicz.
Abstract:
The Estimating replicability of Polish psychology project has two main objectives: (1) the creation of comparable statistics on publication bias among Polish psychological departments and institutes; and (2) the conduct of preregistered (https://doi.org/10.17605/OSF.IO/JGRBF) analyses of the impact that the evaluation process could have on what has been termed the “publish or perish culture.” This latter concept is operationalized as a tendency toward publishing more, but less reliable results due to selective reporting, harking, and other strategies that result in publication bias. For these analyses, we are using the publication base for the period 2017-2021, which is the same period that was used for the evaluation of scientific institutions by the ministerial body. The database was assembled by merging the ORCID and Polish National Bibliography datasets. The texts of the articles were collected from sources available online, both open-access and proprietary, that were accessible through online databases. For the purpose of our analysis, articles were parsed for text data and filtered to return relevant statistical analysis results (p-values) using the adapted Statcheck software. The estimation of publication bias is conducted via the z-curve method (Bartoš & Schimmack, 2022). During the conference, we wish to present the results of our analyses to date and discuss methodological considerations.OSF project site: doi.org/10.17605/OSF.IO/Z4WQK
This presentation was rewarded with the Best Presentation Award, funded by the Ariadna National Research Panel. Congratulations!
On the third day of the conference, Hubert Plisiecki also gave a talk entitled: “Modeling Emotion Intensity in Political Texts: A Comparison of Supervised Methods and Annotation with Popular LLMs”, co-authored by Artur Pokropek, Maria Flakus, and Piotr Koc.
Abstract:
In this study, we have trained an emotion prediction model for Polish political texts that can be used by social psychologist to study sentiment of naturally written text. We have collected10,000 texts from the Polish section of portal X (formerly known as Twitter), concerning political topics. These texts were examined for seven emotional categories: five basic emotions (happiness, sadness, anger, disgust, fear) and two emotional dimensions (positivity and tension), aiming to compare the effectiveness of supervised machine learning models with leading language models (LLM) in the market. A key aspect of our analysis was the assessment of emotion intensity, where twenty annotators rated each of the 10,000 texts using a five-point Likert scale. This methodology allowed us to account for a broader spectrum of emotional fluctuations in the analyzed texts. During the model training process, we utilized two different Polish-language models based on the transformer architecture, conducting extensive searches for optimal parameters. This study includes a performance comparison of the selected model with the “gpt-3.5-turbo-1106″ and GPT-4 Turbo (“gpt-4-0125-preview”) models. After determining the most suitable “multiple shot” configuration on the validation set using the “gpt-3.5-turbo-1106″ model, we proceeded to compare on the test set with both LLM models. The results indicate that while supervised models still seem to be the most optimal choice for predicting emotion intensity, the difference in prediction quality is small. We open source the supervised model for use in future studies.
Also, on the third day of the conference, Maria Flakus gave a talk entitled: “What happens on social media stays on social media? Interpreting the significance of social and traditional media use, and collective narcissism for attitudes towards immigration, war anxiety and belief in disinformation – the case of war in Ukraine”, co-authored by Hubert Plisiecki and Artur Pokropek.
The same research was previously presented by Maria Flakus at the European Conference in Personality (ECP) entitled: “Embracing a Legacy of Education: The 21st European Conference on Personality”, which took place on August 6-9, 2024 at Humboldt-Universität zu Berlin in Germany (talk title: “What happens on social media stays on social media? Interpreting the significance of social and traditional media use, and collective narcissism for attitudes towards immigration, war anxiety and belief in disinformation – the case of war in Ukraine”).
Abstract:
As (social) media designs serve to capture user’s attention instead of helping to overcome information overload, it is no surprise that they promote both beneficial (e.g., solidarity, such as #MeToo movement) and unfavorable (e.g., anti-vaccine sentiment, see Basch et al., 2017) attitudes. Nevertheless, social attitudes are also rooted in more stable, semi-personality-like traits, such as collective narcissism (Golec de Zavala & Lantos, 2020), which – in turn – are associated with higher consumption of social (compared to traditional) media (Martin et al., 2019). Thus, our research aimed to explore the direct and indirect effects of those variables (media use vs. collective narcissism) on some crucial attitudes related to war in Ukraine, i.e., towards immigrants, war anxiety, and beliefs in disinformation.