This is my first published paper, together with Robin De Croon, Katrien Verbert and Vero Vanden Abeele, which was presented at the CHI PLAY 2020 conference. The analysis is based on the data that I collected for my master's thesis.
Paper
Download the preprint underneath or read the paper on ResearchGate or ACM Digital Library.
Abstract
Gamification researchers deem adolescents a particularly interesting audience for tailored gamification. However, empirical validation of popular player typologies and personality trait models thus far has been limited to adults. As adolescents exhibit complex behaviours that differ from older adults, these models may need adaptation. To that end, we collected a unique data set of Big Five Inventory and Hexad questionnaire answers in Dutch from 402 adolescents. Confirmatory factor analysis showed that the Dutch forms of the BFI-10, BFI-44 and Hexad scales performed substandard when used with adolescents. Through exploratory factor analysis, we investigated underlying problems, and provide preliminary suggestions on how to improve measurements. In particular, we propose to simplify the Hexad model, and reformulate specific items. With this study, we hope to contribute to the debate on how to improve the tailoring of interactive systems for adolescents.
Main takeaways
- Adolescents are underexplored in gamification research and HCI in general.
- Dutch forms of the BFI-10, BFI-44 and Hexad scales perform substandard when used with adolescents.
- Low performance of the BFI scales is probably due to issues with the instruments.
- Low performance of the Hexad scales is probably due to issues with the model, so we suggest a simplified model.
- Our unique data set is available on GitHub.
Presentation
Due to the COVID pandemic, the CHI PLAY 2020 conference took place virtually. Here are the prerecorded presentation and the handouts of the slides.
Transcript
Hello, I am Jeroen Ooge and I am very excited to be part of CHI PLAY 2020. Together with my co-authors I am very proud to present you our paper "Tailoring Gamification for Adolescents: a Validation Study of Big Five and Hexad in Dutch".
As you all know, gamification has found to be an effective motivational technique to engage users. Consensus is growing that gamification could be more effective when it is tailored to end-users.
This tailoring can be realised by taking into account things such as personality traits and gamification user types. The key challenge here is to construct valid models and accurate instruments. Personality traits are commonly modelled according to the Big Five Inventory and measured with the BFI-44 scale or its abridged version BFI-10. The Big Five Inventory contains five different personality traits.
Gamification user types are modelled according to the Hexad framework which comes with its own scale and contains six different user types.
Zooming in on the population, we observed that adolescents are underexplored in gamification research. The Big Five Inventory and Hexad have so far only been validated with adults. This is in contrast with the fact that adolescents exhibit complex behaviours which are different from adults.
This led us to the following question: are the Big Five Inventory and Hexad actually suitable for tailoring gamification for adolescents?
To tackle this question, we conducted a validation study in Dutch, following five steps. We used a validated Dutch translation of the BFI-44 scale and an own translation of Hexad which was back-translated with three language experts. Next, we visited six schools to collect responses from over 400 adolescents between 13 and 19. Third, we filtered our data and rigorously checked all statistical assumptions that are needed to perform a factor analysis, which I will now present in steps four and five.
We started with a confirmatory factor analysis in which we tested the models by fitting them to the data. We found that the fit indices were mostly substandard. In the table, the orange fit indices were borderline whereas the red fit indices were far below commonly accepted thresholds. We also found that many items had low loadings as you can for example see in the path diagram of Hexad.
Given the negative results in our confirmatory factor analysis, we performed an additional exploratory factor analysis in which we analysed the factor structure of our data. We started by looking at the existing factor structures and found that those were acceptable for BFI-44 and BFI-10. For Hexad, however, the factor structures were vaguely recognisable. In the table you can see that many items were not primarily loading on the expected factor - indicated with the blue boxes - but on a different factor, or that they were showing cross-loading.
Therefore, we also looked at alternative factor structures by iteratively pruning the items in the model. For hexad we ended up with four instead of six factors. In particular, we removed the Free Spirit type and we also merged Philanthropist and Socializer.
Going back to our question: are the Big Five Inventory and Hexad suitable for tailoring gamification for adolescents? Based on our results, the short answer is "not entirely".
This opens a new question: are these issues due to the models or to the instruments? For personality traits we saw that the factor structure was acceptable and this suggests that mainly the instruments need adaptations when used with adolescents. For the Hexad gamification user types, on the other hand, we observed more severe problems and that is why we proposed an alternative model.
In conclusion, we need reliable models and instruments to better tailor gamification for adolescents. Future studies should look into the issues that we revealed and should also verify whether our simplified model outperforms Hexad for adolescents. To help with this research, we published our unique data set on GitHub.
I would like to end with a call to the whole CHI PLAY community. Being the next generation, adolescents have loads of potential and I think we should acknowledge this more in our research, not only in the gamification domain but in the whole of HCI.
For more details and a more elaborate discussion of our results, we warmly invite you to read our paper. Do not hesitate to contact us via email or social media. We are looking forward to discussing with you how we can better tailor gamified applications for adolescents.
