This paper was written for the Special Issue "Application of Decision Support Systems in Agriculture" in MDPI Agriculture. It builds upon my earlier pilot study for TREX 2021 and evaluates 4 dimensions of a prototypical visual decision support system: usability, usefulness and needs, model understanding, and trust in an unknown prediction model. A rich mix of quantitative data (completed questionnaires) and qualitative data (semi-structured interviews) revealed many interesting relations between these metrics and participants' experience with predictive modelling.
🏆 MDPI Agriculture marked this paper as a Feature Paper, meaning it "represents most advanced research with significant potential for high impact in the field".
Takeaways
- Participants were very positive about our prototype’s usability: the visualisation, its interaction possibilities, and the general workflow were clear overall.
- Participants expressed needs regarding control, comparison, and explanations. Yet, these needs seemed subject to their personal background and job activities. Thus, visual decision support systems should be tailored towards the application context, experience with predictive modelling, and tasks.
- Participants’ understanding of the prediction model was affected on an algorithmic and an outcome level: many gradually grew a better intuition of the model’s technicalities, and could better interpret predictions and their accuracy. Yet, some participants created incorrect mental models.
- Participants' trust in the prediction model evolved differently. Moreover, trust seemed to be based on a continuous trade-off between the model’s performance, understanding the model, uncertainty in the model’s outcomes, and explanations about the development process or the prediction model itself.
- Simply designing for increasing trust is not always desirable. Designing useful decision support systems with user-centred approaches can foster appropriate trust and in turn uptake.
Abstract
The rise of ‘big data’ in agrifood has increased the need for decision support systems that harvest the power of artificial intelligence. While many such systems have been proposed, their uptake is limited, for example because they often lack uncertainty representations and are rarely designed in a user-centred way. We present a prototypical visual decision support system that incorporates price prediction, uncertainty, and visual analytics techniques. We evaluated our prototype with 10 participants who are active in different parts of agrifood. Through semi-structured interviews and questionnaires, we collected quantitative and qualitative data about four metrics: usability, usefulness and needs, model understanding, and trust. Our results reveal that the first three metrics can directly and indirectly affect appropriate trust, and that perception differences exist between people with diverging experience levels in predictive modelling. Overall, this suggests that user-centred approaches are key for increasing uptake of visual decision support systems in agrifood.
