Publications | Danula Hettiachchi

Publications in reversed chronological order. Please check my Google Scholar page for more details.

Filter publications by title, author, venue, or year

2026

CHI
Applying Value Sensitive Design to Location-Based Services: Designing for Shared Spaces and Local Conditions

Hiruni Kegalle, Flora Salim, Mark Sanderson, Jeffrey Chan, and Danula Hettiachchi

In Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems (CHI ’26), 2026

DOI PDF

Location-Based Services (LBS) such as ride-sharing, accommodation, food delivery, and location-driven social media platforms entangle digital systems with physical spaces, thereby generating impacts that extend beyond users to others who share the same environments. Existing design approaches struggle to address the dual challenge of value tensions that arise in shared physical spaces and the locality-specific contexts in which LBS operate. To respond, we introduce Location-Aware Value Sensitive Design (LA-VSD), a domain-specific adaptation of VSD tailored to the distinctive characteristics of LBS. LA-VSD guides designers through three heuristics to help (1) identify and prioritise stakeholders through local space-sharing scenarios, (2) adapt empirical methods to capture values and tensions in context, and (3) support value-aligned interactions across both digital and physical layers of the service. Through a case study of e-scooter sharing in Melbourne, Australia, we demonstrate how LA-VSD enables more grounded, context-aware, and actionable design of LBS.
@inproceedings{Kegalle2026-av, title = {Applying Value Sensitive Design to Location-Based Services: Designing for Shared Spaces and Local Conditions}, author = {Kegalle, Hiruni and Salim, Flora and Sanderson, Mark and Chan, Jeffrey and Hettiachchi, Danula}, booktitle = {Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems}, year = {2026}, series = {CHI '26}, publisher = {ACM}, doi = {10.1145/3772318.3791636}, }
SIGIR
Task-Aware Automated User Profile Generation for Recommendation Simulation Using Large Language Models

Xinye Wanyan, Chenglong Ma, Danula Hettiachchi, Ziqi Xu, and Jeffrey Chan

In Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’26), 2026

DOI arXiv PDF

Large Language Model (LLM)-based agent simulation has emerged as a promising approach to meet the increasing demand for real-time and rigorous evaluation in modern recommender systems. A typical LLM-driven simulation framework comprises three essential components: the profile module, memory module, and action module. However, existing studies have primarily concentrated on enhancing the memory and action modules, with limited attention to profile generation, which plays a pivotal role in ensuring realistic agent behaviours and aligning simulated interactions with real user dynamics. Moreover, the scarcity of datasets specifically designed for recommendation simulations has led to heavy reliance on manually crafted profiles, significantly limiting the scalability and generalisability of simulation frameworks across different datasets. To address these challenges, this work proposes an Automated Profile Generation Framework for Recommendation Simulation, APG4RecSim, that constructs realistic, coherent, and robust user profiles with minimal supervision. Extensive experiments on three benchmark datasets demonstrate that APG4RecSim achieves the best overall performance on discrimination, ranking, and rating tasks, improving ranking quality by up to 7% in nDCG@10 and reducing rating distribution divergence by 8% in JSD compared to existing profile-generation baselines. Beyond overall performance gains, our results show that profiles generated by APG4RecSim are resilient to popularity- and position-induced biases and maintain stable performance across datasets and different LLMs.
@inproceedings{Wanyan2026-ta, title = {Task-Aware Automated User Profile Generation for Recommendation Simulation Using Large Language Models}, author = {Wanyan, Xinye and Ma, Chenglong and Hettiachchi, Danula and Xu, Ziqi and Chan, Jeffrey}, booktitle = {Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval}, series = {SIGIR '26}, year = {2026}, publisher = {ACM}, doi = {10.1145/3805712.3809707}, }

SIGIR

Verifiable User Simulation for Search and Recommendation Systems

Chenglong Ma, Xinye Wanyan, Danula Hettiachchi, Ziqi Xu, Yongli Ren, and Jeffrey Chan

In Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’26), 2026

DOI PDF

@inproceedings{Ma2026-vu,
  title = {Verifiable User Simulation for Search and Recommendation Systems},
  author = {Ma, Chenglong and Wanyan, Xinye and Hettiachchi, Danula and Xu, Ziqi and Ren, Yongli and Chan, Jeffrey},
  booktitle = {Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval},
  series = {SIGIR '26},
  year = {2026},
  publisher = {ACM},
  doi = {10.1145/3805712.3808645},
}

CIKM
Beyond Static Profiles: Intent-Aware Behavioural Control for LLM-based Recommendation Simulation

Xinye Wanyan, Chenglong Ma, Danula Hettiachchi, Ziqi Xu, and Jeffrey Chan

In Proceedings of the 35th ACM International Conference on Information and Knowledge Management (CIKM ’26), 2026

DOI PDF

Large language models (LLMs) are increasingly used in recommender systems for user modelling, ranking, and simulation. Among these applications, LLM-based user simulators provide a flexible way to evaluate recommender systems without relying on live user feedback. However, existing simulators typically reduce user behaviour to static preference matching based on fixed user profiles, overlooking the fact that real decisions may also reflect recent preference shifts and exploratory behaviour. To address this limitation, we propose IntRecSim (Intent-aware Recommendation Simulation), a plug-in behavioural control framework for LLM-based user simulation. IntRecSim introduces a lightweight intent mechanism that guides whether the simulated decision should follow long-term preferences, recent behavioural evidence, or exploratory behaviour. Since standard recommendation datasets lack explicit intent annotations, we construct pseudo-intent supervision from historical interactions using hybrid category and semantic preference signals. The inferred intent is injected into the LLM simulator as a dynamic control signal, making the framework compatible with different profile-generation backbones. Experiments on three benchmark datasets, covering discrimination and ranking tasks show that IntRecSim improves simulation performance in most evaluated settings. Further analysis highlights the importance of explicit behavioural control for modelling realistic user decisions in LLM-based recommender simulation.
@inproceedings{Wanyan2026-bs, author = {Wanyan, Xinye and Ma, Chenglong and Hettiachchi, Danula and Xu, Ziqi and Chan, Jeffrey}, title = {Beyond Static Profiles: Intent-Aware Behavioural Control for LLM-based Recommendation Simulation}, year = {2026}, isbn = {9798400725395}, publisher = {ACM}, doi = {10.1145/3799682.3840925}, booktitle = {Proceedings of the 35th ACM International Conference on Information and Knowledge Management}, pages = {5074–5085}, numpages = {12}, keywords = {recommender system, generative ai, user behaviour simulation}, location = {Rome, Italy}, series = {CIKM '26}, }
CIKM
The ’Curse of Knowledge’ in LLM Query Simulation: Concept Provenance for Tracing Answer-Side Intrusion

Chenglong Ma, Xinye Wanyan, Danula Hettiachchi, Ziqi Xu, and Jeffrey Chan

In Proceedings of the 35th ACM International Conference on Information and Knowledge Management (CIKM ’26), 2026

DOI PDF

LLM-generated search queries are widely used to augment IR evaluation, yet they may contain concepts that presuppose answer-side document knowledge, violating the information-access boundary of pre-search users. Existing validation metrics—overlap, diversity, effectiveness—cannot distinguish rare human-tail variation from candidate answer-side intrusion. We introduce concept provenance, a framework that assigns query concepts to backstory-supported, human-central, human-tail, and candidate answer-side zones, operationalizing a boundary that retrieval metrics alone cannot detect. Applying concept provenance to 77,004 queries across 100 UQV100 topics, 8 LLMs, and 5 prompt conditions with two extraction pipelines (cross-pipeline token-HCIR Spearman = 1.0 over five condition means), we find candidate answer-side concepts constitute 7.40% of non-generic concepts, appearing in 97 of 100 topics with topic explaining around 67% of variance. Human validation yields 68.2% relaxed precision, revealing a dual mechanism: knowledge intrusion (45.5%) and deployment intrusion (45.0%). Diagnostic probes show these concepts carry disproportionate localized retrieval effects (deletion d = -0.47, exceeding random deletion d = -0.34) but explain less than 2% of aggregate evaluation variance, positioning concept provenance as a boundary-compliance diagnostic rather than an evaluation-shift predictor. Under the tested conditions, no prompt condition eliminates intrusion; post-generation concept-provenance selection achieves 99% elimination.
@inproceedings{Ma2026-tc, author = {Ma, Chenglong and Wanyan, Xinye and Hettiachchi, Danula and Xu, Ziqi and Chan, Jeffrey}, title = {The 'Curse of Knowledge' in LLM Query Simulation: Concept Provenance for Tracing Answer-Side Intrusion}, year = {2026}, isbn = {9798400725395}, publisher = {ACM}, doi = {10.1145/3799682.3840922}, booktitle = {Proceedings of the 35th ACM International Conference on Information and Knowledge Management}, pages = {3479–3490}, numpages = {12}, keywords = {concept provenance, ir evaluation, diagnostic intervention, boundary compliance, information and knowledge management}, location = {Rome, Italy}, series = {CIKM '26}, }
L@S
Rubric-guided Prompting for Automatic Assessment Scoring

Samiha Haque, Danula Hettiachchi, and Daniel Beck

In Proceedings of the Thirteenth ACM Conference on Learning @ Scale (L@S ’26), 2026

DOI PDF

Automatic grading with Large Language Models (LLMs) offers scalable assessment but raises concerns about reliability and faithfulness. We study whether LLMs truly evaluate student answers when provided with rubrics and key elements. We compare standard prompting with our rubric-guided approach that decomposes grading into key-element identification, followed by rubric-guided score aggregation and rationale generation. Experiments on the ASAP and SciEntsBank datasets show improved agreement with human grading for a large scale LLM but degraded performance for a smaller model. We also conducted interpretability case studies by experimenting with rationale generation and input token attribution methods. We found that score predictions are primarily driven by instruction and formatting tokens rather than the student response or rubric content. Generated rationales also frequently contain hallucinated rubric elements to justify incorrect scores. These results reveal that while rubric-guided prompting can result in better grading, further work is required in interpretability, which is fundamental in real world assessment scenarios and deployment of LLMs in education in general.
@inproceedings{Haque2026-rg, author = {Haque, Samiha and Hettiachchi, Danula and Beck, Daniel}, title = {Rubric-guided Prompting for Automatic Assessment Scoring}, year = {2026}, isbn = {9798400722936}, publisher = {Association for Computing Machinery}, address = {New York, NY, USA}, doi = {10.1145/3774398.3811581}, booktitle = {Proceedings of the Thirteenth ACM Conference on Learning @ Scale}, pages = {428–432}, numpages = {5}, keywords = {large language models, llms, assessment, rubrics, natural language processing, nlp, education}, location = {Seoul, Republic of Korea}, series = {L@S '26}, }
ICTIR
Reasoning with Large Language Models for Relevance Judgements

Louis Geiger, Danula Hettiachchi, Falk Scholer, and Johanne R. Trippas

In Proceedings of the 2026 International ACM SIGIR Conference on Innovative Concepts and Theories in Information Retrieval (ICTIR) (ICTIR ’26), 2026

DOI PDF

Relevance judgements are central to information retrieval (IR) evaluation, and there is much interest in using large language models (LLMs) for these judgements. This work presents a novel framework that operationalizes explicit reasoning as a control mechanism for LLM-based relevance judgements and systematically quantifies its effects across evaluation settings. We compare judgements with and without reasoning to assess the impact on accuracy and reliability across multiple datasets and models (e.g., gpt-oss, Gemini 2.5). Our results show that using reasoning often leads to lower-quality judgements and a higher rate of formatting failures, while incurring substantially higher token costs. These findings raise important questions about when and how reasoning should be incorporated into evaluation pipelines, particularly for passage-level relevance assessment, suggesting that its benefits depend on task context and complexity rather than being universally beneficial. This work also contributes a novel dataset of LLM reasoning outputs, augmenting two established test collections to support future research.
@inproceedings{Geiger2026-rw, author = {Geiger, Louis and Hettiachchi, Danula and Scholer, Falk and Trippas, Johanne R.}, title = {Reasoning with Large Language Models for Relevance Judgements}, year = {2026}, isbn = {9798400726002}, publisher = {ACM}, doi = {10.1145/3805713.3820403}, booktitle = {Proceedings of the 2026 International ACM SIGIR Conference on Innovative Concepts and Theories in Information Retrieval (ICTIR)}, pages = {105–110}, numpages = {6}, keywords = {relevance judgements, information retrieval}, location = {Melbourne, VIC, Australia}, series = {ICTIR '26}, }
NeurIPS
RMIT-ADM+S at the MMU-RAG NeurIPS 2025 Competition

Kun Ran, Marwah Alaofi, Danula Hettiachchi, Chenglong Ma, Khoi Nguyen Dinh Anh, Khoi Vo Nguyen, Sachin Pathiyan Cherumanal, Lida Rashidi, Falk Scholer, Damiano Spina, and others

In Proceedings of NeurIPS 2025 Workshop: MMU-RAG NeurIPS 2025 Competition, 2026

arXiv PDF Code

This paper presents the award-winning RMIT-ADM+S system for the Text-to-Text track of the NeurIPS 2025 MMU-RAG Competition. We introduce Routing-to-RAG (R2RAG), a research-focused retrieval-augmented generation (RAG) architecture composed of lightweight components that dynamically adapt the retrieval strategy based on inferred query complexity and evidence sufficiency. The system uses smaller LLMs, enabling operation on a single consumer-grade GPU while supporting complex research tasks. It builds on the G-RAG system, winner of the ACM SIGIR 2025 LiveRAG Challenge, and extends it with modules informed by qualitative review of outputs. R2RAG won the Best Dynamic Evaluation award in the Open Source category, demonstrating high effectiveness with careful design and efficient use of resources.
@inproceedings{Ran2026-ra, title = {RMIT-ADM+S at the MMU-RAG NeurIPS 2025 Competition}, author = {Ran, Kun and Alaofi, Marwah and Hettiachchi, Danula and Ma, Chenglong and Anh, Khoi Nguyen Dinh and Nguyen, Khoi Vo and Cherumanal, Sachin Pathiyan and Rashidi, Lida and Scholer, Falk and Spina, Damiano and others}, booktitle = {Proceedings of NeurIPS 2025 Workshop: MMU-RAG NeurIPS 2025 Competition}, year = {2026}, }

2025

IMWUT
Watch Out! E-Scooter Coming Through!: Multimodal Sensing of Mixed Traffic Use and Conflicts Through Riders’ Ego-Centric Views

Hiruni Kegalle, Danula Hettiachchi, Jeffrey Chan, Mark Sanderson, and Flora D. Salim

Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 2025

DOI PDF Code

E-scooters are becoming a popular means of urban transportation. However, this increased popularity brings challenges, such as road accidents and conflicts when sharing space with traditional transport modes. An in-depth understanding of e-scooter rider behaviour is crucial for ensuring rider safety, guiding infrastructure planning, and enforcing traffic rules. This study investigated the rider behaviour through a naturalistic study with 23 participants equipped with a bike computer, eye-tracking glasses and cameras. They followed a pre-determined route, enabling multi-modal data collection. We analysed and compared gaze movements, speed, and video feeds across three transport infrastructure types: a pedestrian-shared path, a cycle lane and a roadway. Our findings reveal unique challenges e-scooter riders face, including difficulty keeping up with cyclists and motor vehicles due to speed limits on shared e-scooters, risks in signalling turns due to control lose, and limited acceptance in mixed-use spaces. The cycle lane showed the highest average speed, the least speed change points, and the least head movements, supporting its suitability as dedicated infrastructure for e-scooters. These findings are facilitated through multimodal sensing and analysing the e-scooter riders’ ego-centric view, which show the efficacy of our method in discovering the behavioural dynamics of the riders in the wild. Our study highlights the critical need to align infrastructure with user behaviour to improve safety and emphasises the importance of targeted safety measures and regulations, especially when e-scooter riders share spaces with pedestrians or motor vehicles. The dataset and analysis code are available at https://github.com/HiruniNuwanthika/Electric-Scooter-Riders-Multi-Modal-Data-Analysis.
@article{Kegalle2025-wo, title = {Watch Out! E-Scooter Coming Through!: Multimodal Sensing of Mixed Traffic Use and Conflicts Through Riders’ Ego-Centric Views}, author = {Kegalle, Hiruni and Hettiachchi, Danula and Chan, Jeffrey and Sanderson, Mark and Salim, Flora D.}, journal = {Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies}, publisher = {ACM}, volume = {9}, number = {1}, doi = {10.1145/3712284}, year = {2025}, address = {New York, NY, USA}, }
IJHCS
Comprehension Is a Double-Edged Sword: Over-Interpreting Unspecified Information in Intelligible Machine Learning Explanations

Yueqing Xuan, Edward Small, Kacper Sokol, Danula Hettiachchi, and Mark Sanderson

International Journal of Human Computer Studies, 2025

DOI PDF Code

Automated decision-making systems are becoming increasingly ubiquitous, which creates an immediate need for their interpretability and explainability. However, it remains unclear whether users know what insights an explanation offers and, more importantly, what information it lacks. To answer this question we conducted an online study with 200 participants, which allowed us to assess explainees’ ability to realise explicated information – i.e., factual insights conveyed by an explanation – and unspecified information – i.e, insights that are not communicated by an explanation – across four representative explanation types: model architecture, decision surface visualisation, counterfactual explainability and feature importance. Our findings uncover that highly comprehensible explanations, e.g., feature importance and decision surface visualisation, are exceptionally susceptible to misinterpretation since users tend to infer spurious information that is outside of the scope of these explanations. Additionally, while the users gauge their confidence accurately with respect to the information explicated by these explanations, they tend to be overconfident when misinterpreting the explanations. Our work demonstrates that human comprehension can be a double-edged sword since highly accessible explanations may convince users of their truthfulness while possibly leading to various misinterpretations at the same time. Machine learning explanations should therefore carefully navigate the complex relation between their full scope and limitations to maximise understanding and curb misinterpretation.
@article{Xuan2024-ci, author = {Xuan, Yueqing and Small, Edward and Sokol, Kacper and Hettiachchi, Danula and Sanderson, Mark}, journal = {International Journal of Human Computer Studies}, series = {IJHCS}, title = {Comprehension Is a Double-Edged Sword: Over-Interpreting Unspecified Information in Intelligible Machine Learning Explanations}, volume = {193}, pages = {103376}, year = {2025}, issn = {1071-5819}, doi = {10.1016/j.ijhcs.2024.103376}, publisher = {Elevier}, }

IASDR

Design Patterns for AI Curated Content

Awais Hameed Khan, Senuri Wijenayake, Sara Fahad Dawood Al Lawati, Damiano Spina, and Danula Hettiachchi

In IASDR 2025: About Design Next (IASDR ’25), 2025

DOI PDF

@inproceedings{Khan2025-dp,
  title = {Design Patterns for AI Curated Content},
  author = {Khan, Awais Hameed and Wijenayake, Senuri and Al Lawati, Sara Fahad Dawood and Spina, Damiano and Hettiachchi, Danula},
  booktitle = {IASDR 2025: About Design Next},
  year = {2025},
  series = {IASDR '25},
  doi = {10.21606/iasdr.2025.624},
}

CIKM
Temporal-Aware User Behaviour Simulation with Large Language Models for Recommender Systems

Xinye Wanyan, Danula Hettiachchi, Chenglong Ma, Ziqi Xu, and Jeffrey Chan

In Proceedings of the 34th ACM International Conference on Information and Knowledge Management (CIKM ’25), 2025

DOI PDF

Large Language Models (LLMs) demonstrate human-like capabilities in language understanding, reasoning, and generation, driving interest in using LLM-based agents to simulate human feedback in recommender systems. However, most existing approaches rely on static user profiling, neglecting the temporal and dynamic nature of user interests. This limitation stems from a disconnect between language modelling and behaviour modelling, which constrains the capacity of agents to represent sequential patterns. To address this challenge, we propose a Dynamic Temporal-aware Agent-based simulator for Recommender Systems, DyTA4Rec, which enables agents to model and utilise evolving user behaviour based on historical interactions. DyTA4Rec features a dynamic updater for real-time profile refinement, temporal-enhanced prompting for sequential context, and self-adaptive aggregation for coherent feedback. Experimental results at group and individual levels show that DyTA4Rec significantly improves the alignment between simulated and actual user behaviour by modelling dynamic characteristics and enhancing temporal awareness in LLM-based agents.
@inproceedings{Wanyan2025-ta, title = {Temporal-Aware User Behaviour Simulation with Large Language Models for Recommender Systems}, author = {Wanyan, Xinye and Hettiachchi, Danula and Ma, Chenglong and Xu, Ziqi and Chan, Jeffrey}, booktitle = {Proceedings of the 34th ACM International Conference on Information and Knowledge Management}, series = {CIKM '25}, year = {2025}, publisher = {ACM}, doi = {10.1145/3746252.3760878}, }

SIGIR Forum

Report from the 4th Strategic Workshop on Information Retrieval in Lorne (SWIRL 2025)

Johanne R. Trippas, J. Shane Culpepper, Mohammad Aliannejadi, James Allan, Enrique Amigó, Jaime Arguello, Leif Azzopardi, Peter Bailey, Jamie Callan, Rob Capra, Nick Craswell, Bruce Croft, Jeff Dalton, Gianluca Demartini, Laura Dietz, Zhicheng Dou, Carsten Eickhoff, Michael Ekstrand, Nicola Ferro, Norbert Fuhr, Dorota Glowacka, Faegheh Hasibi, Danula Hettiachchi, Rosie Jones, Jaap Kamps, Noriko Kando, Sarvnaz Karimi, Makoto P. Kato, Bevan Koopman, Yiqun Liu, Chenglong Ma, Joel Mackenzie, Maria Maistro, Jiaxin Mao, Dana McKay, Bhaskar Mitra, Stefano Mizzaro, Alistair Moffat, Josiane Mothe, Iadh Ounis, Lida Rashidi, Yongli Ren, Mark Sanderson, Rodrygo Santos, Falk Scholer, Chirag Shah, Laurianne Sitbon, Ian Soboroff, Damiano Spina, Paul Thomas, Julián Urbano, Arjen Vries, Ryen White, Abby Yuan, Hamed Zamani, Oleg Zendel, Min Zhang, Shengyao Zhuang, Justin Zobel, and Guido Zuccon

SIGIR Forum, 2025

DOI PDF Website

The purpose of the Strategic Workshop on Information Retrieval in Lorne (SWIRL)1 is to explore the long-range issues of the information retrieval (IR) field, to recognise challenges that are on – or even over – the horizon, to build consensus on key challenges, and to disseminate the resulting information to the research community. The intent is that this description of open problems will help to inspire researchers and graduate students to address the questions and will provide funding agencies with data to focus and coordinate support for IR research.

@article{Trippas2025-rf,
  author = {Trippas, Johanne R. and Culpepper, J. Shane and Aliannejadi, Mohammad and Allan, James and Amig\'{o}, Enrique and Arguello, Jaime and Azzopardi, Leif and Bailey, Peter and Callan, Jamie and Capra, Rob and Craswell, Nick and Croft, Bruce and Dalton, Jeff and Demartini, Gianluca and Dietz, Laura and Dou, Zhicheng and Eickhoff, Carsten and Ekstrand, Michael and Ferro, Nicola and Fuhr, Norbert and Glowacka, Dorota and Hasibi, Faegheh and Hettiachchi, Danula and Jones, Rosie and Kamps, Jaap and Kando, Noriko and Karimi, Sarvnaz and Kato, Makoto P. and Koopman, Bevan and Liu, Yiqun and Ma, Chenglong and Mackenzie, Joel and Maistro, Maria and Mao, Jiaxin and McKay, Dana and Mitra, Bhaskar and Mizzaro, Stefano and Moffat, Alistair and Mothe, Josiane and Ounis, Iadh and Rashidi, Lida and Ren, Yongli and Sanderson, Mark and Santos, Rodrygo and Scholer, Falk and Shah, Chirag and Sitbon, Laurianne and Soboroff, Ian and Spina, Damiano and Thomas, Paul and Urbano, Juli\'{a}n and de Vries, Arjen and White, Ryen and Yuan, Abby and Zamani, Hamed and Zendel, Oleg and Zhang, Min and Zhuang, Shengyao and Zobel, Justin and Zuccon, Guido},
  title = {Report from the 4th Strategic Workshop on Information Retrieval in Lorne (SWIRL 2025)},
  year = {2025},
  issue_date = {June 2025},
  publisher = {Association for Computing Machinery},
  address = {New York, NY, USA},
  volume = {59},
  number = {1},
  issn = {0163-5840},
  url = {https://doi.org/10.1145/3769733.3769739},
  doi = {10.1145/3769733.3769739},
  journal = {SIGIR Forum},
  month = oct,
  pages = {1–68},
  numpages = {68},
}

IMWUT
SenseSeek Dataset: Multimodal Sensing to Study Information Seeking Behaviors

Kaixin Ji, Danula Hettiachchi, Falk Scholer, Flora D. Salim, and Damiano Spina

Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 2025

DOI PDF Code

Information processing tasks involve complex cognitive mechanisms that are shaped by various factors, including individual goals, prior experience, and system environments. Understanding such behaviors requires a sophisticated and personalized data capture of how one interacts with modern information systems (e.g., web search engines). Passive sensors, such as wearables, capturing physiological and behavioral data, have the potential to provide solutions in this context. This paper presents a novel dataset, SenseSeek, designed to evaluate the effectiveness of consumer-grade sensors in a complex information processing scenario: searching via systems (e.g., search engines), one of the common strategies users employ for information seeking. The SenseSeek dataset comprises data collected from 20 participants, 235 trials of the stimulated search process, 940 phases of stages in the search process, including the realization of Information Need (IN), Query Formulation (QF), Query Submission by Typing (QS-T) or Speaking (QS-S), and Relevance Judgment by Reading (RJ-R) or Listening (RJ-L). The data includes Electrodermal Activities (EDA), Electroencephalogram (EEG), PUPIL, GAZE, and MOTION data, which were captured using consumer-grade sensors. It also contains 258 features extracted from the sensor data, the gaze-annotated screen recordings, and task responses. We validate the usefulness of the dataset by providing baseline analyses on the impacts of different cognitive intents and interaction modalities on the sensor data, and effectiveness of the data in discriminating the search stages. To our knowledge, SenseSeek is the first dataset that characterizes multiple stages involved in information seeking with physiological signals collected from multiple sensors. We hope this dataset can serve as a reference for future research on information-seeking behaviors.
@article{Ji2025-sd, title = {SenseSeek Dataset: Multimodal Sensing to Study Information Seeking Behaviors}, author = {Ji, Kaixin and Hettiachchi, Danula and Scholer, Falk and Salim, Flora D. and Spina, Damiano}, journal = {Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies}, publisher = {ACM}, year = {2025}, address = {New York, NY, USA}, doi = {10.1145/3749501}, }
MobileHCI
"Nuisance is Better Than Nothing?": Exploring How Pedestrians and Cyclists Perceive Automated E-Scooter Alerts in Shared Spaces

Hiruni Kegalle, Danula Hettiachchi, Jeffrey Chan, Flora D. Salim, and Mark Sanderson

Proceedings of the ACM on Human-Computer Interaction, 2025

PDF Code

Electric scooters (e-scooters) offer flexible urban mobility but raise safety concerns in shared spaces. This study investigates how e-scooters can better communicate their presence to pedestrians and cyclists in shared active mobility environments. A focus group with e-scooter riders identified notifying others of arrival as a key challenge. To address this, we co-designed audio and visual alerts in a participatory workshop and evaluated them in a real-world Wizard of Oz (WoZ) study involving live encounters. WoZ self-report data showed that voice and bell alerts were rated significantly higher in visibility, safety, communication, and acceptance than continuous sounds and flashing lights. These findings were supported by video analysis, which captured clear spatial responses such as turning or moving aside. Cyclists rated alerts as more distracting than pedestrians. Eye-tracking data revealed increased pedestrian attention during overtaking. By combining self-reports, video, and gaze data, we provide in-situ evidence and design recommendations to improve e-scooter signalling and reduce conflict. The dataset, including anonymized ratings, fixation data, alert designs, and analysis scripts, is available at https://github.com/HiruniNuwanthika/User-Perception-Evaluation-Escooter-Alerts.git.
@article{Kegalle2025-ni, title = {"Nuisance is Better Than Nothing?": Exploring How Pedestrians and Cyclists Perceive Automated E-Scooter Alerts in Shared Spaces}, author = {Kegalle, Hiruni and Hettiachchi, Danula and Chan, Jeffrey and Salim, Flora D. and Sanderson, Mark}, journal = {Proceedings of the ACM on Human-Computer Interaction}, number = {MHCI}, year = {2025}, publisher = {ACM}, }
SIGIR
An LLM-Enhanced Personality-Driven User Behaviour Simulator for Recommender System Evaluation

Chenglong Ma, Ziqi Xu, Yongli Ren, Danula Hettiachchi, and Jeffrey Chan

In Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’25), 2025

DOI PDF

Traditional offline evaluation methods for recommender systems struggle to capture the complexity of modern platforms due to sparse behavioural signals, noisy data, and limited modelling of user personality traits. While simulation frameworks can generate synthetic data to address these gaps, existing methods fail to replicate behavioural diversity, limiting their effectiveness. To overcome these challenges, we propose the Personality-driven User Behaviour Simulator (PUB), an LLM-based simulation framework that integrates the Big Five personality traits to model personalised user behaviour. PUB dynamically infers user personality from behavioural logs (e.g., ratings, reviews) and item metadata, then generates synthetic interactions that preserve statistical fidelity to real-world data. Experiments on the Amazon review datasets show that logs generated by PUB closely align with real user behaviour and reveal meaningful associations between personality traits and recommendation outcomes. These results highlight the potential of the personality-driven simulator to advance recommender system evaluation, offering scalable, controllable, high-fidelity alternatives to resource-intensive real-world experiments.
@inproceedings{Ma2025-al, title = {An LLM-Enhanced Personality-Driven User Behaviour Simulator for Recommender System Evaluation}, author = {Ma, Chenglong and Xu, Ziqi and Ren, Yongli and Hettiachchi, Danula and Chan, Jeffrey}, booktitle = {Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval}, series = {SIGIR '25}, year = {2025}, publisher = {ACM}, doi = {10.1145/3726302.3730238}, }
WWW
Multi-Stage LLM Pipelines Can Outperform GPT-4o in Relevance Assessment

Julian A. Schnabel, Johanne R. Trippas, Falk Scholer, and Danula Hettiachchi

In Companion Proceedings of the ACM Web Conference 2025 (WWW ’25), 2025

DOI arXiv PDF

The effectiveness of search systems is evaluated using relevance labels that indicate the usefulness of documents for specific queries and users. While obtaining these relevance labels from real users is ideal, scaling such data collection is challenging. Consequently, third-party annotators are employed, but their inconsistent accuracy demands costly auditing, training, and monitoring. We propose an LLM-based modular classification pipeline that divides the relevance assessment task into multiple stages, each utilising different prompts and models of varying sizes and capabilities. Applied to TREC Deep Learning (TREC-DL), one of our approaches showed an 18.4% Krippendorff’s α accuracy increase over OpenAI’s GPT-4o mini while maintaining a cost of about 0.2 USD per million input tokens, offering a more efficient and scalable solution for relevance assessment. This approach beats the baseline performance of GPT-4o (5 USD). With a pipeline approach, even the accuracy of the GPT-4o flagship model, measured in α, could be improved by 9.7%.
@inproceedings{Schnabel2025-ml, title = {Multi-Stage LLM Pipelines Can Outperform GPT-4o in Relevance Assessment}, author = {Schnabel, Julian A. and Trippas, Johanne R. and Scholer, Falk and Hettiachchi, Danula}, booktitle = {Companion Proceedings of the ACM Web Conference 2025}, doi = {10.1145/3701716.3715488}, year = {2025}, publisher = {ACM}, pages = {1288–1292}, numpages = {5}, series = {WWW '25}, }
SIGIR-AP
ISMIE: A Framework to Characterize Information Seeking in Modern Information Environments

Shuoqi Sun, Danula Hettiachchi, and Damiano Spina

In Proceedings of the 2025 Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region (SIGIR-AP ’25), 2025

DOI PDF

The modern information environment (MIE) is increasingly complex, shaped by a wide range of techniques designed to satisfy users’ information needs. Information seeking (IS) models are effective mechanisms for characterizing user-system interactions. However, conceptualizing a model that fully captures the MIE landscape poses a challenge. We argue: Does such a model exist? To address this, we propose the Information Seeking in Modern Information Environments (ISMIE) framework as a fundamental step. ISMIE conceptualizes the information seeking process (ISP) via three key concepts: Components (e.g., Information Seeker), Intervening Variables (e.g., Interactive Variables), and Activities (e.g., Acquiring). Using ISMIE’s concepts and employing a case study based on a common scenario - misinformation dissemination – we analyze six existing IS and information retrieval (IR) models to illustrate their limitations and the necessity of ISMIE. We then show how ISMIE serves as an actionable framework for both characterization and experimental design. We characterize three pressing issues and then outline two research blueprints: a user-centric, industry-driven experimental design for the authenticity and trust crisis to AI-generated content and a system-oriented, academic-driven design for tackling dopamine-driven content consumption. Our framework offers a foundation for developing IS and IR models to advance knowledge on understanding human interactions and system design in MIEs.
@inproceedings{Sun2025-ia, title = {ISMIE: A Framework to Characterize Information Seeking in Modern Information Environments}, author = {Sun, Shuoqi and Hettiachchi, Danula and Spina, Damiano}, booktitle = {Proceedings of the 2025 Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region}, year = {2025}, series = {SIGIR-AP '25}, doi = {10.1145/3767695.3769509}, publisher = {ACM}, pages = {385–395}, }

2024

SIGIR
Characterizing Information Seeking Processes with Multiple Physiological Signals

Kaixin Ji, Danula Hettiachchi, Flora D Salim, Falk Scholer, and Damiano Spina

In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’24), 2024

DOI PDF Code

Information access systems are getting complex, and our understanding of user behavior during information seeking processes is mainly drawn from qualitative methods, such as observational studies or surveys. Leveraging the advances in sensing technologies, our study aims to characterize user behaviors with physiological signals, particularly in relation to cognitive load, affective arousal, and valence. We conduct a controlled lab study with 26 participants, and collect data including Electrodermal Activities, Photoplethysmogram, Electroencephalogram, and Pupillary Responses. This study examines informational search with four stages: the realization of Information Need (IN), Query Formulation (QF), Query Submission (QS), and Relevance Judgment (RJ). We also include different interaction modalities to represent modern systems, e.g., QS by text-typing or verbalizing, and RJ with text or audio information. We analyze the physiological signals across these stages and report outcomes of pairwise non-parametric repeated-measure statistical tests. The results show that participants experience significantly higher cognitive loads at IN with a subtle increase in alertness, while QF requires higher attention. QS involves demanding cognitive loads than QF. Affective responses are more pronounced at RJ than QS or IN, suggesting greater interest and engagement as knowledge gaps are resolved. To the best of our knowledge, this is the first study that explores user behaviors in a search process employing a more nuanced quantitative analysis of physiological signals. Our findings offer valuable insights into user behavior and emotional responses in information seeking processes. We believe our proposed methodology can inform the characterization of more complex processes, such as conversational information seeking.
@inproceedings{Ji2024-ci, author = {Ji, Kaixin and Hettiachchi, Danula and Salim, Flora D and Scholer, Falk and Spina, Damiano}, booktitle = {Proceedings of the 47th International {ACM} {SIGIR} Conference on Research and Development in Information Retrieval}, series = {SIGIR '24}, title = {Characterizing Information Seeking Processes with Multiple Physiological Signals}, year = {2024}, publisher = {ACM}, doi = {10.1145/3626772.3657793}, }
AH
E-Scooter Dynamics: Unveiling Rider Behaviours and Interactions with Road Users through Multi-Modal Data Analysis

Hiruni Kegalle, Danula Hettiachchi, Jeffrey Chan, Flora Salim, and Mark Sanderson

In Proceedings of the Augmented Humans International Conference 2024 (AHs ’24), 2024

DOI PDF

Electric scooters (e-scooters), characterised by their small size and lightweight design, have revolutionised urban commuting experiences. Their adaptability to multiple mobility infrastructures introduces advantages for users, enhancing the efficiency and flexibility of urban transit. However, this versatility also causes potential challenges, including increased interactions and conflicts with other road users. Previous research has primarily focused on historical trip data, leaving a gap in our understanding of real-time e-scooter user behaviours and interactions. To bridge this gap, we propose a novel multi-modal data collection and integrated data analysis methodology, aimed at capturing the dynamic behaviours of e-scooter riders and their interactions with other road users in real-world settings. We present the study setup and the analysis approach we used for an in the wild study with 15 participants, each traversing a pre-determined route equipped with off-the-shelf commercially available devices (e.g., cameras, bike computers) and eye-tracking glasses.
@inproceedings{Kegalle2024-ah, author = {Kegalle, Hiruni and Hettiachchi, Danula and Chan, Jeffrey and Salim, Flora and Sanderson, Mark}, title = {E-Scooter Dynamics: Unveiling Rider Behaviours and Interactions with Road Users through Multi-Modal Data Analysis}, year = {2024}, publisher = {ACM}, doi = {10.1145/3652920.3653056}, booktitle = {Proceedings of the Augmented Humans International Conference 2024}, pages = {307–310}, series = {AHs '24}, }
MobileHCI
Towards Detecting and Mitigating Cognitive Bias in Spoken Conversational Search

Kaixin Ji, Sachin Pathiyan Cherumanal, Johanne R. Trippas, Danula Hettiachchi, Flora D. Salim, Falk Scholer, and Damiano Spina

In Adjunct Proceedings of the 26th International Conference on Mobile Human-Computer Interaction (MobileHCI ’24 Adjunct), 2024

DOI PDF

Spoken Conversational Search (SCS) poses unique challenges in understanding user-system interactions due to the absence of visual cues, and the complexity of less structured dialogue. Tackling the impacts of cognitive bias in today’s information-rich online environment, especially when SCS becomes more prevalent, this paper integrates insights from information science, psychology, cognitive science, and wearable sensor technology to explore potential opportunities and challenges in studying cognitive biases in SCS. It then outlines a framework for experimental designs with various experiment setups to multimodal instruments. It also analyzes data from an existing dataset as a preliminary example to demonstrate the potential of this framework and discuss its implications for future research. In the end, it discusses the challenges and ethical considerations associated with implementing this approach. This work aims to provoke new directions and discussion in the community and enhance understanding of cognitive biases in Spoken Conversational Search.
@inproceedings{Ji2014-td, author = {Ji, Kaixin and Pathiyan Cherumanal, Sachin and Trippas, Johanne R. and Hettiachchi, Danula and Salim, Flora D. and Scholer, Falk and Spina, Damiano}, title = {Towards Detecting and Mitigating Cognitive Bias in Spoken Conversational Search}, year = {2024}, isbn = {9798400705069}, publisher = {ACM}, doi = {10.1145/3640471.3680245}, booktitle = {Adjunct Proceedings of the 26th International Conference on Mobile Human-Computer Interaction}, articleno = {10}, numpages = {10}, series = {MobileHCI '24 Adjunct}, }
CHIIR
Walert: Putting Conversational Information Seeking Knowledge into Action by Building and Evaluating a Large Language Model-Powered Chatbot

Sachin Pathiyan Cherumanal, Lin Tian, Futoon M. Abushaqra, Angel Felipe Paula, Kaixin Ji, Danula Hettiachchi, Johanne R. Trippas, Halil Ali, Falk Scholer, and Damiano Spina

In Proceedings of ACM SIGIR Conference on Human Information Interaction And Retrieval (CHIIR ’24), 2024

DOI PDF Code Website

Creating and deploying customized applications is crucial for operational success and enriching user experiences in the rapidly evolving modern business world. A prominent facet of modern user experiences is the integration of chatbots or voice assistants. The rapid evolution of Large Language Models (LLMs) has provided a powerful tool to build conversational applications. We present Walert, a customized LLM-based conversational agent able to answer frequently asked questions about computer science degrees and programs at RMIT University. Our demo aims to showcase how conversational information-seeking researchers can effectively communicate the benefits of using best practices to stakeholders interested in developing and deploying LLM-based chatbots. These practices are well-known in our community but often overlooked by practitioners who may not have access to this knowledge. The methodology and resources used in this demo serve as a bridge to facilitate knowledge transfer from experts, address industry professionals’ practical needs, and foster a collaborative environment. The data and code of the demo are available at https://github.com/rmit-ir/walert.
@inproceedings{Pathiyan2024-wp, author = {Pathiyan Cherumanal, Sachin and Tian, Lin and Abushaqra, Futoon M. and Magnossão de Paula, Angel Felipe and Ji, Kaixin and Hettiachchi, Danula and Trippas, Johanne R. and Ali, Halil and Scholer, Falk and Spina, Damiano}, booktitle = {Proceedings of ACM SIGIR Conference on Human Information Interaction And Retrieval}, series = {CHIIR '24}, title = {Walert: Putting Conversational Information Seeking Knowledge into Action by Building and Evaluating a Large Language Model-Powered Chatbot}, year = {2024}, publisher = {ACM}, doi = {https://doi.org/10.1145/3627508.3638309}, }
OzCHI
Unmaking AI: Engaging Critically and Creatively with Generative AI

Luke Munn, Awais Hameed Khan, Danula Hettiachchi, Samar Sabie, Lida Ghahremanlou, Saarim Saghir, Nicholas Lambourne, and Liam Magee

In Proceedings of the 36th Australasian Conference on Human-Computer Interaction (OzCHI ’24), 2024

DOI PDF Website

Generative AI (GenAI) offers powerful possibilities but also introduces significant socio-cultural, political, and environmental issues. It is therefore imperative that researchers and practitioners, responsible for shaping these technologies, develop the critical capabilities necessary to engage with GenAI systems in creative, responsible, and ethical ways. Recent discourse on ‘unmaking’ indicates that this can be a useful paradigm for us to better understand GenAI. In this workshop we provide a framework for “Unmaking AI” — introducing participants to different GenAI models; real-world examples of researchers using these models in practice; and conduct hands-on unmaking AI activities, using bespoke design cards, created for experimentation and reflection. The workshop requires no prior participant knowledge or engagement with GenAI systems. Through this workshop we aim to bring together a community of researchers and practitioners interested in shaping discourse about critically and creatively unmaking AI and collaboratively develop tools, resources, and readings to support them.
@inproceedings{Munn2024-ua, author = {Munn, Luke and Khan, Awais Hameed and Hettiachchi, Danula and Sabie, Samar and Ghahremanlou, Lida and Saghir, Saarim and Lambourne, Nicholas and Magee, Liam}, title = {Unmaking AI: Engaging Critically and Creatively with Generative AI}, year = {2024}, isbn = {9798400715099}, publisher = {ACM}, address = {New York, NY, USA}, url = {https://doi.org/10.1145/3726986.3727943}, doi = {10.1145/3726986.3727943}, booktitle = {Proceedings of the 36th Australasian Conference on Human-Computer Interaction}, pages = {892–897}, numpages = {6}, keywords = {Design Cards, Generative AI, Unmaking}, series = {OzCHI '24}, }

2023

CIKM
Designing and Evaluating Presentation Strategies for Fact-Checked Content

Danula Hettiachchi, Kaixin Ji, Jenny Kennedy, Anthony McCosker, Flora D Salim, Mark Sanderson, Falk Scholer, and Damiano Spina

In Proceedings of the 32nd ACM International Conference on Information & Knowledge Management (CIKM ’23), 2023

DOI PDF Code

With the rapid growth of online misinformation, it is crucial to have reliable fact-checking methods. Recent research on finding check-worthy claims and automated fact-checking have made significant advancements. However, limited guidance exists regarding the presentation of fact-checked content to effectively convey verified information to users. We address this research gap by exploring the critical design elements in fact-checking reports and investigating whether credibility and presentation-based design improvements can enhance users’ ability to interpret the report accurately. We co-developed potential content presentation strategies through a workshop involving fact-checking professionals, communication experts, and researchers. The workshop examined the significance and utility of elements such as veracity indicators and explored the feasibility of incorporating interactive components for enhanced information disclosure. Building on the workshop outcomes, we conducted an online experiment involving 76 crowd workers to assess the efficacy of different design strategies. The results indicate that proposed strategies significantly improve users’ ability to accurately interpret the verdict of fact-checking articles. Our findings underscore the critical role of effective presentation of fact reports in addressing the spread of misinformation. By adopting appropriate design enhancements, the effectiveness of fact-checking reports can be maximized, enabling users to make informed judgments.
@inproceedings{Hettiachchi2023-po, title = {Designing and Evaluating Presentation Strategies for {Fact-Checked} Content}, booktitle = {Proceedings of the 32nd {ACM} International Conference on Information \& Knowledge Management}, author = {Hettiachchi, Danula and Ji, Kaixin and Kennedy, Jenny and McCosker, Anthony and Salim, Flora D and Sanderson, Mark and Scholer, Falk and Spina, Damiano}, publisher = {ACM}, series = {CIKM '23}, year = {2023}, doi = {10.1145/3583780.3614841}, }
HCMOP
How Crowd Worker Factors Influence Subjective Annotations: A Study of Tagging Misogynistic Hate Speech in Tweets

Danula Hettiachchi, Indigo Holcombe-James, Stephanie Livingstone, Anjalee Silva, Matthew Lease, Flora D Salim, and Mark Sanderson

In Proceedings of the Eleventh AAAI Conference on Human Computation and Crowdsourcing (HCOMP ’23), 2023

DOI PDF Code

Crowdsourced annotation is vital to both collecting labelled data to train and test automated content moderation systems and to support human-in-the-loop review of system decisions. However, annotation tasks such as judging hate speech are subjective and thus highly sensitive to biases stemming from annotator beliefs, characteristics and demographics. We conduct two crowdsourcing studies on Mechanical Turk to examine annotator bias in labelling sexist and misogynistic hate speech. Results from 109 annotators show that annotator political inclination, moral integrity, personality traits, and sexist attitudes significantly impact annotation accuracy and the tendency to tag content as hate speech. In addition, semi-structured interviews with nine crowd workers provide further insights regarding the influence of subjectivity on annotations. In exploring how workers interpret a task - shaped by complex negotiations between platform structures, task instructions, subjective motivations, and external contextual factors - we see annotations not only impacted by worker factors but also simultaneously shaped by the structures under which they labour.
@inproceedings{Hettiachchi2023-hj, title = {How Crowd Worker Factors Influence Subjective Annotations: A Study of Tagging Misogynistic Hate Speech in Tweets}, booktitle = {Proceedings of the Eleventh {AAAI} Conference on Human Computation and Crowdsourcing}, author = {Hettiachchi, Danula and Holcombe-James, Indigo and Livingstone, Stephanie and de Silva, Anjalee and Lease, Matthew and Salim, Flora D and Sanderson, Mark}, series = {HCOMP '23}, volume = {11}, number = {1}, year = {2023}, publisher = {AAAI}, doi = {10.1609/hcomp.v11i1.27546}, }
MDM
Are footpaths encroached by shared e-scooters? Spatio-temporal Analysis of Micro-mobility Services

Hiruni Kegalle, Danula Hettiachchi, Jeffrey Chan, Flora Salim, and Mark Sanderson

In 2023 24th IEEE International Conference on Mobile Data Management (MDM ’23), 2023

DOI arXiv PDF

Micro-mobility services (e.g., e-bikes, e-scooters) are increasingly popular among urban communities, being a flexible transport option that brings both opportunities and challenges. As a growing mode of transportation, insights gained from micro-mobility usage data are valuable in policy formulation and improving the quality of services. Existing research analyses patterns and features associated with usage distributions in different localities, and focuses on either temporal or spatial aspects. In this paper, we employ a combination of methods that analyse both spatial and temporal characteristics related to e-scooter trips in a more granular level, enabling observations at different time frames and local geographical zones that prior analysis wasn’t able to do. The insights obtained from anonymised, restricted data on shared e-scooter rides show the applicability of the employed method on regulated, privacy preserving micro-mobility trip data. Our results showed population density is the topmost important feature, and it associates with e-scooter usage positively. Population owning motor vehicles is negatively associated with shared e-scooter trips, suggesting a reduction in e-scooter usage among motor vehicle owners. Furthermore, we found that the effect of humidity is more important than precipitation in predicting hourly e-scooter trip count. Buffer analysis showed, nearly 29% trips were stopped, and 27% trips were started on the footpath, revealing higher utilisation of footpaths for parking e-scooters in Melbourne.
@inproceedings{Kegalle2023-ux, title = {Are footpaths encroached by shared e-scooters? Spatio-temporal Analysis of Micro-mobility Services}, booktitle = {2023 24th {IEEE} International Conference on Mobile Data Management}, author = {Kegalle, Hiruni and Hettiachchi, Danula and Chan, Jeffrey and Salim, Flora and Sanderson, Mark}, series = {MDM '23}, pages = {255--264}, publisher = {IEEE}, year = {2023}, issn = {2375-0324}, doi = {10.1109/MDM58254.2023.00049}, }
SIGIR
Examining the Impact of Uncontrolled Variables on Physiological Signals in User Studies for Information Processing Activities

Kaixin Ji, Damiano Spina, Danula Hettiachchi, Flora D Salim, and Falk Scholer

In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’23), 2023

DOI PDF

Physiological signals can potentially be applied as objective measures to understand the behavior and engagement of users interacting with information access systems. However, the signals are highly sensitive, and many controls are required in laboratory user studies. To investigate the extent to which controlled or uncontrolled (i.e., confounding) variables such as task sequence or duration influence the observed signals, we conducted a pilot study where each participant completed four types of information-processing activities (READ, LISTEN, SPEAK, and WRITE). Meanwhile, we collected data on blood volume pulse, electrodermal activity, and pupil responses. We then used machine learning approaches as a mechanism to examine the influence of controlled and uncontrolled variables that commonly arise in user studies. Task duration was found to have a substantial effect on the model performance, suggesting it represents individual differences rather than giving insight into the target variables. This work contributes to our understanding of such variables in using physiological signals in information retrieval user studies.
@inproceedings{Ji2023-aq, title = {Examining the Impact of Uncontrolled Variables on Physiological Signals in User Studies for Information Processing Activities}, booktitle = {Proceedings of the 46th International {ACM} {SIGIR} Conference on Research and Development in Information Retrieval}, author = {Ji, Kaixin and Spina, Damiano and Hettiachchi, Danula and Salim, Flora D and Scholer, Falk}, publisher = {ACM}, series = {SIGIR '23}, year = {2023}, doi = {10.1145/3539618.3591981}, }
WWW
Combining Worker Factors for Heterogeneous Crowd Task Assignment

Senuri Wijenayake, Danula Hettiachchi, and Jorge Goncalves

In Proceedings of the ACM Web Conference 2023 (WWW ’23), 2023

DOI PDF

Optimising the assignment of tasks to workers is an effective approach to ensure high quality in crowdsourced data - particularly in heterogeneous micro tasks. However, previous attempts at heterogeneous micro task assignment based on worker characteristics are limited to using cognitive skills, despite literature emphasising that worker performance varies based on other parameters. This study is an initial step towards understanding whether and how multiple parameters such as cognitive skills, mood, personality, alertness, comprehension skill, and social and physical context of workers can be leveraged in tandem to improve worker performance estimations in heterogeneous micro tasks. Our predictive models indicate that these parameters have varying effects on worker performance in the five task types considered – sentiment analysis, classification, transcription, named entity recognition and bounding box. Moreover, we note 0.003 - 0.018 reduction in mean absolute error of predicted worker accuracy across all tasks, when task assignment is based on models that consider all parameters vs. models that only consider workers’ cognitive skills. Our findings pave the way for the use of holistic approaches in micro task assignment that effectively quantify worker context.
@inproceedings{Wijenayake2023-yc, title = {Combining Worker Factors for Heterogeneous Crowd Task Assignment}, booktitle = {Proceedings of the {ACM} Web Conference 2023}, author = {Wijenayake, Senuri and Hettiachchi, Danula and Goncalves, Jorge}, publisher = {Association for Computing Machinery}, pages = {3794--3805}, series = {WWW '23}, year = {2023}, address = {New York, NY, USA}, location = {Austin, TX, USA}, isbn = {9781450394161}, doi = {10.1145/3543507.3583190}, talk = {https://youtu.be/cZoBigEPAi4}, }
IJHCS
Mapping 20 years of accessibility research in HCI: A co-word analysis

Zhanna Sarsenbayeva, Niels van Berkel, Danula Hettiachchi, Benjamin Tag, Eduardo Velloso, Jorge Goncalves, and Vassilis Kostakos

International Journal of Human Computer Studies, 2023

DOI PDF

We employ hierarchical clustering, strategic diagrams, and network core–periphery analysis to assess and visualise the intellectual progress of accessibility research within HCI in the past two decades. The study quantifies and explains the development of accessibility research and its thematic evolution based on 1,535 papers published at TACCESS, ASSETS, IJHCS, and CHI and their respective 3,470 author-assigned keywords. The novelty of this work is based on employing a quantitative methodological approach to provide an overview of accessibility research progress and insights into its driving and trending themes through the period 2001–2021. In addition, we identify declining, emerging, and core backbone themes of accessibility research. Finally, we discuss the opportunities for research that arise from our findings. These contributions provide a roadmap for researchers working on accessibility.
@article{Sarsenbayeva2023-mo, title = {Mapping 20 years of accessibility research in {HCI}: A co-word analysis}, author = {Sarsenbayeva, Zhanna and {van Berkel}, Niels and Hettiachchi, Danula and Tag, Benjamin and Velloso, Eduardo and Goncalves, Jorge and Kostakos, Vassilis}, journal = {International Journal of Human Computer Studies}, volume = {175}, pages = {103018}, year = {2023}, issn = {1071-5819}, doi = {10.1016/j.ijhcs.2023.103018}, }
NTCIR
RMIT_IR at the NTCIR-17 FairWeb-1 Task

Sachin Pathiyan Cherumanal, Kaixin Ji, Danula Hettiachchi, Johanne R Trippas, Falk Scholer, and Damiano Spina

In Proceedings of 17th Conference on Evaluation of Information Access Technologies (NTCIR-17), 2023

DOI PDF Code

This report describes the participation of the RMIT IR group at the NTCIR-17 FairWeb-1 task. We submitted five runs with the aim of exploring the role of explicit search result diversification (SRD) and ranking fusion to generate fair rankings considering multiple fairness attributes. We also explored the use of a linear combination-based technique (LC) to take into consideration the relevance while re-ranking. In this report, we compared results from all our submitted runs against each other and the retrieval baselines along each topic type separately (i.e., Researcher, Movie, YouTube). Overall, our results show that neither the SRD-based runs nor the linear combination-based runs show any statistically significant improvement over the retrieval baselines. The source code of the framework for generating group memberships is made available at https://github.com/rmit-ir/fairweb-1/
@inproceedings{Cherumanal2023-or, title = {{RMIT\_IR} at the {NTCIR-17} {FairWeb-1} Task}, booktitle = {Proceedings of 17th Conference on Evaluation of Information Access Technologies}, author = {Cherumanal, Sachin Pathiyan and Ji, Kaixin and Hettiachchi, Danula and Trippas, Johanne R and Scholer, Falk and Spina, Damiano}, publisher = {National Institute of Informatics}, series = {NTCIR-17}, year = {2023}, address = {Tokyo, Japan}, doi = {10.20736/0002001315}, }
UbiComp
Towards Detecting Tonic Information Processing Activities with Physiological Data

Kaixin Ji, Damiano Spina, Danula Hettiachchi, Falk Scholer, and Flora D. Salim

In Adjunct Proceedings of the 2023 ACM International Joint Conference on Pervasive and Ubiquitous Computing & the 2023 ACM International Symposium on Wearable Computing (UbiComp/ISWC ’23 Adjunct), 2023

DOI PDF

Characterizing Information Processing Activities (IPAs) such as reading, listening, speaking, and writing, with physiological signals captured by wearable sensors can broaden the understanding of how people produce and consume information. However, sensors are highly sensitive to external conditions that are not trivial to control – not even in lab user studies. We conducted a pilot study (N = 7) to assess the robustness and sensitivity of physiological signals across four IPAs (READ, LISTEN, SPEAK, and WRITE) using multiple sensors. The collected signals include Electrodermal Activities, Blood Volume Pulse, gaze, and head motion. We observed consistent trends across participants, and ten features with statistically significant differences across the four IPAs. Our results provide preliminary quantitative evidence of differences in physiological responses when users encounter IPAs, revealing the necessity to inspect the signals separately according to the IPAs. The next step of this study moves into a specific context, information retrieval, and the IPAs are considered as the interaction modalities with the search system, for instance, submitting the search query by speaking or typing.
@inproceedings{Ji2023-td, author = {Ji, Kaixin and Spina, Damiano and Hettiachchi, Danula and Scholer, Falk and Salim, Flora D.}, title = {Towards Detecting Tonic Information Processing Activities with Physiological Data}, year = {2023}, isbn = {9798400702006}, publisher = {Association for Computing Machinery}, address = {New York, NY, USA}, url = {https://doi.org/10.1145/3594739.3610679}, doi = {10.1145/3594739.3610679}, booktitle = {Adjunct Proceedings of the 2023 ACM International Joint Conference on Pervasive and Ubiquitous Computing \& the 2023 ACM International Symposium on Wearable Computing}, pages = {1–5}, numpages = {5}, location = {Cancun, Mexico}, series = {UbiComp/ISWC '23 Adjunct}, }
CHI
Helpful, Misleading or Confusing: How Humans Perceive Fundamental Building Blocks of Artificial Intelligence Explanations

Edward Small, Yueqing Xuan, Danula Hettiachchi, and Kacper Sokol

In ACM CHI Workshop Human-Centered Perspectives in Explainable AI, 2023

PDF

Explainable artificial intelligence techniques are developed at breakneck speed, but suitable evaluation approaches lag behind. With explainers becoming increasingly complex and a lack of consensus on how to assess their utility, it is challenging to judge the benefit and effectiveness of different explanations. To address this gap, we take a step back from sophisticated predictive algorithms and instead look into explainability of simple decision-making models. In this setting, we aim to assess how people perceive comprehensibility of their different representations such as mathematical formulation, graphical representation and textual summarisation (of varying complexity and scope). This allows us to capture how diverse stakeholders – engineers, researchers, consumers, regulators and the like – judge intelligibility of fundamental concepts that more elaborate artificial intelligence explanations are built from. This position paper charts our approach to establishing appropriate evaluation methodology as well as a conceptual and practical framework to facilitate setting up and executing relevant user studies.
@inproceedings{Small2023-by, title = {Helpful, Misleading or Confusing: How Humans Perceive Fundamental Building Blocks of Artificial Intelligence Explanations}, booktitle = {{ACM} {CHI} Workshop {Human-Centered} Perspectives in Explainable {AI}}, author = {Small, Edward and Xuan, Yueqing and Hettiachchi, Danula and Sokol, Kacper}, month = mar, year = {2023}, }
CSCW
Towards Measuring Sensitivity of Psychometrics in Crowdsourcing Tasks: Engaging with Fact-checked Content Online

Stanislaus Krisna, Danula Hettiachchi, and Damiano Spina

In Proceedings of the CSCW 2023 Workshop on Undertanding and Mitigating Cognitive Biases in Human-AI Collaboration (CSCW ’23 Workshop), 2023

PDF

This study attempts to explore the relationships between psychometric test outcomes and how participants rate the truthfulness of online news articles. We seek to be able to help in the identification of misinformation. We attempt to explore the relationship between political orientation and several other psychometrics. We then ask the user to rate social media article as well as fact-checked article before finding the relationship between the variables and the difference in rating of each articles. We faced an issue in aggregating the result of each participant as well as computing the distance in ordered scale (magnitude given values in limited range).
@inproceedings{Krisna2023-tm, author = {Krisna, Stanislaus and Hettiachchi, Danula and Spina, Damiano}, booktitle = {Proceedings of the CSCW 2023 Workshop on Undertanding and Mitigating Cognitive Biases in Human-AI Collaboration}, pages = {4}, title = {Towards Measuring Sensitivity of Psychometrics in Crowdsourcing Tasks: Engaging with Fact-checked Content Online}, year = {2023}, series = {CSCW '23 Workshop}, talk = {https://www.damianospina.com/publication/krisna-2023-towards/krisna-2023-towards.mp4}, }

2022

CSUR
A Survey on Task Assignment in Crowdsourcing

Danula Hettiachchi, Vassilis Kostakos, and Jorge Goncalves

ACM Computing Surveys, 2022

DOI PDF

Quality improvement methods are essential to gathering high-quality crowdsourced data, both for research and industry applications. A popular and broadly applicable method is task assignment that dynamically adjusts crowd workflow parameters. In this survey, we review task assignment methods that address: heterogeneous task assignment, question assignment, and plurality problems in crowdsourcing. We discuss and contrast how these methods estimate worker performance, and highlight potential challenges in their implementation. Finally, we discuss future research directions for task assignment methods, and how crowdsourcing platforms and other stakeholders can benefit from them.
@article{Hettiachchi2022-vq, title = {A Survey on Task Assignment in Crowdsourcing}, author = {Hettiachchi, Danula and Kostakos, Vassilis and Goncalves, Jorge}, journal = {ACM Computing Surveys}, publisher = {ACM}, volume = {55}, number = {3}, pages = {1--35}, month = feb, year = {2022}, address = {New York, NY, USA}, issn = {0360-0300}, doi = {10.1145/3494522}, }
AJCAI
Does a Face Mask Protect My Privacy?: Deep Learning to Predict Protected Attributes from Masked Face Images

Sachith Seneviratne, Nuran Kasthuriarachchi, Sanka Rasnayaka, Danula Hettiachchi, and Ridwan Shariffdeen

In AI 2021: Advances in Artificial Intelligence (AJCAI ’21), 2022

DOI PDF

Contactless and efficient systems are implemented rapidly to advocate preventive methods in the fight against the COVID-19 pandemic. Despite the positive benefits of such systems, there is potential for exploitation by invading user privacy. In this work, we analyse the privacy invasiveness of face biometric systems by predicting privacy-sensitive soft-biometrics using masked face images. We train and apply a CNN based on the ResNet-50 architecture with 20,003 synthetic masked images and measure the privacy invasiveness. Despite the popular belief of the privacy benefits of wearing a mask among people, we show that there is no significant difference to privacy invasiveness when a mask is worn. In our experiments we were able to accurately predict sex (94.7%), race (83.1%) and age (MAE 6.21 and RMSE 8.33) from masked face images. Our proposed approach can serve as a baseline utility to evaluate the privacy-invasiveness of artificial intelligence systems that make use of privacy-sensitive information. We open-source all contributions for reproducibility and broader use by the research community.
@inproceedings{Seneviratne2022-ll, title = {Does a Face Mask Protect My Privacy?: Deep Learning to Predict Protected Attributes from Masked Face Images}, booktitle = {{AI} 2021: Advances in Artificial Intelligence}, author = {Seneviratne, Sachith and Kasthuriarachchi, Nuran and Rasnayaka, Sanka and Hettiachchi, Danula and Shariffdeen, Ridwan}, publisher = {Springer}, pages = {91--102}, year = {2022}, series = {AJCAI '21}, doi = {10.1007/978-3-030-97546-3\_8}, }
CHI
REGROW: Reimagining Global Crowdsourcing for Better Human-AI Collaboration

Andy Alorwu, Saiph Savage, Niels van Berkel, Dmitry Ustalov, Alexey Drutsa, Jonas Oppenlaender, Oliver Bates, Danula Hettiachchi, Ujwal Gadiraju, Jorge Goncalves, and Simo Hosio

In Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems (CHI EA ’22), 2022

DOI PDF

Crowdworkers silently enable much of today’s AI-based products, with several online platforms offering a myriad of data labelling and content moderation tasks through convenient labour marketplaces. The HCI community has been increasingly interested in investigating the worker-centric issues inherent in the current model and seeking for potential improvements that could be implemented in the future. This workshop explores how a reimagined perspective on crowdsourcing platforms could provide a more equitable, fair, and rewarding experience. This includes not only the workers but also the platforms, who could benefit e.g. from better processes for worker onboarding, skills-development, and growth. We invite visionary takes in various formats on this topic to spread awareness of worker-centric research and developments to the CHI community. As a result of interactive ideation work in the workshop, we articulate a future direction roadmap for research centred around crowdsourcing platforms. Finally, as a specific interest area, the workshop seeks to study crowdwork from the context of the Global South, which has been arising as an important but critically understudied crowdsourcing market in recent years.
@inproceedings{Alorwu2022-oi, title = {{REGROW}: Reimagining Global Crowdsourcing for Better {Human-AI} Collaboration}, booktitle = {Extended Abstracts of the 2022 {CHI} Conference on Human Factors in Computing Systems}, author = {Alorwu, Andy and Savage, Saiph and {van Berkel}, Niels and Ustalov, Dmitry and Drutsa, Alexey and Oppenlaender, Jonas and Bates, Oliver and Hettiachchi, Danula and Gadiraju, Ujwal and Goncalves, Jorge and Hosio, Simo}, publisher = {ACM}, number = {Article 88}, pages = {1--7}, series = {CHI EA '22}, year = {2022}, address = {New York, NY, USA}, location = {New Orleans, LA, USA}, isbn = {9781450391566}, doi = {10.1145/3491101.3503725}, }

2021

CSCW
The Challenge of Variable Effort Crowdsourcing and How Visible Gold Can Help

Danula Hettiachchi, Mike Schaekermann, Tristan McKinney, and Matthew Lease

Proceedings of the ACM on Human-Computer Interaction, 2021

DOI PDF

We consider a class of variable effort human annotation tasks in which the number of labels required per item can greatly vary (e.g., finding all faces in an image, named entities in a text, bird calls in an audio recording, etc.). In such tasks, some items require far more effort than others to annotate. Furthermore, the per-item annotation effort is not known until after each item is annotated since determining the number of labels required is an implicit part of the annotation task itself. On an image bounding-box task with crowdsourced annotators, we show that annotator accuracy and recall consistently drop as effort increases. We hypothesize reasons for this drop and investigate a set of approaches to counteract it. Firstly, we benchmark on this task a set of general best-practice methods for quality crowdsourcing. Notably, only one of these methods actually improves quality: the use of visible gold questions that provide periodic feedback to workers on their accuracy as they work. Given these promising results, we then investigate and evaluate variants of the visible gold approach, yielding further improvement. Final results show a 7% improvement in bounding-box accuracy over the baseline. We discuss the generality of the visible gold approach and promising directions for future research.
@article{Hettiachchi2021-vm, title = {The Challenge of Variable Effort Crowdsourcing and How Visible Gold Can Help}, author = {Hettiachchi, Danula and Schaekermann, Mike and McKinney, Tristan and Lease, Matthew}, journal = {Proceedings of the ACM on Human-Computer Interaction}, volume = {5}, number = {CSCW2}, pages = {26}, month = may, year = {2021}, doi = {10.1145/3476073}, talk = {https://youtu.be/uN8syzhUnRU}, }
Internet Computing
Effect of Conformity on Perceived Trustworthiness of News in Social Media

Senuri Wijenayake, Danula Hettiachchi, Simo Hosio, Vassilis Kostakos, and Jorge Goncalves

IEEE Internet Computing, 2021

DOI PDF

A catalyst for the spread of fake news is the existence of comments that users make in support of, or against, such articles. In this article, we investigate whether critical and supportive comments can induce conformity in how readers perceive trustworthiness of news articles and respond to them. We find that individuals tend to conform to the majority’s opinion of an article’s trustworthiness (58%), especially when challenged by larger majorities who are critical of the article’s credibility, or when less confident about their personal judgment. Moreover, we find that individuals who conform are more inclined to take action: to report articles they perceive as fake, and to comment on and share articles they perceive as real. We conclude with a discussion on the implications of our findings for mitigating the dispersion of fake news on social media.
@article{Wijenayake2021-pn, title = {Effect of Conformity on Perceived Trustworthiness of News in Social Media}, author = {Wijenayake, Senuri and Hettiachchi, Danula and Hosio, Simo and Kostakos, Vassilis and Goncalves, Jorge}, journal = {IEEE Internet Computing}, volume = {25}, number = {1}, pages = {12--19}, month = jan, year = {2021}, issn = {1089-7801, 1941-0131}, doi = {10.1109/MIC.2020.3032410}, }
JMIR Med. Inform.
Team Dynamics in Hospital Workflows: An Exploratory Study of a Smartphone Task Manager

Danula Hettiachchi, Lachie Hayes, Jorge Goncalves, and Vassilis Kostakos

JMIR Medical Informatics, 2021

DOI PDF

BACKGROUND: Although convenient and reliable modern messaging apps like WhatsApp enable efficient communication among hospital staff, hospitals are now pivoting toward purpose-built structured communication apps for various reasons, including security and privacy concerns. However, there is limited understanding of how we can examine and improve hospital workflows using the data collected through such apps as an alternative to costly and challenging research methods like ethnography and patient record analysis. OBJECTIVE: We seek to identify whether the structure of the collected communication data provides insights into hospitals’ workflows. Our analysis also aims to identify ways in which task management platforms can be improved and designed to better support clinical workflows. METHODS: We present an exploratory analysis of clinical task records collected over 22 months through a smartphone app that enables structured communication between staff to manage and execute clinical workflows. We collected over 300,000 task records between July 2018 and May 2020 completed by staff members including doctors, nurses, and pharmacists across all wards in an Australian hospital. RESULTS: We show that important insights into how teams function in a clinical setting can be readily drawn from task assignment data. Our analysis indicates that predefined labels such as urgency and task type are important and impact how tasks are accepted and completed. Our results show that both task sent-to-accepted (P<.001) and sent-to-completed (P<.001) times are significantly higher for routine tasks when compared to urgent tasks. We also show how task acceptance varies across teams and roles and that internal tasks are more efficiently managed than external tasks, possibly due to increased trust among team members. For example, task sent-to-accepted time (minutes) is significantly higher (P<.001) for external assignments (mean 22.10, SD 91.45) when compared to internal assignments (mean 19.03, SD 82.66). CONCLUSIONS: Smartphone-based task assignment apps can provide unique insights into team dynamics in clinical settings. These insights can be used to further improve how well these systems support clinical work and staff.
@article{Hettiachchi2021-ql, title = {Team Dynamics in Hospital Workflows: An Exploratory Study of a Smartphone Task Manager}, author = {Hettiachchi, Danula and Hayes, Lachie and Goncalves, Jorge and Kostakos, Vassilis}, journal = {JMIR Medical Informatics}, volume = {9}, number = {8}, pages = {e28245}, month = aug, year = {2021}, language = {en}, issn = {2291-9694}, doi = {10.2196/28245}, }
INTERACT
Us Vs. Them – Understanding the Impact of Homophily in Political Discussions on Twitter

Danula Hettiachchi, Tanay Arora, and Jorge Goncalves

In Human-Computer Interaction (INTERACT ’21), 2021

DOI PDF

Analysing homophily, i.e. people’s tendency to associate with others with similar social attributes, can help us unravel and better understand user behaviour in social media. In our work, we analyse the impact of homophily in discussions regarding the Citizenship Amendment Act (CAA) on Twitter. The Indian Government enacted CAA to provide relaxation in the citizenship process to religious minorities in three neighbouring countries. While it was lauded by many, it also fuelled backlash amongst some Indian citizens, resulting in the emergence of two distinctive political dispositions regarding this matter. We collected 78,004 Tweets, including 11,794 original Tweets during a period of two weeks shortly after the ruling, and examined ways of potentially reducing homophily and therefore minimise the presence of echo chambers. In particular, we investigated users’ political dispositions and expressed sentiment, and how these two social attributes influence homophilic social ties and interactions. Further, we discuss how our findings can be used in social networks to allow people with diverse viewpoints and emotional attitudes to interact with each other in a positive and constructive manner.
@inproceedings{Hettiachchi2021-fr, title = {Us Vs. Them -- Understanding the Impact of Homophily in Political Discussions on Twitter}, booktitle = {{Human-Computer} Interaction}, author = {Hettiachchi, Danula and Arora, Tanay and Goncalves, Jorge}, publisher = {Springer}, pages = {476--497}, year = {2021}, series = {INTERACT '21}, doi = {10.1007/978-3-030-85610-6\_27}, talk = {https://youtu.be/1-1-Dbw1qJ0}, }

CSCW

Investigating and Mitigating Biases in Crowdsourced Data

Danula Hettiachchi, Mark Sanderson, Jorge Goncalves, Simo Hosio, Gabriella Kazai, Matthew Lease, Mike Schaekermann, and Emine Yilmaz

In Companion Publication of the 2021 Conference on Computer Supported Cooperative Work and Social Computing (CSCW ’21 Companion), 2021

DOI PDF Website

@inproceedings{Hettiachchi2021-gf,
  title = {Investigating and Mitigating Biases in Crowdsourced Data},
  booktitle = {Companion Publication of the 2021 Conference on Computer
                       Supported Cooperative Work and Social Computing},
  author = {Hettiachchi, Danula and Sanderson, Mark and Goncalves, Jorge and Hosio, Simo and Kazai, Gabriella and Lease, Matthew and Schaekermann, Mike and Yilmaz, Emine},
  publisher = {ACM},
  year = {2021},
  series = {{CSCW} '21 Companion},
  location = {Virtual},
  doi = {10.1145/3462204.3481729},
}

ISS
Investigating Human Scale Spatial Experience

Ville Paananen, Jonas Oppenlaender, Jorge Goncalves, Danula Hettiachchi, and Simo Hosio

Proceedings of the ACM on Human-Computer Interaction, 2021

DOI PDF

Spatial experience, or how humans experience a given space, has been a pivotal topic especially in urban-scale environments. On the human scale, HCI researchers have mostly investigated personal meanings or aesthetic and embodied experiences. In this paper, we investigate the human scale as an ensemble of individual spatial features. Through large-scale online questionnaires we first collected a rich set of spatial features that people generally use to characterize their surroundings. Second, we conducted a set of field interviews to develop a more nuanced understanding of the feature identified as most important: perceived safety. Our combined quantitative and qualitative analysis contributes to spatial understanding as a form of context information and presents a timely investigation into the perceived safety of human scale spaces. By connecting our results to the broader scientific literature, we contribute to the field of HCI spatial understanding.
@article{Paananen2021-mz, title = {Investigating Human Scale Spatial Experience}, author = {Paananen, Ville and Oppenlaender, Jonas and Goncalves, Jorge and Hettiachchi, Danula and Hosio, Simo}, journal = {Proceedings of the ACM on Human-Computer Interaction}, publisher = {ACM}, volume = {5}, number = {ISS}, pages = {1--18}, year = {2021}, address = {New York, NY, USA}, keywords = {architecture, spatial experience, interaction design}, doi = {10.1145/3488541}, }

2020

CHI
“Hi! I am the Crowd Tasker” Crowdsourcing through Digital Voice Assistants

Danula Hettiachchi, Zhanna Sarsenbayeva, Fraser Allison, Niels van Berkel, Tilman Dingler, Gabriele Marini, Vassilis Kostakos, and Jorge Goncalves

In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (CHI ’20), 2020

DOI PDF

Inspired by the increasing prevalence of digital voice assistants, we demonstrate the feasibility of using voice interfaces to deploy and complete crowd tasks. We have developed Crowd Tasker, a novel system that delivers crowd tasks through a digital voice assistant. In a lab study, we validate our proof-of-concept and show that crowd task performance through a voice assistant is comparable to that of a web interface for voice-compatible and voice-based crowd tasks for native English speakers. We also report on a field study where participants used our system in their homes. We find that crowdsourcing through voice can provide greater flexibility to crowd workers by allowing them to work in brief sessions, enabling multi-tasking, and reducing the time and effort required to initiate tasks. We conclude by proposing a set of design guidelines for the creation of crowd tasks for voice and the development of future voice-based crowdsourcing systems.
@inproceedings{Hettiachchi2020-qj, title = {``Hi! {I} am the Crowd Tasker'' Crowdsourcing through Digital Voice Assistants}, booktitle = {Proceedings of the 2020 {CHI} Conference on Human Factors in Computing Systems}, author = {Hettiachchi, Danula and Sarsenbayeva, Zhanna and Allison, Fraser and {van Berkel}, Niels and Dingler, Tilman and Marini, Gabriele and Kostakos, Vassilis and Goncalves, Jorge}, publisher = {ACM}, pages = {1--14}, series = {CHI '20}, month = apr, year = {2020}, address = {New York, NY, USA}, isbn = {9781450367080}, doi = {10.1145/3313831.3376320}, }
CSCW
CrowdCog: A Cognitive Skill based System for Heterogeneous Task Assignment and Recommendation in Crowdsourcing

Danula Hettiachchi, Niels van Berkel, Vassilis Kostakos, and Jorge Goncalves

Proceedings of the ACM on Human-Computer Interaction, 2020

DOI PDF Code

While crowd workers typically complete a variety of tasks in crowdsourcing platforms, there is no widely accepted method to successfully match workers to different types of tasks. Researchers have considered using worker demographics, behavioural traces, and prior task completion records to optimise task assignment. However, optimum task assignment remains a challenging research problem due to limitations of proposed approaches, which in turn can have a significant impact on the future of crowdsourcing. We present ’CrowdCog’, an online dynamic system that performs both task assignment and task recommendations, by relying on fast-paced online cognitive tests to estimate worker performance across a variety of tasks. Our work extends prior work that highlights the effect of workers’ cognitive ability on crowdsourcing task performance. Our study, deployed on Amazon Mechanical Turk, involved 574 workers and 983 HITs that span across four typical crowd tasks (Classification, Counting, Transcription, and Sentiment Analysis). Our results show that both our assignment method and recommendation method result in a significant performance increase (5% to 20%) as compared to a generic or random task assignment. Our findings pave the way for the use of quick cognitive tests to provide robust recommendations and assignments to crowd workers.
@article{Hettiachchi2020-bc, title = {{CrowdCog}: A Cognitive Skill based System for Heterogeneous Task Assignment and Recommendation in Crowdsourcing}, author = {Hettiachchi, Danula and {van Berkel}, Niels and Kostakos, Vassilis and Goncalves, Jorge}, journal = {Proceedings of the ACM on Human-Computer Interaction}, publisher = {ACM}, volume = {4}, number = {CSCW2}, pages = {1--22}, month = oct, year = {2020}, address = {New York, NY, USA}, doi = {10.1145/3415181}, talk = {https://youtu.be/5c6ZXDGqrqk}, }
HCOMP
How Context Influences Cross-Device Task Acceptance in Crowd Work

Danula Hettiachchi, Senuri Wijenayake, Simo Hosio, Vassilis Kostakos, and Jorge Goncalves

In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing (HCOMP ’20), 2020

DOI PDF

Although crowd work is typically completed through desktop or laptop computers by workers at their home, literature has shown that crowdsourcing is feasible through a wide array of computing devices, including smartphones and digital voice assistants. An integrated crowdsourcing platform that operates across multiple devices could provide greater flexibility to workers, but there is little understanding of crowd workers’ perceptions on uptaking crowd tasks across multiple contexts through such devices. Using a crowdsourcing survey task, we investigate workers’ willingness to accept different types of crowd tasks presented on three device types in different scenarios of varying location, time and social context. Through analysis of over 25,000 responses received from 329 crowd workers on Amazon Mechanical Turk, we show that when tasks are presented on different devices, the task acceptance rate is 80.5% on personal computers, 77.3% on smartphones and 70.7% on digital voice assistants. Our results also show how different contextual factors such as location, social context and time influence workers decision to accept a task on a given device. Our findings provide important insights towards the development of effective task assignment mechanisms for cross-device crowd platforms.
@inproceedings{Hettiachchi2020-ei, title = {How Context Influences {Cross-Device} Task Acceptance in Crowd Work}, booktitle = {Proceedings of the {AAAI} Conference on Human Computation and Crowdsourcing}, author = {Hettiachchi, Danula and Wijenayake, Senuri and Hosio, Simo and Kostakos, Vassilis and Goncalves, Jorge}, publisher = {AAAI}, volume = {8}, pages = {53--62}, year = {2020}, series = {HCOMP '20}, doi = {10.1609/hcomp.v8i1.7463}, talk = {https://youtu.be/MwzZZupcSg0}, }
PACIS
Augmenting Automated Kinship Verification with Targeted Human Input

Danula Hettiachchi, Niels Berkel, Simo Hosio, Miguel Bordallo López, Vassilis Kostakos, and Jorge Goncalves

In Twenty-Fourth Pacific Asia Conference on Information Systems (PACIS ’20), 2020

PDF

Kinship verification is the problem whereby a third party determines whether two people are related. Despite previous research in Psychology and Machine Vision, the factors affecting a person’s verification ability are poorly understood. Through an online crowdsourcing study, we investigate the impact of gender, race and medium type (image vs video) on kinship verification - taking into account the demographics of both raters and ratees. A total of 325 workers completed over 50,000 kinship verification tasks consisting of pairs of faces shown in images and videos from three widely used datasets. Our results identify an own-race bias and a higher verification accuracy for same-gender image pairs than opposite-gender image pairs. Our results demonstrate that humans can still outperform current state-of-the-art automated unsupervised approaches. Furthermore, we show that humans perform better when presented with videos instead of still images. Our findings contribute to the design of future human-in-the-loop kinship verification tasks, including time-critical use cases such as identifying missing persons.
@inproceedings{Hettiachchi2020-aa, title = {Augmenting Automated Kinship Verification with Targeted Human Input}, author = {Hettiachchi, Danula and van Berkel, Niels and Hosio, Simo and L{\'o}pez, Miguel Bordallo and Kostakos, Vassilis and Goncalves, Jorge}, booktitle = {Twenty-Fourth Pacific Asia Conference on Information Systems}, pages = {141}, year = {2020}, series = {PACIS '20}, url = {https://aisel.aisnet.org/pacis2020/141}, }

2019

INTERACT
Effect of Cognitive Abilities on Crowdsourcing Task Performance

Danula Hettiachchi, Niels van Berkel, Simo Hosio, Vassilis Kostakos, and Jorge Goncalves

In Human-Computer Interaction – INTERACT 2019, 2019

DOI PDF

Matching crowd workers to suitable tasks is highly desirable as it can enhance task performance, reduce the cost for requesters, and increase worker satisfaction. In this paper, we propose a method that considers workers’ cognitive ability to predict their suitability for a wide range of crowdsourcing tasks. We measure cognitive ability via fast-paced online cognitive tests with a combined average duration of 6.2 min. We then demonstrate that our proposed method can effectively assign or recommend workers to five different popular crowd tasks: Classification, Counting, Proofreading, Sentiment Analysis, and Transcription. Using our approach we demonstrate a significant improvement in the expected overall task accuracy. While previous methods require access to worker history or demographics, our work offers a quick and accurate way to determine which workers are more suitable for which tasks.
@inproceedings{Hettiachchi2019-lo, title = {Effect of Cognitive Abilities on Crowdsourcing Task Performance}, booktitle = {{Human-Computer} Interaction -- {INTERACT} 2019}, author = {Hettiachchi, Danula and {van Berkel}, Niels and Hosio, Simo and Kostakos, Vassilis and Goncalves, Jorge}, publisher = {Springer}, pages = {442--464}, year = {2019}, doi = {10.1007/978-3-030-29381-9\_28}, }
CSCW
Crowdsourcing Perceptions of Fair Predictors for Machine Learning: A Recidivism Case Study

Niels van Berkel, Jorge Goncalves, Danula Hettiachchi, Senuri Wijenayake, Ryan M Kelly, and Vassilis Kostakos

Proceedings of the ACM on Human-Computer Interaction, 2019

DOI PDF

The increased reliance on algorithmic decision-making in socially impactful processes has intensified the calls for algorithms that are unbiased and procedurally fair. Identifying fair predictors is an essential step in the construction of equitable algorithms, but the lack of ground-truth in fair predictor selection makes this a challenging task. In our study, we recruit 90 crowdworkers to judge the inclusion of various predictors for recidivism. We divide participants across three conditions with varying group composition. Our results show that participants were able to make informed decisions on predictor selection. We find that agreement with the majority vote is higher when participants are part of a more diverse group. The presented workflow, which provides a scalable and practical approach to reach a diverse audience, allows researchers to capture participants’ perceptions of fairness in private while simultaneously allowing for structured participant discussion.
@article{Van_Berkel2019-aw, title = {Crowdsourcing Perceptions of Fair Predictors for Machine Learning: A Recidivism Case Study}, author = {{van Berkel}, Niels and Goncalves, Jorge and Hettiachchi, Danula and Wijenayake, Senuri and Kelly, Ryan M and Kostakos, Vassilis}, journal = {Proceedings of the ACM on Human-Computer Interaction}, publisher = {ACM}, volume = {3}, number = {CSCW}, pages = {1--21}, year = {2019}, address = {New York, NY, USA}, doi = {10.1145/3359130}, }
IMWUT
Measuring the Effects of Stress on Mobile Interaction

Zhanna Sarsenbayeva, Niels van Berkel, Danula Hettiachchi, Weiwei Jiang, Tilman Dingler, Eduardo Velloso, Vassilis Kostakos, and Jorge Goncalves

Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 2019

DOI PDF

Research shows that environmental factors such as ambient noise and cold ambience can render users situationally impaired, adversely affecting interaction with mobile devices. However, an internal factor which is known to negatively impact cognitive abilities – stress – has not been systematically investigated in terms of its impact on mobile interaction. In this paper, we report a study where we use the Trier Social Stress Test to induce stress on participants, and investigate its effect on three aspects of mobile interaction: target acquisition, visual search, and text entry. We find that stress reduces completion time and accuracy during target acquisition tasks, as well as completion time during visual search tasks. Finally, we are able to directly contrast the magnitude of these effects to previously published effects of environmentally-caused impairments. Our work contributes to the growing body of literature on situational impairments.
@article{Sarsenbayeva2019-mk, title = {Measuring the Effects of Stress on Mobile Interaction}, author = {Sarsenbayeva, Zhanna and {van Berkel}, Niels and Hettiachchi, Danula and Jiang, Weiwei and Dingler, Tilman and Velloso, Eduardo and Kostakos, Vassilis and Goncalves, Jorge}, journal = {Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies}, publisher = {ACM}, volume = {3}, number = {1}, pages = {1--18}, year = {2019}, address = {New York, NY, USA}, doi = {10.1145/3314411}, }
OzCHI
Towards Effective Crowd-Powered Online Content Moderation

Danula Hettiachchi and Jorge Goncalves

In Proceedings of the 31st Australian Conference on Human-Computer-Interaction (OZCHI ’19), 2019

DOI PDF

Content moderation is an important element of social computing systems that facilitates positive social interaction in online platforms. Current solutions for moderation including human moderation via commercial teams are not effective and have failed to meet the demands of growing volumes of online user generated content. Through a study where we ask crowd workers to moderate tweets, we demonstrate that crowdsourcing is a promising solution for content moderation. We also report a strong relationship between the sentiment of a tweet and its appropriateness to appear in public media. Our analysis on worker responses further reveals several key factors that affect the judgement of crowd moderators when deciding on the suitability of text content. Our findings contribute towards the development of future robust moderation systems that utilise crowdsourcing.
@inproceedings{Hettiachchi2019-fi, title = {Towards Effective {Crowd-Powered} Online Content Moderation}, booktitle = {Proceedings of the 31st Australian Conference on {Human-Computer-Interaction}}, author = {Hettiachchi, Danula and Goncalves, Jorge}, publisher = {ACM}, pages = {342--346}, series = {OZCHI '19}, year = {2019}, address = {New York, NY, USA}, location = {Fremantle, WA, Australia}, isbn = {9781450376969}, doi = {10.1145/3369457.3369491}, }

INTERACT

Effect of Ambient Light on Mobile Interaction

Zhanna Sarsenbayeva, Niels van Berkel, Weiwei Jiang, Danula Hettiachchi, Vassilis Kostakos, and Jorge Goncalves

In Human-Computer Interaction (INTERACT ’19), 2019

DOI PDF

@inproceedings{Sarsenbayeva2019-cn,
  title = {Effect of Ambient Light on Mobile Interaction},
  author = {Sarsenbayeva, Zhanna and {van Berkel}, Niels and Jiang, Weiwei and Hettiachchi, Danula and Kostakos, Vassilis and Goncalves, Jorge},
  booktitle = {Human-Computer Interaction},
  pages = {465--475},
  year = {2019},
  series = {INTERACT '19},
  doi = {10.1007/978-3-030-29387-1\_26},
}

CHI Workshop

Enabling Creative Crowd Work Through Smart Speakers

Danula Hettiachchi, Niels van Berkel, Tilman Dingler, Fraser Allison, Vassilis Kostakos, and Jorge Goncalves

In Workshop on Designing Crowd-powered Creativity Support Systems (CHI’19 Workshop), 2019

DOI PDF

@inproceedings{Hettiachchi2019-pw,
  title = {Enabling Creative Crowd Work Through Smart Speakers},
  booktitle = {Workshop on Designing Crowd-powered Creativity Support Systems},
  author = {Hettiachchi, Danula and {van Berkel}, Niels and Dingler, Tilman and Allison, Fraser and Kostakos, Vassilis and Goncalves, Jorge},
  pages = {1--5},
  series = {{CHI'19} Workshop},
  year = {2019},
  doi = {10.5281/zenodo.2648986},
}