Trust-Aware Human–AI Collaborative Decision Making Using Dual-Process Large Language Model Architectures
Keywords:
trust-aware AI, dual-process theory, large language models, human-AI collaboration, decision making, cognitive architecture, governance, fairnessAbstract
The integration of large language models into human decision-making processes offers transformative potential across sectors such as healthcare, finance, law, and public administration. However, the opacity, unpredictability, and occasional hallucinations of these models challenge the development of appropriate trust between human users and AI systems. This paper proposes a trust-aware collaborative decision-making framework grounded in dual-process cognitive architectures, inspired by Kahneman’s distinction between fast, intuitive reasoning and slow, deliberative reasoning. By designing large language model systems that explicitly separate rapid pattern-based responses from slower, verifiable reasoning steps, we create a structural foundation for calibrated human trust. The architecture introduces a meta-cognitive monitor that assesses confidence, uncertainty, and potential bias in both fast and slow pathways, enabling transparent communication of model reliability to human collaborators. We examine the structural trade-offs involved in deploying such systems, including computational overhead, latency, interpretability, and user interface design. Governance implications are discussed, particularly regarding auditability, accountability, and the need for dynamic trust calibration mechanisms. The paper further explores robustness against adversarial inputs, fairness across demographic groups, and sustainability of dual-process deployments at scale. Through case illustrations in medical diagnosis, legal reasoning, and crisis management, we demonstrate how the architecture can foster appropriate reliance and mitigate over-trust or under-trust. Cross-domain comparisons with existing human-AI interaction paradigms reveal that dual-process architectures offer unique advantages for high-stakes environments where both speed and accuracy are critical. The paper concludes with forward-looking recommendations for policy, system design, and empirical validation.
References
1. D. Amodei, C. Olah, J. Steinhardt, P. Christiano, J. Schulman, and D. Mané, “Concrete problems in AI safety,” arXiv preprint arXiv:1606.06565, 2016.
2. Z. C. Lipton, “The mythos of model interpretability,” Queue, vol. 16, no. 3, pp. 31–57, 2018.
3. D. Kahneman, Thinking, Fast and Slow. Farrar, Straus and Giroux, 2011.
4. J. W. Johnson, “Applying dual-process theory to artificial intelligence: A conceptual framework,” AI & Society, vol. 37, pp. 1123–1135, 2022.
5. S. Das, K. Patel, and L. Wang, “Fast and slow reasoning in large language models: A survey,” arXiv preprint arXiv:2403.15241, 2024.
6. R. K. E. Bellamy, K. Dey, M. Hind, et al., “AI Fairness 360: An extensible toolkit for detecting and mitigating algorithmic bias,” IBM Journal of Research and Development, vol. 63, no. 4/5, pp. 4:1–4:15, 2019.
7. J. Wei, X. Wang, D. Schuurmans, et al., “Chain-of-thought prompting elicits reasoning in large language models,” in Advances in Neural Information Processing Systems, 2022, pp. 24824–24837.
8. X. Wang, J. Wei, D. Schuurmans, et al., “Self-consistency improves chain of thought reasoning in language models,” in International Conference on Learning Representations, 2023.
9. P. Lewis, E. Perez, A. Piktus, et al., “Retrieval-augmented generation for knowledge-intensive NLP tasks,” in Advances in Neural Information Processing Systems, 2020, pp. 9459–9474.
10. H. Chase, “LangChain: Building applications with LLMs through composability,” GitHub repository, 2022.
11. J. D. Lee and K. A. See, “Trust in automation: Designing for appropriate reliance,” Human Factors, vol. 46, no. 1, pp. 50–80, 2004.
12. C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger, “On calibration of modern neural networks,” in International Conference on Machine Learning, 2017, pp. 1321–1330.
13. S. R. K. Branco, F. K. Mendes, and M. H. Oliveira, “Visual trust indicators for human-AI interaction: A systematic review,” International Journal of Human-Computer Studies, vol. 180, 103145, 2023.
14. I. D. Raji, M. C. Scheuerman, and R. Amironesei, “You can’t sit with us: Exclusionary bias in AI systems,” in Conference on Fairness, Accountability, and Transparency, 2021, pp. 187–197.
15. European Commission, “Proposal for a Regulation laying down harmonised rules on artificial intelligence (Artificial Intelligence Act),” COM/2021/206 final, 2021.
16. N. Papernot, P. McDaniel, X. Wu, S. Jha, and A. Swami, “Distillation as a defense to adversarial perturbations against deep neural networks,” in IEEE Symposium on Security and Privacy, 2016, pp. 582–597.
17. Dou, Z., Cui, D., Yan, J., Wang, W., Chen, B., Wang, H., ... & Zhang, S. (2025). Dsadf: Thinking fast and slow for decision making. arXiv preprint arXiv:2505.08189.
18. E. Strubell, A. Ganesh, and A. McCallum, “Energy and policy considerations for deep learning in NLP,” in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019, pp. 3645–3650.
19. M. Turpin, J. Michael, E. Perez, and S. Bowman, “When to use slow thinking: Adaptive reasoning with large language models,” arXiv preprint arXiv:2402.04825, 2024.
20. A. J. London, “Artificial intelligence in medicine: Overcoming or recapitulating structural challenges?,” New England Journal of Medicine, vol. 380, pp. 1193–1195, 2019.
21. C. Rudin, “Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead,” Nature Machine Intelligence, vol. 1, pp. 206–215, 2019.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2026 Computer Science and Engineering Transactions
This work is licensed under a Creative Commons Attribution 4.0 International License.
This article is published under the Creative Commons Attribution 4.0 International License (CC BY 4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
