Do we have the knowledge we need? Rethinking human-AI decision-making in corporations

Researchers have proposed frameworks to improve AI decision-making by allocating agency between humans and AI based on task attributes and knowledge availability.

A recent study^[1] suggests that organizational knowledge is fragmented across various software systems, tacit expertise, and manual documents, and that AI systems require access to this knowledge to make decisions. The researchers propose a framework that maps task attributes and knowledge availability to recommended agency allocations and control mechanisms. Meanwhile, another study^[2] explores the use of Contextual Multi-Armed Bandits (CMABs) for non-episodic decision-making problems with textual and numerical information. The study found that lightweight numerical bandits operating on text embeddings can match or exceed the accuracy of Large Language Model (LLM)-based solutions at a fraction of their cost. The researchers also introduced LLMP-UCB, a bandit algorithm that derives uncertainty estimates from LLMs via repeated inference. The proposed frameworks have implications for the development of more effective human-AI decision-making systems.