Mindstorms in natural language-based societies of mind  

作　　者：Mingchen Zhuge Haozhe Liu Francesco Faccio Dylan R.Ashley Róbert Csordás Anand Gopalakrishnan Abdullah Hamdi Hasan Abed Al Kader Hammoud Vincent Herrmann Kazuki Irie Louis Kirsch Bing Li Guohao Li Shuming Liu Jinjie Mai Piotr Piękos Aditya A.Ramesh Imanol Schlag Weimin Shi Aleksandar Stanić Wenyi Wang Yuhui Wang Mengmeng Xu Deng-Ping Fan Bernard Ghanem and Jürgen Schmidhuber 

机构地区：[1]Center of Excellence for Generative AI,King Abdullah University of Science and Technology,Thuwal,Saudi Arabia [2]Dalle Molle Institute for Artificial Intelligence Research,Lugano,Switzerland [3]Stanford University,California,USA [4]Oxford University,Oxford,UK [5]Harvard University,Cambridge,USA [6]ETH AI Center,Zurich,Switzerland [7]Beihang University,Beijing,China [8]CS&VCIP,Nankai University,Tianjin,China

出　　处：《Computational Visual Media》2025年第1期29-81,共53页计算可视媒体(英文版)

基　　金：supported by the European Research Council(ERC,Advanced Grant Number 742870;the Swiss National Science Foundation(SNF,Grant Numbers 200021 and 192356);the National Natural Science Foundation of China(Grant Number 62476143).

摘　　要：Inspired by Minsky’s Society of Mind,Schmidhuber’s Learning to Think,and other more 9-16 recent works,this paper proposes and advocates for the concept of natural language-based societies of mind(NLSOMs).We imagine these societies as consisting of a collection of multimodal neural networks,including large language models,which engage in a“mindstorm”to solve problems using a shared natural language interface.Here,we work to identify and discuss key questions about the social structure,governance,and economic principles for NLSOMs,emphasizing their impact on the future of AI.Our demonstrations with NLSOMs—which feature up to 129 agents—show their effectiveness in various tasks,including visual question answering,image captioning,and prompt generation for text-to-image synthesis.

关 键 词：mindstorm society of mind(SOM) large languagemodels(LLMs) multimodal learning learning to think 

分 类 号：TP18[自动化与计算机技术—控制理论与控制工程]