优化策略模型下的匹配律算法  

Algorithm of matching law based on optimal policy search model

作　　者：程振波[1,2,3] 邓志东[1,2,3] 

机构地区：[1]清华大学智能技术与系统国家重点实验室,北京100084 [2]清华信息科学与技术国家实验室,北京100084 [3]清华大学计算机科学与技术系,北京100084

出　　处：《东南大学学报（自然科学版）》2009年第S1期146-151,共6页Journal of Southeast University：Natural Science Edition

基　　金：国家自然科学基金资助项目(60621062;60775040)

摘　　要：利用基于部分可观马尔可夫决策过程的策略搜索模型,提出了一种具有优化行为的策略搜索算法,并推导出满足匹配律的策略算法.被试可通过调整策略参数,最大化目标值函数的期望值,并根据已往的经验调整策略参数.假定被试所处的环境具有马尔可夫性,通过计算值函数期望值的梯度可求得优化行为的策略搜索算法.理论分析与仿真结果表明,如果策略参数与值函数的期望值仅受当前经验的影响,则可由获得优化行为的策略算法推导出符合匹配律的策略算法.研究结果揭示了匹配行为与优化策略搜索算法之间的关系,表明满足匹配律的决策行为是一类达到次优的决策行为.Based on the policy search algorithm in partially observable Markov decision process(POMDP),an optimal policy search algorithm is proposed.An algorithm leading to matching law is then derived from the optimal algorithm.The aim of the subject can find a policy parameter that can maximize the expected value of a value function,and the policy parameter is updated on the experience of the subject.Due to the Markov assumption for the environment,the optimal policy algorithm can be obtained from computing the gradient of the expected value of the value function.Theoretical analysis and simulation results show that the decision behavior achieved by this algorithm is able to reach matching law.The matching law can be met if one subject tries to maximize the expected value of the value function under the simple assumption that past choice behaviors do not affect the expected value of the value function and the current policy.It reveals the relationship between the matching behavior and the optimal policy search algorithm,and suggests that the matching behavior is a suboptimal decision behavior.

关 键 词：部分可观马尔可夫决策过程 再励学习 优化策略搜索 匹配律 

分 类 号：TP301.6[自动化与计算机技术—计算机系统结构]