Reinforcement learning and meta-decision-making

A key aspect of cognitive flexibility is to efficiently make use of earlier experience to attain one's goals. This requires learning, but also a modular, and more specifically hierarchical, structure. We hold that both are required, but combining them leads to several computational challenges that brains and artificial agents (learn to) deal with. In a hierarchical structure, metadecisions must be made, of which two types can be distinguished. First, a (meta-)decision may involve choosing which (lower-level) modules to select (module choice). Second, it may consist of choosing appropriate parameter settings within a module (parameter tuning). Furthermore, prediction error monitoring may allow determining the right meta-decision (module choice or parameter tuning). We discuss computational challenges and empirical evidence relative to how these two meta-decisions may be implemented to support learning for cognitive flexibility.