Decision Making in the Presence of Uncertainty - Lecture Slides | CS 1571, Study notes of Computer Science

Download Decision Making in the Presence of Uncertainty - Lecture Slides | CS 1571 and more Study notes Computer Science in PDF only on Docsity! 1 CS 1571 Intro to AI M. Hauskrecht CS 1571 Introduction to AI Lecture 26 Milos Hauskrecht milos@cs.pitt.edu 5329 Sennott Square Decision making in the presence of uncertainty CS 1571 Intro to AI M. Hauskrecht Selection based on expected values • Until now: The optimal action choice was the option that maximized the expected monetary value. • But is the expected monetary value always the quantity we want to optimize? Stock 1 Stock 2 Bank 0.6 0.4 110 90 0.4 0.6 140 80 101 1.0 100 1.0 Home 102 104 101 100 (up) (down) (up) (down) 2 CS 1571 Intro to AI M. Hauskrecht Selection based on expected values • Is the expected monetary value always the quantity we want to optimize? • Answer: Yes, but only if we are risk-neutral. • But what if we do not like the risk (we are risk-averse)? • In that case we may want to get the premium for undertaking the risk (of loosing the money) • Example: – we may prefer to get $101 for sure against $102 in expectation but with the risk of loosing the money • Problem: How to model decisions and account for the risk? • Solution: use utility function, and utility theory CS 1571 Intro to AI M. Hauskrecht Utility function • Utility function (denoted U) – Quantifies how we “value” outcomes, i.e., it reflects our preferences – Can be also applied to “value” outcomes other than money and gains (e.g. utility of a patient being healthy, or ill) • Decision making: – uses expected utilities (denoted EU) the utility of outcome x Important !!! • Under some conditions on preferences we can always design the utility function that fits our preferences )()()( xXUxXPXEU Xx === ∑ Ω∈ )( xXU = 5 CS 1571 Intro to AI M. Hauskrecht Utility functions We can design a utility function that fits our preferences if they satisfy the axioms of utility theory. • But how to design the utility function for monetary values so that they incorporate the risk? • What is the relation between utility function and monetary values? • Assume we loose or gain $1000. – Typically this difference is more significant for lower values (around $100 -1000) than for higher values (~ $1,000,000) • What is the relation between utilities and monetary value for a typical person? CS 1571 Intro to AI M. Hauskrecht Utility functions • What is the relation between utilities and monetary value for a typical person? • Concave function that flattens at higher monetary values utility Monetary value100,000 6 CS 1571 Intro to AI M. Hauskrecht Utility functions • Expected utility of a sure outcome of 750 is 750 utility Monetary value1000500 750 EU(sure 750) U(x) CS 1571 Intro to AI M. Hauskrecht Utility functions Assume a lottery L [0.5: 500, 0.5:1000] • Expected value of the lottery = 750 • Expected utility of the lottery EU(L) is different: – EU(L) = 0.5U(500) + 0.5*U(1000) utility Monetary value1000500 750 EU line for lotteries with outcomes 500 and 1000EU(lottery L) Lottery L: [0.5: 500, 0.5:1000] U(x) 7 CS 1571 Intro to AI M. Hauskrecht Utility functions • Expected utility of the lottery EU(lottery L) < EU(sure 750) • Risk aversion – a bonus is required for undertaking the risk utility Monetary value1000500 750 EU(lottery L) EU(sure 750) Lottery L: [0.5: 500, 0.5:1000] U(x) CS 1571 Intro to AI M. Hauskrecht Decision making with utility function • Original problem with monetary outcomes Stock 1 Stock 2 Bank 0.6 0.4 110 90 0.4 0.6 140 80 101 1.0 100 1.0 Home 102 104 101 100 (up) (down) (up) (down) 10 CS 1571 Intro to AI M. Hauskrecht Types of learning • Supervised learning – Learning mapping between inputs x and desired outputs y – Teacher gives me y’s for the learning purposes • Unsupervised learning – Learning relations between data components – No specific outputs given by a teacher • Reinforcement learning – Learning mapping between inputs x and desired outputs y – Critic does not give me y’s but instead a signal (reinforcement) of how good my answer was • Other types of learning: – Concept learning, explanation-based learning, etc. CS 1571 Intro to AI M. Hauskrecht Supervised learning Data: a set of n examples is input vector, and y is desired output (given by a teacher) Objective: learn the mapping s.t. Two types of problems: • Regression: X discrete or continuous Y is continuous • Classification: X discrete or continuous Y is discrete },..,,{ 21 ndddD = >=< iii yd ,x ix YXf →: nixfy ii ,..,1allfor)( =≈ 11 CS 1571 Intro to AI M. Hauskrecht Supervised learning examples • Regression: Y is continuous Debt/equity Earnings company stock price Future product orders • Classification: Y is discrete Handwritten digit (array of 0,1s) Label “3” CS 1571 Intro to AI M. Hauskrecht Unsupervised learning • Data: vector of values No target value (output) y • Objective: – learn relations between samples, components of samples Types of problems: • Clustering Group together “similar” examples, e.g. patient cases • Density estimation – Model probabilistically the population of samples, e.g. relations between the diseases, symptoms, lab tests etc. },..,,{ 21 ndddD = iid x= 12 CS 1571 Intro to AI M. Hauskrecht Unsupervised learning example. • Density estimation. We want to build the probability model of a population from which we draw samples -2 -1.5 -1 -0.5 0 0.5 1 1.5 2 -1 -0.5 0 0.5 1 1.5 2 2.5 3 iid x= CS 1571 Intro to AI M. Hauskrecht Unsupervised learning. Density estimation • A probability density of a point in the two dimensional space – Model used here: Mixture of Gaussians 15 CS 1571 Intro to AI M. Hauskrecht Learning • Choosing a parametric model or a set of models is not enough Still too many functions – One for every pair of parameters a, b baxxf +=)( -2.5 -2 -1.5 -1 -0.5 0 0.5 1 1.5 2 -10 -8 -6 -4 -2 0 2 4 6 8 10 x y CS 1571 Intro to AI M. Hauskrecht Learning • Optimize the model using some criteria that reflects the fit of the model to data • Example: mean squared error -2.5 -2 -1.5 -1 -0.5 0 0.5 1 1.5 2 -10 -8 -6 -4 -2 0 2 4 6 8 10 x y 2 1 ))((1 ii n i xfy n −∑ = 16 CS 1571 Intro to AI M. Hauskrecht Typical learning Assume we have an access to the dataset D (past data) Three basic steps: • Select a model with parameters • Select the error function to be optimized – Reflects the goodness of fit of the model to the data • Find the set of parameters optimizing the error function – The model and parameters with the smallest error represent the best fit of the model to the data baxxf +=)( 2 1 ))((1 ii n i xfy n −∑ =