Decision Making in the Presence of Uncertainty: Foundations of AI Lecture 19, Study notes of Computer Science

Download Decision Making in the Presence of Uncertainty: Foundations of AI Lecture 19 and more Study notes Computer Science in PDF only on Docsity! 1 CS 2710 Foundations of AI CS 2710 Foundations of AI Lecture 19 Milos Hauskrecht milos@cs.pitt.edu 5329 Sennott Square Decision making in the presence of uncertainty CS 2710 Foundations of AI Decision-making in the presence of uncertainty • Many real-world problems require to choose future actions in the presence of uncertainty • Examples: patient management, investments Main issues: • How to model the decision process in the computer ? • How to make decisions about actions in the presence of uncertainty? 2 CS 2710 Foundations of AI Decision tree representation of the problem Investing $100 for 6 months Stock 1 Stock 2 Bank 0.6 0.4 110 90 0.4 0.6 140 80 101 1.0 100 1.0 Home (up) (down) (up) (down) CS 2710 Foundations of AI Expected value Stock 1 0.6 0.4 110 90100 • Let X be a random variable representing the monetary outcome with a discrete set of values . • Expected value of X is: • Expected value summarizes all stochastic outcomes into a single quantity • Example: Expected value for the outcome of the Stock 1 option is: ∑ Ω∈ == Xx xXxPXE )()( XΩ 1023666904.01106.0 =+=×+× 102 5 CS 2710 Foundations of AI • But this may not be the case. In decision trees: – Later outcomes can be conditioned on the earlier stochastic outcomes and actions Example: stock movement probabilities. Assume: P(1st=up)=0.4 P(2nd=up|1st=up)=0.4 P(2nd=up|1st=down)=0.5 Conditioning in the decision tree Stock Bank 0.6 200 130 0.4 0.6 60 90 100 0.4 1.0 1.0 0.5 0.5 125 (2nd up) (2nd down) (2nd up) (2nd down) (1st up) (1st down) Stock Bank Stock Bank CS 2710 Foundations of AI Multi-step problems. Conditioning. Tree Structure: every observed stochastic outcome = 1 branch P(1st=up)=0.4 P(2nd=up|1st=up)=0.4 P(2nd=up|1st=down)=0.5 140 105 0.6 0.4 0.6 0.4 1.0 1.0 0.5 0.5 (2nd up) (2nd down) (2nd up) (2nd down) (1st up) (1st down) 140 80 105 80 Stock Bank Stock Bank Bank Stock 0.6 0.4 0.6 0.4 1.0 1.0 0.5 0.5 (2nd up) (2nd down) (2nd up) (2nd down) (1st up) (1st down) 200 130 60 90 100 125 Stock Bank Stock Bank 6 CS 2710 Foundations of AI Trajectory payoffs • Outcome values at leaf nodes (e.g. monetary values) – Rewards and costs for the path trajectory Example: stock fees and gains. Assume: Fee per period: $5 paid at the beginning Gain for up: 15%, loss for down 10% Stock Bank 0.6 0.4 0.6 0.4 1.0 1.0 0.5 0.5 (2nd up) (2nd down) (2nd up) (2nd down) (1st up) (1st down) Stock Bank Stock Bank 1000 1000-5 (1000-5)*1.15 (1000-5)*1.15-5 [(1000-5)*1.15-5]*1.15=1310.14 [(1000-5)*1.15-5]*0.9=1025.33 1310.14 1025.33 CS 2710 Foundations of AI • The decision tree is rarely given to you directly. – Part of the problem is to construct the tree. Example: stocks, bonds, bank for k periods Stock: – Probability of stocks going up in the first period: 0.3 – Probability of stocks going up in subsequent periods: • P(kth step=Up| (k -1)th step =Up)=0.4 • P(kth step =Up| (k -1)th step=Down)=0.5 – Return if stock goes up: 15 % if down: 10% – Fixed fee per investment period: $5 Bonds: – Probability of value up: 0.5, down: 0.5 – Return if bond value is going up: 7%, if down: 3% – Fee per investment period: $2 Bank: – Guaranteed return of 3% per period, no fee Constructing a decision tree 7 CS 2710 Foundations of AI Information-gathering actions • Some actions and their outcomes irreversibly change the world • Information-gathering (exploratory) actions: – make an inquiry about the world – Key benefit: reduction in the uncertainty • Example: medicine – Assume a patient is admitted to the hospital with some set of initial complaints – We are uncertain about the underlying problem and consider a surgery, or a medication to treat them – But there are often lab tests or observations that can help us to determine more closely the disease the patient suffers from – Goal of lab tests: Reduce the uncertainty of outcomes of treatments so that better treatment option can be chosen CS 2710 Foundations of AI Decision-making with exploratory actions In decision trees: • Exploratory actions can be represented and reasoned about the same way as other actions. How do we capture the effect of exploratory actions in the decision tree model? • Information obtained through exploratory actions may affect the probabilities of later outcomes – Recall that the probabilities on later outcomes can be conditioned on past observed outcomes and past actions – Sequence of past actions and outcomes is “remembered” within the decision tree branch 10 CS 2710 Foundations of AI Oil wildcatter problem. Drill 0.4 0.6 220-70=150 No-drill 0 1.0 -70 18 0 • Alternative model Drill 0.16 0.84 220-70-10=140 No-drill 0-10=-10 1.0 0-70-10=-80 18 0 Drill 0.64 0.36 220-70-10=140 No-drill 0-10=-10 1.0 0-70-10=-80 18 0 0 Test NoTest 0.5 0.5 (closed) (diffuse) (oil) (oil) (oil) (no-oil) (no-oil) (no-oil) CS 2710 Foundations of AI Oil wildcatter problem. • Decision tree probabilities Drill 0.64 0.36 220-70-10=140 No-drill 0-10=-10 1.0 0-70-10=-80 18 0 0 Test NoTest 0.5 0.5 (closed) (oil) (oil) (no-oil) )|( closedTestOil =P )( )()|()|( closedTestP TOilPTOilclosedTestPclosedTestTOilP = === === )( )()|()|( closedTP FOilPFOilclosedTestPclosedTestFOilP = === === )()|()()|()( TOilPTOilclosedTestPFOilPFOilclosedTestPclosedTestP ===+===== 11 CS 2710 Foundations of AI Oil wildcatter problem. • Decision tree probabilities Drill 0.64 0.36 220-70-10=140 No-drill 0-10=-10 1.0 0-70-10=-80 18 0 0 Test NoTest 0.5 0.5 (closed) (oil) (oil) (no-oil) )(TestP )()|()()|()( TOilPTOilclosedTestPFOilPFOilclosedTestPclosedTestP ===+===== (diffuse) )()|()()|()( TOilPTOildiffTestPFOilPFOildiffTestPdiffTestP ===+===== CS 2710 Foundations of AI Oil wildcatter problem. Drill 0.4 0.6 220-70=150 No-drill 0 1.0 -70 18 0 • Decision tree Drill 0.16 0.84 220-70-10=140 No-drill 0-10=-10 1.0 0-70-10=-80 -44.8 -10 Drill 0.64 0.36 220-70-10=140 No-drill 0-10=-10 1.0 0-70-10=-80 60.8 -10 0 Test NoTest 0.5 0.5 (closed) (diffuse) (oil) (oil) (oil) (no-oil) (no-oil) (no-oil) 60.8 -10 18 18 25.4 12 CS 2710 Foundations of AI Oil wildcatter problem. Drill 0.4 0.6 220-70=150 No-drill 0 1.0 -70 18 0 • Decision tree Drill 0.16 0.84 220-70-10=140 No-drill 0-10=-10 1.0 0-70-10=-80 -44.8 -10 Drill 0.64 0.36 220-70-10=140 No-drill 0-10=-10 1.0 0-70-10=-80 60.8 -10 0 Test NoTest 0.5 0.5 (closed) (diffuse) (oil) (oil) (oil) (no-oil) (no-oil) (no-oil) 60.8 -10 18 18 25.4 The presence of the test and its result affected our decision: if test =closed then drill if test=diffuse then do not drill CS 2710 Foundations of AI Value of information • When the test makes sense? • Only when its result makes the decision maker to change his mind, that is he decides not to drill. • Value of information: – Measure of the goodness of the information from the test – Difference between the expected value with and without the test information • Oil wildcatter example: – Expected value without the test = 18 – Expected value with the test =25.4 – Value of information for the seismic test = 7.4