Quiz: Markov Decision Process

Question 1

Is the following statement True or False?

If the only difference between two MDPs is the value of the discount factor then they must have the same optimal policy.

True

False

Question 2

Is the following statement True or False?

For an infinite horizon MDP with a finite number of states and actions and with a discount factor γ that satisfies 0 < γ < 1, value iteration is guaranteed to converge.

True

False

Question 3

Is the following statement True or False?

When running value iteration, if the policy (the greedy policy with respect to the values) has converged, the values must have converged as well.

True

False

Question 4

Is the following statement True or False?

If one is using value iteration and the values have converged, the policy must have converged as well.

True

False

Question 5

Is the following statement True or False?

For an infinite horizon MDP with a finite number of states and actions and with a discount factor γ that satisfies 0 < γ < 1, policy iteration is guaranteed to converge.

True

False

Question 6

Is the following statement True or False?

“Q-values” are determined by immediate expected reward plus the best utility from the next state onwards.

True

False

Question 7

Note: One round of policy iteration = performing policy evaluation followed by performing policy improvement.

Is the following statement True or False?

It is guaranteed that ∀s ∈ S : V _{π James} (s) ≥ V _{π Alvin} (s)

True

False

Question 8

Note: One round of policy iteration = performing policy evaluation followed by performing policy improvement.

Is the following statement True or False?

It is guaranteed that ∀s ∈ S : V _{π Michael} (s) ≥ V _{π Alvin} (s)

True

False

Question 9

Note: One round of policy iteration = performing policy evaluation followed by performing policy improvement.

Is the following statement True or False?

It is guaranteed that ∀s ∈ S : V _{π Michael} (s) > V _{π John} (s)

True

False

Question 10

Note: One round of policy iteration = performing policy evaluation followed by performing policy improvement.

Is the following statement True or False?

It is guaranteed that ∀s ∈ S : V _{π James} (s) > V _{π John} (s)

True

False

Question 11

Is the following statement about value iteration True or False? We assume the MDP has a finite number of actions and states, and that the discount factor satisfies 0 < γ < 1.

Value iteration is guaranteed to converge.

True

False

Question 12

Is the following statement about value iteration True or False? We assume the MDP has a finite number of actions and states, and that the discount factor satisfies 0 < γ < 1.

Value iteration will converge to the same vector values (V*) no matter what values we use to initialize V.

True

False

Question 13

What is the value of the discount factor γ if we want to maximize immediate rewards?

Question 14

In an MDP with consisting of 2 states and finite action space consisting of 4 actions, what is the dimension of the transition probability matrix? (multiply all dimensions together)

Question 15

Consider the gridworld MDP for which Left and Right actions are always successful. Specifically, the available actions in each state are to move to the neighboring grid squares. From state a, there is also an exit action available, which results in going to the terminal state and collecting a reward of 10. Similarly, in state e, the reward for the exit action is 1. Exit actions are always successful. Immediate rewards at other squares are 0. Let the discount factor γ = 0.2. Fill in the following quantity. V*(a) = V _∞ (a) =

Question 16

Consider the gridworld MDP for which Left and Right actions are always successful. Specifically, the available actions in each state are to move to the neighboring grid squares. From state a, there is also an exit action available, which results in going to the terminal state and collecting a reward of 10. Similarly, in state e, the reward for the exit action is 1. Exit actions are always successful. Immediate rewards at other squares are 0. Let the discount factor γ = 0.2. Fill in the following quantity. V*(b) = V _∞ (b) =

Question 17

Consider the gridworld MDP for which Left and Right actions are always successful. Specifically, the available actions in each state are to move to the neighboring grid squares. From state a, there is also an exit action available, which results in going to the terminal state and collecting a reward of 10. Similarly, in state e, the reward for the exit action is 1. Exit actions are always successful. Immediate rewards at other squares are 0. Let the discount factor γ = 0.2. Fill in the following quantity. V*(c) = V _∞ (c) =

Question 18

Consider the gridworld MDP for which Left and Right actions are always successful. Specifically, the available actions in each state are to move to the neighboring grid squares. From state a, there is also an exit action available, which results in going to the terminal state and collecting a reward of 10. Similarly, in state e, the reward for the exit action is 1. Exit actions are always successful. Immediate rewards at other squares are 0. Let the discount factor γ = 0.2. Fill in the following quantity. V*(d) = V _∞ (d) =

Question 19

Consider the gridworld MDP for which Left and Right actions are always successful. Specifically, the available actions in each state are to move to the neighboring grid squares. From state a, there is also an exit action available, which results in going to the terminal state and collecting a reward of 10. Similarly, in state e, the reward for the exit action is 1. Exit actions are always successful. Immediate rewards at other squares are 0. Let the discount factor γ = 0.2. Fill in the following quantity. V*(e) = V _∞ (e) =

Question 20

Consider the gridworld where Left and Right actions are always successful. Specifically, the available actions in each state are to move to the neighboring grid squares. From state a, there is also an exit action available, which results in going to the terminal state and collecting a reward of 10. Similarly, in state e, the reward for the exit action is 1. Exit actions are always successful. The discount factor γ = 1. Consider the policy π shown below, and evaluate the following quantity for this policy.

V _π (a) =

Question 21

Consider the gridworld where Left and Right actions are always successful. Specifically, the available actions in each state are to move to the neighboring grid squares. From state a, there is also an exit action available, which results in going to the terminal state and collecting a reward of 10. Similarly, in state e, the reward for the exit action is 1. Exit actions are always successful. The discount factor γ = 1. Consider the policy π shown below, and evaluate the following quantity for this policy.

V _π (b) =

Question 22

Consider the gridworld where Left and Right actions are always successful. Specifically, the available actions in each state are to move to the neighboring grid squares. From state a, there is also an exit action available, which results in going to the terminal state and collecting a reward of 10. Similarly, in state e, the reward for the exit action is 1. Exit actions are always successful. The discount factor γ = 1. Consider the policy π shown below, and evaluate the following quantity for this policy.

V _π (c) =

Question 23

Consider the gridworld where Left and Right actions are always successful. Specifically, the available actions in each state are to move to the neighboring grid squares. From state a, there is also an exit action available, which results in going to the terminal state and collecting a reward of 10. Similarly, in state e, the reward for the exit action is 1. Exit actions are always successful. The discount factor γ = 1. Consider the policy π shown below, and evaluate the following quantity for this policy.

V _π (d) =

Question 24

Consider the gridworld where Left and Right actions are always successful. Specifically, the available actions in each state are to move to the neighboring grid squares. From state a, there is also an exit action available, which results in going to the terminal state and collecting a reward of 10. Similarly, in state e, the reward for the exit action is 1. Exit actions are always successful. The discount factor γ = 1. Consider the policy π shown below, and evaluate the following quantity for this policy.

V _π (e) =

Question 25

Consider the gridworld where Left and Right actions are always successful. Specifically, the available actions in each state are to move to the neighboring grid squares. From state a, there is also an exit action available, which results in going to the terminal state and collecting a reward of 10. Similarly, in state e, the reward for the exit action is 1. Exit actions are always successful. The discount factor γ = 1. Now, consider the policy π shown below, and evaluate the following quantity for this policy.

V _π (a) =

Question 26

Consider the gridworld where Left and Right actions are always successful. Specifically, the available actions in each state are to move to the neighboring grid squares. From state a, there is also an exit action available, which results in going to the terminal state and collecting a reward of 10. Similarly, in state e, the reward for the exit action is 1. Exit actions are always successful. The discount factor γ = 1. Now, consider the policy π shown below, and evaluate the following quantity for this policy.

V _π (b) =

Question 27

Consider the gridworld where Left and Right actions are always successful. Specifically, the available actions in each state are to move to the neighboring grid squares. From state a, there is also an exit action available, which results in going to the terminal state and collecting a reward of 10. Similarly, in state e, the reward for the exit action is 1. Exit actions are always successful. The discount factor γ = 1. Now, consider the policy π shown below, and evaluate the following quantity for this policy.

V _π (c) =

Question 28

Consider the gridworld where Left and Right actions are always successful. Specifically, the available actions in each state are to move to the neighboring grid squares. From state a, there is also an exit action available, which results in going to the terminal state and collecting a reward of 10. Similarly, in state e, the reward for the exit action is 1. Exit actions are always successful. The discount factor γ = 1. Now, consider the policy π shown below, and evaluate the following quantity for this policy.

V _π (d) =

Question 29

Consider the gridworld where Left and Right actions are always successful. Specifically, the available actions in each state are to move to the neighboring grid squares. From state a, there is also an exit action available, which results in going to the terminal state and collecting a reward of 10. Similarly, in state e, the reward for the exit action is 1. Exit actions are always successful. The discount factor γ = 1. Now, consider the policy π shown below, and evaluate the following quantity for this policy.

V _π (e) =

Posting submission...