Tobias Jung, University of Mainz, Germany Daniel Polani ...tjung/Neuer Ordner/adprl07-slides.pdf ·...

19
λ Tobias Jung, University of Mainz, Germany Daniel Polani, University of Hertfordshire, U.K. Motto: APE/API with kernel-based function approximation to tackle high-dimensional control tasks heretofore impossible to solve with traditional RL approaches (e.g. Sarsa+Tilecoding) Kernelizing LSPE(λ) — ADPRL 2007 – p.1/19

Transcript of Tobias Jung, University of Mainz, Germany Daniel Polani ...tjung/Neuer Ordner/adprl07-slides.pdf ·...

Page 1: Tobias Jung, University of Mainz, Germany Daniel Polani ...tjung/Neuer Ordner/adprl07-slides.pdf · Daniel Polani, University of Hertfordshire, U.K. ... Total reward per episode Kernelizing

�� � � �� �� � � � � �

λ

�Tobias Jung, University of Mainz, Germany

Daniel Polani, University of Hertfordshire, U.K.

Motto: APE/API with kernel-based function approximation to tackle high-dimensional control tasks

heretofore impossible to solve with traditional RL approaches (e.g. Sarsa+Tilecoding)

��� ��� � ��� �� � ��� � � ��� � � � � � � � � ! � �" � � � �$# # #

Kernelizing LSPE(λ) — ADPRL 2007 – p.1/19

Page 2: Tobias Jung, University of Mainz, Germany Daniel Polani ...tjung/Neuer Ordner/adprl07-slides.pdf · Daniel Polani, University of Hertfordshire, U.K. ... Total reward per episode Kernelizing

� � �� � � � � � � � � � � ��

� �

�� � ��� � � � � � � � � � ! � � � � � � �� http://www.cs.mcgill.ca/dprecup/workshops/ICML06/octopus.html�

���� ��� ���� � ! " #$ %$& #' $ (*) & $ % #$+ , -) ' . / ' 0) % 12& 354 #) - 6 7contract transversal muscle

contract longitudal muscle

contract longitudal muscle(dorsal side)

(ventral side)

Actions

arm base

arm tip

pair N+1

N compartments

point masses

pair #1

89�� � ��� "�: �9 ! "9<; = !*> � : !� � � - #) # 0 -+ ) % 0 ⊂ R

(2N+2)×4

/ 0@? 3? ACB D %$ .+ ) ' # . 0& # - =⇒ R72

7

9 ! "9<; = !*> � : !� � �) & EF � G ! � H: � F G !� : I� F � ⊂ RN×3

? ? ? 1 0' 0J E 2 - %' 0 #2LK 0 E 2& # $ M) % #2LN ) #2 $& + ) # # 0'& -

Kernelizing LSPE(λ) — ADPRL 2007 – p.2/19

Page 3: Tobias Jung, University of Mainz, Germany Daniel Polani ...tjung/Neuer Ordner/adprl07-slides.pdf · Daniel Polani, University of Hertfordshire, U.K. ... Total reward per episode Kernelizing

� � � � � � � � �� � � �� � � � � � � � � � ��

� �

���� � � � � � � � � � � � http://www.cs.utexas.edu/users/AustinVilla/sim/keepaway/

���� ��� ���� � 1 $ #$ .) 2 . 2LK 0 # 1 0 #2 . 0 # 1 0 6 0 0+ 0' - %$ & #' $ ( # 1 0 �) ( ( /� N -? 7

89�� � ��� "�: �= !> � : !� � � ! G� $ � # 1 0 - #) # 0 -+ ) % 0 /� � E 2 . 0& -2 $ & - 7

: G� F 9�� : G !F G �� : ! G !� : /& $ 2 -� + 0' % 0+ #2 $ & -) & E) % #2 $ & -�� . , ( #2+ ( 0 � , ( ( � ) , # $ & $ . $ , -) 3 0& # -

& 0 0 E #$ %$ $+ 0') # 0 7

��� �; G !*> � ���� � ! " / , - 0 - � $ � %2) ( � -$ % % 0' - 0' N 0' 7

Kernelizing LSPE(λ) — ADPRL 2007 – p.3/19

Page 4: Tobias Jung, University of Mainz, Germany Daniel Polani ...tjung/Neuer Ordner/adprl07-slides.pdf · Daniel Polani, University of Hertfordshire, U.K. ... Total reward per episode Kernelizing

� � � � � � � � � � �� � � � � �

� H � � !: 9 � !: G�12 3 1 E 2 . 0& -2 $ & -

- 0� , 0& #2) ( � $& (2& 0 ( 0) '& 2& 3

�) - # � 0 � %2 0& # � ' 0 (2) � ( 0

(2& 0) ' ( � + ) ') . 0 # 0' 2LK 0 E N ) ( , 0 � ,& % #2 $ & /

� � � � �� �+ $ (2 % � 0N ) ( ,) #2 $ & 7

� H �� I I ��� F 9 � �� �� �� �') . 0 $ ' 6

� � # 1 ' $ , 3 1 � �� � � � � 2 # 1 6 0'& 0 ( 4 �) - 0 E � ,& % #2 $ & ) + + ' $ 2 .) #2 $ &

� 0 3 , () ' 2LK ) #2 $ & & 0 # $ ' 6 - � 6 0'& 0 ( ' 2 E 3 0 ' 0 3' 0 - -2 $ & �� ) , - -2) & + ' $ % 0 - - ' 0 3' 0 - -2 $& ? ? ?

$ %$ ,& # 0' # 1 0 O(n3) - %) (2& 3 $ � 6 0'& 0 ( 4 �) - 0 E ( 0) '& 2& 3

- , � - 0 # $ � ' 0 3' 0 - -$ ' -) + + ' $ 2 .) #2 $&

$& (2& 0 - 0 ( 0 % #2 $ & $ � ' 0 ( 0N ) & # �) -2 - 0 ( 0 . 0& # -

��: H � G� & 0 � %2 0& # � ' 0 % , ' -2LN 0 $& (2& 0 2 .+ ( 0 . 0& #) #2 $ & 2 # 1) , # $ .) #2 % - ,+ 0' N 2 - 0 E - 0 ( 0 % #2 $& $ � ' 0 ( 0N ) & # �) -2 -

� ,& % #2 $& -) & E+ 0' 4 - # 0+ %$ .+ ( 0 2 #� O(m2)?

Kernelizing LSPE(λ) — ADPRL 2007 – p.4/19

Page 5: Tobias Jung, University of Mainz, Germany Daniel Polani ...tjung/Neuer Ordner/adprl07-slides.pdf · Daniel Polani, University of Hertfordshire, U.K. ... Total reward per episode Kernelizing

� �� � �

�� � !F � � �� � H� G !� � / � $ E 0 ( 4 �' 0 0 � 7

Qπ(s, a) = Es′|s,a

{R(s′|s, a) + γQπ(s′, π(s′))

}

1 0' 0

S ⊂ Rd - #) # 0 -+ ) % 0 � A) % #2 $ & -+ ) % 0 / E 2 - %' 0 # 0 7

π : S → A+ $ (2 % �

P (s′|s, a)

#') & 2 -2 #2 $ & + ' $ �) �2 (2 #2 0 -

R(s′|s, a) ' 0 ) ' E / � 4 - # 0+ ' 0 # , '& 7

γ ∈ (0, 1)

E2 - %$ ,& # �) % #$ '

Kernelizing LSPE(λ) — ADPRL 2007 – p.5/19

Page 6: Tobias Jung, University of Mainz, Germany Daniel Polani ...tjung/Neuer Ordner/adprl07-slides.pdf · Daniel Polani, University of Hertfordshire, U.K. ... Total reward per episode Kernelizing

� � � � �� � � � � � � �� �� �� � � � � �� � �

��� � H F G !� � I I ��� !> � G !� � ' 0+ ' 0 - 0& #

Qπ # 1 ' $ , 3 1 (2& 0) ' ( � + ) ') . 0 # 0' 2LK 0 E) + + ' $ 2 .) #2 $ &

Q(x;w) =

m∑

i=1

wiϕi(x)

1 0' 0

x := (s, a) ∈ Rd ×A - #) # 0& ) % #2 $& # ,+ ( 0 -

ϕi : Rd ×A → R

�) -2 - � ,& % #2 $ & - /

i = 1, . . . , m

7w =

(w1, . . . , wm

)T 02 3 1 # - /& 0 0 E #$ � 0 E 0 # 0' . 2& 0 E 7

�� � > I ��� : � ) + + ' $ 2 .) # 0 P (· |s, a), R(· |s, a)�' $ . -) .+ ( 0 # ') & -2 #2 $ & - / -2 . , () #2 $ & 7

strt+1

πk(st)// st+1

rt+2

πk(st+1)// st+2 // · · ·

�� � " � ��� � � � � � � � " � � � � � � � � � � � � � � � � ��� � � � � � � � � �

Qπ # # #

Kernelizing LSPE(λ) — ADPRL 2007 – p.6/19

Page 7: Tobias Jung, University of Mainz, Germany Daniel Polani ...tjung/Neuer Ordner/adprl07-slides.pdf · Daniel Polani, University of Hertfordshire, U.K. ... Total reward per episode Kernelizing

� � � � �� � � � � � � �� �� � � �� � �� � �

Temporal Difference Learning

Observed transitions

(e.g. LSTD, LSPE)

Approximate

Policy Evaluation

Approximate

Policy Improvement

...

Value function

Policy (greedy) wrt

πk

πk+1

wk

Q(· ;wk) ≈ Qπk

Q(· ;wk)

{si, ai, ri+1, si+1}

� G G9 � � = � 0N 0' � # 1 2& 3 � $ 2 ( - E $ & # $ # 1 0 � , 0 - #2 $ & -

� ? � � 1) # . 0 # 1 $ E E $ 0 % 1 $ $ - 0 # 1 0+ ) ') . 0 # ' 2LK ) #2 $ & $ �

Q) & E %) ' ' � $ , # ' 0 3' 0 - -2 $& �

? � � 1) # . 0 # 1 $ E E $ 0 ( 0) '& # 1 0 02 3 1 # N 0 % #$ '

w� 32LN 0& -) .+ ( 0 #') & -2 #2 $ & - �

Kernelizing LSPE(λ) — ADPRL 2007 – p.7/19

Page 8: Tobias Jung, University of Mainz, Germany Daniel Polani ...tjung/Neuer Ordner/adprl07-slides.pdf · Daniel Polani, University of Hertfordshire, U.K. ... Total reward per episode Kernelizing

� �� � � �� � �

� ? �9 � �� � � �; �� : � = � H F G !� � I I ��� !> � G !� �

N $ 2 E - % , ' - 0 $ � E2 . 0& -2 $ & ) (2 #� �� ' 0+ ' 0 - 0& #2& 3 -$ ( , #2 $ & # 1 ' $ , 3 1 # 1 0 E) #)

� (2 . 2& ) # 0 - 0 + (2 %2 #& $ E 0 � �) -2 - � ,& % #2 $& - 0 ( 0 % #2 $& /) # ( 0) - # 2& # 1 0$ ' � 7

� & - # 0) E � � , - # % 1 $ $ - 0 6 0'& 0 ( � %$ N ) ' 2) & % 0 � ,& % #2 $& ? � ? 3? # 1 0 � ) , - -2) &k(x,y) = exp{h−1 ‖x − y‖2}

? �9 � ���� : G; : � H� ��: �� : � = I� � !F � � �� � H� G !� �

� �� � /

λ

7 � � � /

λ

7 - 1) ' 0 - %$ & N 0' 3 0& % 0 2 # 1 /λ

7

� , # %$& N 0' 3 0 - %$ & -2 E 0') � ( � �) - # 0' / �) ' � 0 0' -) .+ ( 0 -& 0 % 0 - -) ' � 7

� ( $ - 0 E -$ ( , #2 $& N -@? - # $ % 1) - #2 % 3') E 2 0& # 4 E 0 - % 0& #

� ? � � �: � � �

� �� � 2 - 2& %' 0 . 0& #) () & E %) & $ ' 6 2 # 1 $+ #2 . 2 - #2 %+ $ (2 % � 2 # 0') #2 $ &

� � 2 - �) # % 1

⇒& $ � ��

/� 0' 0 � 0 2 ( ( %$ & -2 E 0' � �� � � � , # $ , ' ) ( 3$ ' 2 # 1 .) ( -$ # ') & - () # 0 - #$ � � 7

� � � � � � � � � �� � ! � # # #Kernelizing LSPE(λ) — ADPRL 2007 – p.8/19

Page 9: Tobias Jung, University of Mainz, Germany Daniel Polani ...tjung/Neuer Ordner/adprl07-slides.pdf · Daniel Polani, University of Hertfordshire, U.K. ... Total reward per episode Kernelizing

� � �

λ

� � � � �� � � � � � � �� � �� � � � � � � � � �

� � � $ � #) 2& Q(x;w∗) =∑m

i=1 w∗i ϕi(x)) -) & ) + + ' $ 2 .) #2 $& �$ ' Qπ(x)

� ! G !� � !�� �

w0 = 0� % 1 $ $ - 0 ( 0) '& 2& 3 ') # 0 η ∈ [0, 1]� 3 0& 0') # 0

x0 = (s0, π(s0))

�� �

t = 0, 1, . . . $ � - 0' N 0 # ') & -2 #2 $& # $

xt+1, r∗

/ -2 . , () #2& 3 π 7

wt+1 := argminw

‖Φt+1w − Φt+1wt − Λt+1(rt+1 − Φt+1wt)‖2

� � � � � � � � " � � � � � � � � � � � � � � �� �� " � � � � � � � �� � �

wt+1 := wt + η(wt+1 − wt)8� �� � "� F � �

wt → w∗ # 1 0 -) . 0) - � � /

λ

7 � /λ

7

� � G� G !� �

0) # , ' 0N 0 % # $ ' J

ϕt+1 := ϕ(xt+1) =(ϕ1(xt+1), . . . , ϕm(xt+1)

)T

� 0 % , ' -2 $ & J / 0N 0' � & 0 $ � - 0' N ) #2 $& ) E E -) & 0 ' $ 7

Φt+1 =

Φt

ϕTt

Φt+1 =

Φt

ϕTt − γϕ

Tt+1

rt+1 =

rt

r∗

Λt+1 = . . .

� � � � � ! � � � � ��� � � �� �λ

� �� � � � �� � � � � � � � � �" � � � ��� � �

Kernelizing LSPE(λ) — ADPRL 2007 – p.9/19

Page 10: Tobias Jung, University of Mainz, Germany Daniel Polani ...tjung/Neuer Ordner/adprl07-slides.pdf · Daniel Polani, University of Hertfordshire, U.K. ... Total reward per episode Kernelizing

�� � � �� � � �� � � � � � � � � �

� #) ' #2& 3 �' $ .+ ' 2 .) ( �$ ' . / E ' $+ + 2& 3 2& E 0 t + 1

�' $ . .) # ' 2 % 0 - 7

wt+1 := argminw

‖Φw − Φwt − Λ(r − Φwt)‖2 + σ2‖w − wt‖

2

︸ ︷︷ ︸

weight regularizer as in RR

� $ .+ , #2& 3 E 0' 2LN ) #2LN 0 ' #

w

) & E 0� ,) #2& 3 2 # 1 K 0' $

=⇒ wt+1 = wt + (ΦTΦ + σ2I)−1ΦTΛ(r − Φwt)

+ + ( � � 1 0' .) & 4 � $ ' ' 2 -$ & 4 �$ $ E � , ' �

(ΦTΦ + σ2I)−1ΦT = ΦT(ΦΦT + σ2I)−1

#$ $ � #) 2&

wt+1 = wt + ΦT(ΦΦT + σ2I)−1Λ(r − Φwt)

=⇒ ∀t) ( ( -$ ( , #2 $& -

wt

(2 0 2& %$ ( , .& -+ ) % 0 $ �ΦT � 2 ? 0@? . � � 0 0 + ' 0 - - 0 E) -

wt = ΦTα =

t∑

i=1

ϕ(xi)αi

�$ ' -$ . 0 E ,) ( N ) ' 2) � ( 0 -α =

(α1, . . . , αt

)T

� � � � � � � � �� � � � � � � � � � � � � � " � �� � � � � � � ��� � � � �� � � � � � �

w

� � ��

ΦTα

� � # # #

Kernelizing LSPE(λ) — ADPRL 2007 – p.10/19

Page 11: Tobias Jung, University of Mainz, Germany Daniel Polani ...tjung/Neuer Ordner/adprl07-slides.pdf · Daniel Polani, University of Hertfordshire, U.K. ... Total reward per episode Kernelizing

�� � � �� � � �� � � � � � � � �

� 0+ () %2& 3 2 # 0') # 0 -

wt+1,wt+1

�� E ,) ( N ) ' 2) � ( 0 -

αt+1 = αt + (ΦΦT + σ2I)−1Λ(r − ΦΦTαt)

� � � � � G � !F �� 6 0'& 0 ( %$ .+ , # 0 - 2+ 2& � 0) # , ' 0 -+ ) % 0J k(x,y) = 〈ϕ(x), ϕ(y)〉

0 �& 0J

K := ΦΦT � [K]ij = k(xi,yj)

H := ΦΦT � [H]ij = k(xi,xj) − γk(xi+1,xj)

k(·) :=(k(x1, ·, ), . . . , k(xt, ·)

)T

1 0' 0 �$ ' 0

αt+1 = αt + (K + σ2I)−1Λ(r − Hαt)

� 0& % 0 � # 1 0 � �� � ,+ E) # 0 2 - /2& E ,) ( N ) ' 2) � ( 0 - 7

αt+1 = αt + η(K + σ2I)−1Λ(r − Hαt)

� �� � � � � � � � � � � � " � � ��� � � � �� �λ

� � � � � � � � �

Kernelizing LSPE(λ) — ADPRL 2007 – p.11/19

Page 12: Tobias Jung, University of Mainz, Germany Daniel Polani ...tjung/Neuer Ordner/adprl07-slides.pdf · Daniel Polani, University of Hertfordshire, U.K. ... Total reward per episode Kernelizing

�� � � �� �� � � � � �

� ' 0 E 2 % #2 $ & - 2 # 1

Q

2& ) ' �2 # ') ' � x∗) ( -$ + $ - -2 � ( 0 �' $ . E ,) ( N ) ' 2) � ( 0 -J

Q(x∗;wt) = 〈ϕ(x∗),wt〉 = 〈ϕ(x∗),t∑

i=0

α(i)t ϕ(xi)〉 =

t∑

i=0

α(i)t k(xi,x

∗) = Q(x∗; αt)

� 0& % 0J %$ .+ ( 0 # 0 � �� �) ( 3$ ' 2 # 1 . .) � � 0 - #) # 0 E 2& E ,) ( N ) ' 2) � ( 0 -?

� = �� G� "� : $ � 6 0'& 0 ( 4 �) - 0 E) + + ' $ ) % 1

0 (2 .2& ) # 0 - 0 + (2 %2 # % 1 $ 2 % 0 $ �& $ E 0 - � �) -2 - � ,& % #2 $ & - ϕi(·)

/) # � ' - # 3 () & % 0 7

� , - # % 1 $ $ - 0) 6 0'& 0 ( � 0@? 3? # 1 0 � ) , - -2) & k(x,y) = exp−h−1‖x − y‖2

� �� � ��� > � � �� � 2 -) & 2& %' 0 . 0& #) () & E $& (2& 0) ( 3$ ' 2 # 1 . # 1) # %$& #2& ,) ( ( � ,+ E) # 0 - # 1 0 -$ ( , #2 $ & ? $ ' 0) % 1

#2 . 0 4 - # 0+ t 0& 0 0 E #$ #) % 6 ( 0) # 4 �� 4 # + ' $ � ( 0 . ? ? ?

Kernelizing LSPE(λ) — ADPRL 2007 – p.12/19

Page 13: Tobias Jung, University of Mainz, Germany Daniel Polani ...tjung/Neuer Ordner/adprl07-slides.pdf · Daniel Polani, University of Hertfordshire, U.K. ... Total reward per episode Kernelizing

� � � � � � � � � � �� � � � �� � � � � � � �� � � �� � �

� � � �� � � � �� � � � � � �� � �� � � � � � � � � � � �� � �# # # �

� = �� �

89 � � : � � ) - , � - 0 # $ � # 1 0 E) #) {x}mi=1

� 1 0' 0 m � t

� I I ��� !*> � G� � # 1 0 6 0'& 0 ( �� /) ( -$ ) ' 2 - 0 - �' $ . # 1 0 A � - # ' � .) + + ' $ 2 .) #2 $& 7

k(x,x′) = km(x)TK−1mmkm(x′) ∀x,x′

1 0' 0

km(·) =(k(x1, ·), . . . , k(xm, ·)

)T) & E

[Kmm]ij = k(xi, xj)?

�� = HF � = I �� � � � > ��� I �� F � � 0N 0' � $ % % , ') & % 0 $ �

k(x,x′)

�� km(x)TK−1mmkm(x′)J

Q(· ; α) =∑m

i=1 α(i)k(xi, · )

αt+1,m = αtm + ηP−1t+1,m(ZT

t+1,mrt+1 − ZTt+1,mHt+1,mαtm)

1 0' 0

Pt+1,m = (KTt+1,mKt+1,m + σ2Kmm)� [Kt+1,m]ij = k(xi, xj)�

[Ht+1,m]ij = k(xi, xj) − γk(xi+1, xj)� ZTt+1,m := KT

t+1,mΛt+1

� � G9 � : H �: � G

{x}mi=1

�� �� : � > � 9 � � � � � ! � = �� F � �

� 0 E , % 0 - #$ ) ��� � �) -2 - � ,& % #2 $ & & 0 # $ ' 6

� 0 % , ' -2LN 0 2 .+ ( 0 . 0& #) #2 $& + $ - -2 � ( 0) #

O(m2)+ 0' 4 - # 0+

" � � � � �� � � � �� � � � � �" � � � �� � �� � � � � � �� � � � � ��� � �� " � � � � � � ��� # # #

Kernelizing LSPE(λ) — ADPRL 2007 – p.13/19

Page 14: Tobias Jung, University of Mainz, Germany Daniel Polani ...tjung/Neuer Ordner/adprl07-slides.pdf · Daniel Polani, University of Hertfordshire, U.K. ... Total reward per episode Kernelizing

� � � � � � �� � � � � � � �� � � � � � � � �� � � �� � �

� � � �� � � � �� � � �� � � � � �� � � � � �� � � � � � � � �# �� � � � � �

� � ! � : � ��� F G !� � ) - - , . 0 # ') 2& 2& 3 E) #) � 0 %$ . 0 -) N ) 2 () � ( 0: � � H � G !� � � � ) #

t = 1, 2, . . .

� #) ' # 2 # 1) & 0 .+ #� - , � - 0 # / � E 2 % #2 $ & ) ' � � $ � �) -2 - � ,& % #2 $ & - 7

# #2 . 0 t # ' � # $ ) + + ' $ 2 .) # 0 # 1 0& 0 2& + , # E) #)

xt

�' $ . # 1 0 % , ' ' 0& # E2 % #2 $& ) ' � J

Feature spaceϕt+1

∥∥ϕ

⊥t+1

∥∥ =

√δt+1

ϕt+1

span{ϕ1, . . . , ϕm}

� ' 2 # 0' 2 $ & J 2 �

δt+1 = k(xt,xt) − km(xt)TK−1mmkm(xt) > TOL

# 1 0&

xt

2 -) E E 0 E #$ - , � - 0 #

� N 0') ( ( %$ - # -J O(m2)� 1 0' 0 m 2 - # 1 0 % , ' ' 0& # -2LK 0 $ � - , � - 0 #

� � � � � � � � � � � � �� � � � � � � � � � �

Kernelizing LSPE(λ) — ADPRL 2007 – p.14/19

Page 15: Tobias Jung, University of Mainz, Germany Daniel Polani ...tjung/Neuer Ordner/adprl07-slides.pdf · Daniel Polani, University of Hertfordshire, U.K. ... Total reward per episode Kernelizing

� � � � � � � �� � � � � � � � �� � � �� � �

� � G� � � � ! � . 0) & - # 1) # 0N 0' � #2 . 0 - # 0+ $ & ( � # 1 0F H � �� G 0 ( 0 . 0& # - 2& # 1 0 E2 % #2 $ & ) ' � ) ' 0 , - 0 E ? , # , ' 0

0 ( 0 . 0& # - E $ & $ # ' 0 # ' $ ) % #2LN 0 ( � %$ & # ' 2 � , # 0 # $ # 1 0) + + ' $ 2 .) #2 $& - 2& # 1 0+ ) - # �

�� F G� 1 0& 0N 0' 0 $ , ( E& 0 0 E #$ %$& -2 E 0' ) ( (+ ) - # 0 ) .+ ( 0 -�� 0@? 3? 2& ) t × m . $ E 0 (

1 0& ) E E2& 3) & 0 �) -2 - � ,& % #2 $ &

1 0& %$ .+ , #2& 3 # 1 0 - %$ ' 0 $ � �) -2 - � ,& % #2 $& %) & E2 E) # 0 - 2& 3' 0 0 E � �$ ' ) ' E - 0 ( 0 % #2 $&

1 0& %$ .+ , #2& 3+ ' 0 E2 % #2LN 0 N ) ' 2) & % 0 2& � � 2 # 1 �) , 3 . 0& #) #2 $& � � � � �� " � � � � � �� # � � � � � �

0 %) & 0 + ( $ 2 # # 1) #) E E2& 3) & 0 �) -2 - � ,& % #2 $& % 0& # 0' 0 E $&xt+1 � # 1) # 2 -

Kt+1,m+1 =[

Kt+1,m q

]

Ktm Ktmat+1

km(xt+1)T k(xt+1,xt+1)

=: Kt+1,m+1,

1 0' 0

at+1 = K−1mmkm(xt+1)� $& ( � %$ - # - O(m2) $+ 0') #2 $& - 2& - # 0) E $ � # 1 0 , - ,) (

O(tm)?

� �� � � � � �� � �� � � � � � � � � �� � � � � � � � � � � �� � � �� � � � � � �� � # # #

Kernelizing LSPE(λ) — ADPRL 2007 – p.15/19

Page 16: Tobias Jung, University of Mainz, Germany Daniel Polani ...tjung/Neuer Ordner/adprl07-slides.pdf · Daniel Polani, University of Hertfordshire, U.K. ... Total reward per episode Kernelizing

�� � � � � � � �

λ

� � � � �� � � � � �� � � � � � � �� � �

� I H G� + $ (2 % � π

� H G I H G� E2 % #2 $ & ) ' � D $ � �) -2 - � ,& % #2 $ & -

� 4 � ,& % #2 $ & J Q(·) =∑|D|

i=1 αik(xi, · )

� & 2 #2) (2LK 0

� �

t = 1, 2, . . .

� 0 % , # 0) % #2 $ & at

� � - 0' N 0& 0 # - #) # 0 st+1

) & E ' 0 ) ' E

rt+1

� 1 $ $ - 0) % #2 $ & at+1 = π(st+1)? $ ' .

xt+1 := (st+1, at+1)?

� ∥∥ϕ

⊥t+1

∥∥2

> TOL O(m2)

� � �+ E) # 0 02 3 1 # N 0 % # $ ' ) & E*) , 3 . 0& # �) -2 -

αt+1,m+1 =

αtm

0

+ · · · O(m2)

D = D ∪ {xt+1}� m = m + 1

� � � �� � �+ E) # 0 02 3 1 # N 0 % # $ ' 2 # 1 $ , #) , 3 . 0& #2& 3 �) -2 -

αt+1,m = αtm + · · · O(m2)

st = st+1, at = at+1

Kernelizing LSPE(λ) — ADPRL 2007 – p.16/19

Page 17: Tobias Jung, University of Mainz, Germany Daniel Polani ...tjung/Neuer Ordner/adprl07-slides.pdf · Daniel Polani, University of Hertfordshire, U.K. ... Total reward per episode Kernelizing

�� �� � � � � � � � � � � �� � �

�: G � � = � D 4 %$ .+ ) ' # . 0& # 4 $ % #$+ , -) ' . 2 # 1 � � ) ' E ) - 6 � �' $ .� � � � 4 �� � 0& % 1 .) ' 6

��: H � G� � �� 2 # 1 � 0'& 0 ( 4 � � � �

0 20 40 60 80 100 120 140 160 180 200−1000

−800

−600

−400

−200

0

200

400

600

800

1000

Episodes

Tot

al r

ewar

d pe

r ep

isod

e

Kernelizing LSPE(λ) — ADPRL 2007 – p.17/19

Page 18: Tobias Jung, University of Mainz, Germany Daniel Polani ...tjung/Neuer Ordner/adprl07-slides.pdf · Daniel Polani, University of Hertfordshire, U.K. ... Total reward per episode Kernelizing

�� �� � � � � � � � � � � � � � � � � �� � � �� � � � �

8� > I� �� � � , ' � 0'& 0 ( 4 � � 2 # 1 �� /& $ � �� 7 N -? # 1 0 # 0 # � $ $ 6) + + ' $ ) % 1 -) ' -) /

λ

7 � #2 ( 0 %$ E2& 3

0 5 10 15 20 25 30 35 404

6

8

10

12

14

16

18

20

223vs2 keepaway (field size 20m x 20m)

Training time (hours)

Epi

sode

dur

atio

n (s

ecs)

Our approach

Stone, Sutton & Kuhlman (2005)

Random behavior

Optimized handcoded behavior

Kernelizing LSPE(λ) — ADPRL 2007 – p.18/19

Page 19: Tobias Jung, University of Mainz, Germany Daniel Polani ...tjung/Neuer Ordner/adprl07-slides.pdf · Daniel Polani, University of Hertfordshire, U.K. ... Total reward per episode Kernelizing

� � � � � � �

� � �� =� �� H G� � I !F � � & (2& 0 � � 2 # 1 � �� �) & E ' 0 3 , () ' 2LK ) #2 $ & & 0 # $ ' 6 -

� � G9 � =: �� , � - 0 # $ � ' 0 3' 0 - -$ ' -) + + ' $ 2 .) #2 $&

� & (2& 0 3' 0 0 E � - 0 ( 0 % #2 $ & $ � ' 0 ( 0N ) & # �) -2 - � ,& % #2 $ & - / - ,+ 0' N 2 - 0 E 7

� � %2 0& # ' 0 % , ' -2LN 0 2 .+ ( 0 . 0& #) #2 $ & J O(m2)+ 0' - # 0+ /2& E 0+ 0& E 0& # $ � # 1 0 #$ #) (& , . � 0'

$ � E) #) 7

��: H � G: �� �$ ' # ( 0 - - ( � - %) ( 0 - #$ 1 2 3 1 4 E2 . 0& -2 $ & ) ( %$ & # ' $ ( #) - 6 -

� $ 0N 0' � , ( #2 .) # 0 ( � (2 .2 # 0 E �� � �<4 -2LK 0 $ � E2 % #2 $& ) ' � /

m ∼ 2000� E 0+ 0& E2& 3 $ & �� � 7

� I I � !F � G !� : �

� � � � �� � � � # # 0 E N ) ( , 0 2 # 0') #2 $ & /& $ # # ' 2 0 E 7 � ? ? ?

� & �) % # � 0N 0' � # 1 2& 3 # 1) # 2 -) ( 0) - # 4 -� ,) ' 0 - #�+ 0 $ �+ ' $ � ( 0 .) & E $ , ( E � 0& 0 � # �' $ .

- 0� , 0& #2) ( ( 0) '& 2& 3 / 0@? 3? #2 . 0 4 - 0' 2 0 -+ ' 0 E2 % #2 $& ? ? ? 7

Kernelizing LSPE(λ) — ADPRL 2007 – p.19/19