原论文demo - metamotivo , GitHub - metamotivo
这种无奖励强化学习本质上是通过采用的方式，将奖励 r ( s ) r(s) r ( s ) B ( s ) B(s) B ( s ) R d \mathbb{R}^d R d z ∈ R d z\in\mathbb{R}^d z ∈ R d π \pi π Π \Pi Π Q π ( s , a ) Q^{\pi}(s,a) Q π ( s , a ) F ( s , a ∣ π ) ∈ R d F(s,a|\pi)\in\mathbb{R}^d F ( s , a ∣ π ) ∈ R d Q π ( s , a ) = ⟨ F ( s , a ∣ π ) , B ( s ) ⟩ = F ( s , a ∣ π ) T B ( s ) Q^{\pi}(s,a) = \lang F(s,a|\pi), B(s)\rang = F(s,a|\pi)^TB(s) Q π ( s , a ) = ⟨ F ( s , a ∣ π ) , B ( s ) ⟩ = F ( s , a ∣ π ) T B ( s )
这里将低维空间 z z z d \sqrt{d} d
首先进行符号定义，设 S , A S,A S , A d ∈ Z + d\in \mathbb{Z}^+ d ∈ Z + Pr t ( s ′ ∣ s , a , π ) \text{Pr}_{t}(s'|s,a,\pi) Pr t ( s ′ ∣ s , a , π ) s , a s,a s , a s s s a a a π \pi π t t t s ′ s' s ′ E [ r t ∣ s , a , π ] \mathbb{E}[r_t|s,a,\pi] E [ r t ∣ s , a , π ] s s s a a a π \pi π t t t r ( s t ) r(s_t) r ( s t )
上述思路可以用完整的理论进行描述，定义
Forward Function: F ( s , a , z ) : S × A × R d → R d F(s,a,z): S\times A\times \mathbb{R}^d\to \mathbb{R}^d F ( s , a , z ) : S × A × R d → R d F F F Π \Pi Π R d \mathbb{R}^d R d π \pi π Π \Pi Π R d \mathbb{R}^d R d π \pi π z z z Π \Pi Π
Backward Function: B ( s ) : S → R d B(s): S\to \mathbb{R}^d B ( s ) : S → R d
Policy Function: π ( s , z ) : S × R d → A \pi(s,z): S\times \mathbb{R}^d\to A π ( s , z ) : S × R d → A
那么 F , B , π F,B,\pi F , B , π 后继度量 （Successor measures）与奖励无关。
∀ X ⊂ S , s ∈ S , a ∈ A , π ∈ Π \forall X\subset S, s\in S, a\in A, \pi\in \Pi ∀ X ⊂ S , s ∈ S , a ∈ A , π ∈ Π M π : S × S × A → R M^{\pi}: S\times S\times A\to \mathbb{R} M π : S × S × A → R
M π ( X ∣ s , a ) : = ∑ t = 0 ∞ γ t Pr t + 1 ( s ′ ∈ X ∣ s , a , π ) M^{\pi}(X|s,a):=\sum_{t=0}^{\infty}\gamma^t\text{Pr}_{t+1}(s'\in X|s,a,\pi)
M π ( X ∣ s , a ) : = t = 0 ∑ ∞ γ t Pr t + 1 ( s ′ ∈ X ∣ s , a , π )
其中 γ ∈ R \gamma\in\mathbb{R} γ ∈ R
后继度量可以表示在策略 π \pi π s , a s,a s , a X X X Blog - PPO 推论1 策略回报参数形式 中的 ρ π ~ ( S ) \rho_{\tilde{\pi}}(S) ρ π ~ ( S )
不难发现，后继度量的Bellman方程为
M π ( X ∣ s , a ) = P ( X ∣ s , a ) + γ E s ′ ∼ p ( ⋅ ∣ s , a ) a ′ ∼ π ( ⋅ ∣ s ′ ) [ M π ( X ∣ s ′ , a ′ ) ] (1) \tag{1}M^{\pi}(X|s,a) = P(X|s,a) + \gamma\mathbb{E}_{\substack{s'\sim p(\cdot|s,a)\\a'\sim\pi(\cdot|s')}}[M^{\pi}(X|s',a')]
M π ( X ∣ s , a ) = P ( X ∣ s , a ) + γ E s ′ ∼ p ( ⋅ ∣ s , a ) a ′ ∼ π ( ⋅ ∣ s ′ ) [ M π ( X ∣ s ′ , a ′ ) ] ( 1 )
设动作价值函数 Q π ( s , a ) : = ∑ t = 0 ∞ γ t E [ r t + 1 ∣ s , a , π ] Q^{\pi}(s,a) := \sum_{t=0}^{\infty}\gamma^t\mathbb{E}[r_{t+1}|s,a,\pi] Q π ( s , a ) : = ∑ t = 0 ∞ γ t E [ r t + 1 ∣ s , a , π ]
Q π ( s , a ) = ∫ s ′ ∈ S M π ( s ′ ∣ s , a ) r ( s ′ ) d s ′ Q^{\pi}(s,a) = \int_{s'\in S}M_{\pi}(s'|s,a)r(s')\mathrm{d}s'
Q π ( s , a ) = ∫ s ′ ∈ S M π ( s ′ ∣ s , a ) r ( s ′ ) d s ′
证明 ：
Q π ( s , a ) = ∑ t = 0 ∞ γ t E [ r t + 1 ∣ s , a , π ] = ∑ t = 0 ∞ γ t E s t + 1 [ r ( s t + 1 ) ∣ s , a , π ] = ∑ t = 0 ∞ γ t ∫ s ′ ∈ S Pr t + 1 ( s ′ ∣ s , a , π ) r ( s ′ ) d s ′ = ∫ s ′ ∈ S r ( s ′ ) ∑ t = 0 ∞ γ t Pr t + 1 ( s ′ ∣ s , a , π ) d s ′ = ∫ s ′ ∈ S M π ( s ′ ∣ s , a ) r ( s ′ ) d s ′ \begin{aligned}
Q^{\pi}(s,a) =&\ \sum_{t=0}^{\infty}\gamma^t\mathbb{E}[r_{t+1}|s,a,\pi] = \sum_{t=0}^{\infty}\gamma^t\mathbb{E}_{s_{t+1}}[r(s_{t+1})|s,a,\pi] \\
=&\ \sum_{t=0}^{\infty}\gamma^t\int_{s'\in S}\text{Pr}_{t+1}(s'|s,a,\pi)r(s')\mathrm{d}s'\\
=&\ \int_{s'\in S}r(s')\sum_{t=0}^{\infty}\gamma^t\text{Pr}_{t+1}(s'|s,a,\pi)\mathrm{d}s'\\
=&\ \int_{s'\in S}M_{\pi}(s'|s,a)r(s')\mathrm{d}s'\\
\end{aligned}
Q π ( s , a ) = = = = t = 0 ∑ ∞ γ t E [ r t + 1 ∣ s , a , π ] = t = 0 ∑ ∞ γ t E s t + 1 [ r ( s t + 1 ) ∣ s , a , π ] t = 0 ∑ ∞ γ t ∫ s ′ ∈ S Pr t + 1 ( s ′ ∣ s , a , π ) r ( s ′ ) d s ′ ∫ s ′ ∈ S r ( s ′ ) t = 0 ∑ ∞ γ t Pr t + 1 ( s ′ ∣ s , a , π ) d s ′ ∫ s ′ ∈ S M π ( s ′ ∣ s , a ) r ( s ′ ) d s ′
这里假如我们将 M π z M_{\pi_z} M π z F ( s , a , z ) T B ( s ′ ) F(s,a,z)^T B(s') F ( s , a , z ) T B ( s ′ ) B ( s ′ ) r ( s ′ ) B(s')r(s') B ( s ′ ) r ( s ′ ) r r r z z z F ( ⋅ , ⋅ , z ) F(\cdot,\cdot,z) F ( ⋅ , ⋅ , z ) z z z π ( ⋅ , z ) \pi(\cdot,z) π ( ⋅ , z )
∀ z ∈ R d \forall z\in\mathbb{R}^d ∀ z ∈ R d F ( s , a , z ) : S × A × R d → R d , B ( s ) : S → R d , π z ( s ) = π ( s , z ) : S × R d → A F(s,a,z): S\times A\times \mathbb{R}^d\to \mathbb{R}^d, B(s): S\to \mathbb{R}^d, \pi_z(s)=\pi(s,z): S\times \mathbb{R}^d\to A F ( s , a , z ) : S × A × R d → R d , B ( s ) : S → R d , π z ( s ) = π ( s , z ) : S × R d → A ρ : S → R \rho: S\to \mathbb{R} ρ : S → R
M π z ( X ∣ s , a ) = ∫ s ′ ∈ X F ( s , a , z ) T B ( s ′ ) ρ ( s ′ ) d s ′ = F ( s , a , z ) T E s ′ ∼ ρ , s ′ ∈ X [ B ( s ′ ) ] M_{\pi_z}(X|s,a) = \int_{s'\in X}F(s,a,z)^TB(s')\rho(s')\mathrm{d}s' = F(s,a,z)^T\mathbb{E}_{s'\sim\rho,s'\in X}[B(s')]
M π z ( X ∣ s , a ) = ∫ s ′ ∈ X F ( s , a , z ) T B ( s ′ ) ρ ( s ′ ) d s ′ = F ( s , a , z ) T E s ′ ∼ ρ , s ′ ∈ X [ B ( s ′ ) ]
则当 z = E s ∼ ρ [ B ( s ) r ( s ) ] z=\mathbb{E}_{s\sim\rho}[B(s)r(s)] z = E s ∼ ρ [ B ( s ) r ( s ) ] Q π z ( s , a ) = F ( s , a , z ) T z Q^{\pi_z}(s,a)=F(s,a,z)^Tz Q π z ( s , a ) = F ( s , a , z ) T z 证明 ：
Q π z ( s , a ) = ∫ s ′ ∈ S M π z ( s ′ ∣ s , a ) r ( s ′ ) d s ′ = ∫ s ′ ∈ S F ( s , a , z ) T B ( s ′ ) ρ ( s ′ ) r ( s ′ ) d s ′ = F ( s , a , z ) T E s ∼ ρ [ B ( s ) r ( s ) ] = F ( s , a , z ) T z \begin{aligned}
Q^{\pi_z}(s,a) =&\ \int_{s'\in S}M_{\pi_z}(s'|s,a)r(s')\mathrm{d}s'\\
=&\ \int_{s'\in S}F(s,a,z)^TB(s')\rho(s')r(s')\mathrm{d}s'\\
=&\ F(s,a,z)^T\mathbb{E}_{s\sim \rho}[B(s)r(s)] = F(s,a,z)^Tz
\end{aligned}
Q π z ( s , a ) = = = ∫ s ′ ∈ S M π z ( s ′ ∣ s , a ) r ( s ′ ) d s ′ ∫ s ′ ∈ S F ( s , a , z ) T B ( s ′ ) ρ ( s ′ ) r ( s ′ ) d s ′ F ( s , a , z ) T E s ∼ ρ [ B ( s ) r ( s ) ] = F ( s , a , z ) T z
这样我们就把奖励函数 r r r R d \mathbb{R}^d R d r r r
假设存在 F ( s , a , z ) : S × A × R d → R d , B ( s ) : S → R d , π z ( s ) : S × R d → A F(s,a,z): S\times A\times \mathbb{R}^d\to \mathbb{R}^d, B(s): S\to \mathbb{R}^d, \pi_z(s): S\times \mathbb{R}^d\to A F ( s , a , z ) : S × A × R d → R d , B ( s ) : S → R d , π z ( s ) : S × R d → A ∀ z ∈ R d , s , s ′ ∈ S , a ∈ A , X ⊂ S \forall z\in\mathbb{R}^d, s,s'\in S, a\in A, X\subset S ∀ z ∈ R d , s , s ′ ∈ S , a ∈ A , X ⊂ S
M π z ( X ∣ s , a ) = F ( s , a , z ) T E s ′ ∼ ρ , s ′ ∈ X [ B ( s ′ ) ] (2) \tag{2}M_{\pi_z}(X|s,a)=F(s,a,z)^T\mathbb{E}_{s'\sim\rho, s'\in X}[B(s')]
M π z ( X ∣ s , a ) = F ( s , a , z ) T E s ′ ∼ ρ , s ′ ∈ X [ B ( s ′ ) ] ( 2 )
π z ( s ) = arg max a ∈ A F ( s , a , z ) T z (3) \tag{3}\pi_z(s)=\argmax_{a\in A}F(s,a,z)^Tz
π z ( s ) = a ∈ A a r g m a x F ( s , a , z ) T z ( 3 )
另一种写法
{ M π z ( s ′ ∣ s , a ) = F ( s , a , z ) T B ( s ′ ) ρ ( s ′ ) π z ( s ) = arg max a ∈ A F ( s , a , z ) T z \begin{cases}
M_{\pi_z}(s'|s,a)=F(s,a,z)^TB(s')\rho(s')\\
\pi_z(s)=\argmax_{a\in A}F(s,a,z)^Tz
\end{cases}
{ M π z ( s ′ ∣ s , a ) = F ( s , a , z ) T B ( s ′ ) ρ ( s ′ ) π z ( s ) = a r g m a x a ∈ A F ( s , a , z ) T z
对任意奖励函数 r : A → R r: A\to \mathbb{R} r : A → R z = E s ∼ ρ [ B ( s ) r ( s ) ] z=\mathbb{E}_{s\sim\rho}[B(s)r(s)] z = E s ∼ ρ [ B ( s ) r ( s ) ] π z \pi_z π z MDP = { S , A , r , P , μ } \text{MDP}=\{S,A,r,P,\mu\} MDP = { S , A , r , P , μ } 证明： 由命题2可得 π z ( s ) = arg max a ∈ A Q π z ( s , a ) \pi_z(s)=\argmax_{a\in A}Q^{\pi_z}(s,a) π z ( s ) = a r g m a x a ∈ A Q π z ( s , a ) π z ( s ) \pi_z(s) π z ( s )
这样我们就找到一个目标等式 ( 2 ) (2) ( 2 ) ( 3 ) (3) ( 3 ) ( 3 ) (3) ( 3 ) ( 1 ) (1) ( 1 )
将 F , B F,B F , B θ , ω \theta, \omega θ , ω
L F B ( θ , ω ) : = 1 2 [ E s ′ ∼ p ( ⋅ ∣ s , a ) , a ′ ∼ π ( ⋅ ∣ s ′ ) s ′ ′ ∼ ρ , s ′ ′ ∈ X [ F θ ( s , a , z ) T B ω ( s ′ ′ ) − γ F ˉ θ ( s ′ , a ′ , z ) T B ˉ ω ( s ′ ′ ) ] 2 − F θ ( s , a , z ) T E s ′ ∼ ρ s ′ ∈ X [ B ω ( s ′ ) ] \begin{aligned}
\mathcal{L}_{FB}(\theta,\omega):=&\ \frac{1}{2}\left[\mathbb{E}_{\substack{s'\sim p(\cdot|s,a),a'\sim\pi(\cdot|s')\\s''\sim\rho,s''\in X}}[F_{\theta}(s,a,z)^TB_{\omega}(s'')-\gamma\bar{F}_{\theta}(s',a',z)^T\bar{B}_{\omega}(s'')\right]^2 \\
&\qquad - F_{\theta}(s,a,z)^T\mathbb{E}_{\substack{s'\sim\rho\\s'\in X}}[B_{\omega}(s')]
\end{aligned}
L F B ( θ , ω ) : = 2 1 [ E s ′ ∼ p ( ⋅ ∣ s , a ) , a ′ ∼ π ( ⋅ ∣ s ′ ) s ′ ′ ∼ ρ , s ′ ′ ∈ X [ F θ ( s , a , z ) T B ω ( s ′ ′ ) − γ F ˉ θ ( s ′ , a ′ , z ) T B ˉ ω ( s ′ ′ ) ] 2 − F θ ( s , a , z ) T E s ′ ∼ ρ s ′ ∈ X [ B ω ( s ′ ) ]
解释 ：将 M π M^{\pi} M π ( 2 ) (2) ( 2 ) ( 1 ) (1) ( 1 ) ℓ 2 \ell_2 ℓ 2
L ( θ , ω ) = [ F θ ( s , a , z ) E s ′ ∼ ρ s ′ ∈ X [ B ω ( s ′ ) ] − P ( X ∣ s , a ) − γ E s ′ ∼ p ( ⋅ ∣ s , a ) a ′ ∼ π ( ⋅ ∣ s ′ ) F ˉ θ ( s ′ , a ′ , z ) E s ′ ′ ∼ ρ s ′ ′ ∈ X [ B ˉ ω ( s ′ ′ ) ] ] 2 = [ E s ′ ∼ p ( ⋅ ∣ s , a ) , a ′ ∼ π ( ⋅ ∣ s ′ ) s ′ ′ ∼ ρ , s ′ ′ ∈ X [ F θ ( s , a , z ) B ω ( s ′ ′ ) − γ F ˉ θ ( s ′ , a ′ , z ) B ˉ ω ( s ′ ′ ) ] ] 2 − 2 P ( X ∣ s , a ) F θ ( s , a , z ) E s ′ ∼ ρ s ′ ∈ X [ B ω ( s ′ ) ] − 2 γ P ( X ∣ s , a ) E s ′ ∼ p ( ⋅ ∣ s , a ) a ′ ∼ π ( ⋅ ∣ s ′ ) F ˉ θ ( s ′ , a ′ , z ) E s ′ ′ ∼ ρ s ′ ′ ∈ X [ B ˉ ω ( s ′ ′ ) ] \begin{aligned}
\mathcal{L}(\theta,\omega) =&\ \left[F_{\theta}(s,a,z)\mathbb{E}_{\substack{s'\sim\rho\\s'\in X}}[B_{\omega}(s')]-P(X|s,a)-\gamma\mathbb{E}_{\substack{s'\sim p(\cdot|s,a)\\a'\sim\pi(\cdot|s')}}\bar{F}_{\theta}(s',a',z)\mathbb{E}_{\substack{s''\sim\rho\\s''\in X}}[\bar{B}_{\omega}(s'')]\right]^2\\
=&\ \left[\mathbb{E}_{\substack{s'\sim p(\cdot|s,a),a'\sim\pi(\cdot|s')\\s''\sim\rho,s''\in X}}[F_{\theta}(s,a,z)B_{\omega}(s'')-\gamma\bar{F}_{\theta}(s',a',z)\bar{B}_{\omega}(s'')]\right]^2\\
&\quad -2P(X|s,a)F_{\theta}(s,a,z)\mathbb{E}_{\substack{s'\sim\rho\\s'\in X}}[B_{\omega}(s')] - 2\gamma P(X|s,a)\mathbb{E}_{\substack{s'\sim p(\cdot|s,a)\\a'\sim\pi(\cdot|s')}}\bar{F}_{\theta}(s',a',z)\mathbb{E}_{\substack{s''\sim\rho\\s''\in X}}[\bar{B}_{\omega}(s'')]
\end{aligned}
L ( θ , ω ) = = [ F θ ( s , a , z ) E s ′ ∼ ρ s ′ ∈ X [ B ω ( s ′ ) ] − P ( X ∣ s , a ) − γ E s ′ ∼ p ( ⋅ ∣ s , a ) a ′ ∼ π ( ⋅ ∣ s ′ ) F ˉ θ ( s ′ , a ′ , z ) E s ′ ′ ∼ ρ s ′ ′ ∈ X [ B ˉ ω ( s ′ ′ ) ] ] 2 [ E s ′ ∼ p ( ⋅ ∣ s , a ) , a ′ ∼ π ( ⋅ ∣ s ′ ) s ′ ′ ∼ ρ , s ′ ′ ∈ X [ F θ ( s , a , z ) B ω ( s ′ ′ ) − γ F ˉ θ ( s ′ , a ′ , z ) B ˉ ω ( s ′ ′ ) ] ] 2 − 2 P ( X ∣ s , a ) F θ ( s , a , z ) E s ′ ∼ ρ s ′ ∈ X [ B ω ( s ′ ) ] − 2 γ P ( X ∣ s , a ) E s ′ ∼ p ( ⋅ ∣ s , a ) a ′ ∼ π ( ⋅ ∣ s ′ ) F ˉ θ ( s ′ , a ′ , z ) E s ′ ′ ∼ ρ s ′ ′ ∈ X [ B ˉ ω ( s ′ ′ ) ]
对 θ , ω \theta,\omega θ , ω min \min min X X X ( s , a , s ′ ) (s,a,s') ( s , a , s ′ )
min θ , ω L F B ( θ , ω ) ⟺ min θ , ω 1 2 [ E s ′ ∼ p ( ⋅ ∣ s , a ) , a ′ ∼ π ( ⋅ ∣ s ′ ) s ′ ′ ∼ ρ , s ′ ′ ∈ X [ F θ ( s , a , z ) T B ω ( s ′ ′ ) − γ F ˉ θ ( s ′ , a ′ , z ) T B ˉ ω ( s ′ ′ ) ] 2 − F θ ( s , a , z ) T E s ′ ∼ P ( ⋅ ∣ s , a ) s ′ ∼ ρ B ω ( s ′ ) \begin{aligned}
\min_{\theta,\omega}\mathcal{L}_{FB}(\theta,\omega)\iff&\ \min_{\theta,\omega}\frac{1}{2}\left[\mathbb{E}_{\substack{s'\sim p(\cdot|s,a),a'\sim\pi(\cdot|s')\\s''\sim\rho,s''\in X}}[F_{\theta}(s,a,z)^TB_{\omega}(s'')-\gamma\bar{F}_{\theta}(s',a',z)^T\bar{B}_{\omega}(s'')\right]^2 \\
&\qquad\qquad -F_{\theta}(s,a,z)^T\mathbb{E}_{\substack{s'\sim P(\cdot|s,a)\\s'\sim \rho}}B_{\omega}(s')
\end{aligned}
θ , ω min L F B ( θ , ω ) ⟺ θ , ω min 2 1 [ E s ′ ∼ p ( ⋅ ∣ s , a ) , a ′ ∼ π ( ⋅ ∣ s ′ ) s ′ ′ ∼ ρ , s ′ ′ ∈ X [ F θ ( s , a , z ) T B ω ( s ′ ′ ) − γ F ˉ θ ( s ′ , a ′ , z ) T B ˉ ω ( s ′ ′ ) ] 2 − F θ ( s , a , z ) T E s ′ ∼ P ( ⋅ ∣ s , a ) s ′ ∼ ρ B ω ( s ′ )
假设batch大小为 n n n z i z_i z i ( s i , a i , s i ′ , z i ) , ⋯ (s_i,a_i,s'_i,z_i),\cdots ( s i , a i , s i ′ , z i ) , ⋯ a i ′ ∼ π ( ⋅ ∣ s i , z i ) a'_i\sim \pi(\cdot|s_i,z_i) a i ′ ∼ π ( ⋅ ∣ s i , z i )
L F B ( θ , ω ) = 1 n ( n − 1 ) ∑ i ≠ j [ F θ ( s i , a i , z i ) T B ω ( s j ) − γ F ˉ θ ( s i ′ , a i ′ , z i ) T B ˉ ω ( s j ) ] 2 − 1 n ∑ i F θ ( s i , a i , z i ) T B ω ( s i ′ ) \mathcal{L}_{FB}(\theta,\omega) = \frac{1}{n(n-1)}\sum_{i\neq j}\left[F_{\theta}(s_i,a_i,z_i)^TB_{\omega}(s_j)-\gamma\bar{F}_{\theta}(s'_i,a'_i,z_i)^T\bar{B}_{\omega}(s_j)\right]^2 - \frac{1}{n}\sum_{i}F_{\theta}(s_i,a_i,z_i)^TB_{\omega}(s'_i)
L F B ( θ , ω ) = n ( n − 1 ) 1 i = j ∑ [ F θ ( s i , a i , z i ) T B ω ( s j ) − γ F ˉ θ ( s i ′ , a i ′ , z i ) T B ˉ ω ( s j ) ] 2 − n 1 i ∑ F θ ( s i , a i , z i ) T B ω ( s i ′ )
上述推导虽然已经对 F θ , B ω , π ϕ F_{\theta}, B_{\omega}, \pi_{\phi} F θ , B ω , π ϕ π ϕ \pi_{\phi} π ϕ π ϕ \pi_{\phi} π ϕ
专家数据集仅有状态构成，记为 M = { ( s 1 , ⋯ , s l ( τ ) ) } = { τ } M=\{(s_1,\cdots,s_{l(\tau)})\}=\{\tau\} M = { ( s 1 , ⋯ , s l ( τ ) ) } = { τ }
思考 ：为什么专家数据集 M = { τ } M=\{\tau\} M = { τ } s s s s s s s s s π \pi π p M ( s , π ) p_{M}(s,\pi) p M ( s , π ) π \pi π π \pi π z ∈ R d z\in\mathbb{R}^d z ∈ R d p M ( s , z ) p_{M}(s,z) p M ( s , z )
策略神经网络 π ( ⋅ ∣ s , z ) \pi(\cdot|s,z) π ( ⋅ ∣ s , z ) z z z s s s
那么类似地，对于 π ϕ ( ⋅ ∣ s , z ) \pi_{\phi}(\cdot|s,z) π ϕ ( ⋅ ∣ s , z ) p π z ( s , z ) p_{\pi_{z}}(s,z) p π z ( s , z ) s s s π ϕ \pi_{\phi} π ϕ p π z ( s , z ) p_{\pi_z}(s,z) p π z ( s , z ) p M ( s , z ) p_{M}(s,z) p M ( s , z )
这里可以用 K L KL K L p M ( s , z ) p_{M}(s,z) p M ( s , z ) D ψ : S × R d → [ 0 , 1 ] D_{\psi}:S\times \mathbb{R}^d\to [0,1] D ψ : S × R d → [ 0 , 1 ] s s s z z z p M ( ⋅ ∣ s , z ) p_{M}(\cdot|s,z) p M ( ⋅ ∣ s , z ) p π ϕ ( ⋅ ∣ s , z ) p_{\pi_{\phi}}(\cdot|s,z) p π ϕ ( ⋅ ∣ s , z ) π ϕ \pi_{\phi} π ϕ p π ϕ ( ⋅ ∣ s , z ) p_{\pi_{\phi}}(\cdot|s,z) p π ϕ ( ⋅ ∣ s , z ) p M ( ⋅ ∣ s , z ) p_M(\cdot|s,z) p M ( ⋅ ∣ s , z ) D ψ D_{\psi} D ψ
L d i s c r i m i n a t o r ( ψ ) = − E ( s , z ) ∼ p M [ log ( D ψ ( s , z ) ) ] − E ( s , z ) ∼ p π ϕ [ log ( 1 − D ψ ( s , z ) ) ] = − E s ∼ M [ log ( D ψ ( s , E s ′ ∼ τ ( s ) [ B ( s ′ ) ] ) ] − E z ∼ υ , s ∼ ρ π z [ log ( 1 − D ψ ( s , z ) ) ] \begin{aligned}
\mathcal{L}_{discriminator}(\psi) =&\ -\mathbb{E}_{(s,z)\sim p_{M}}[\log(D_{\psi}(s,z))]-\mathbb{E}_{(s,z)\sim p_{\pi_\phi}}[\log(1-D_{\psi}(s,z))]\\
=&\ -\mathbb{E}_{s\sim M}\left[\log(D_{\psi}(s,\mathbb{E}_{s'\sim\tau(s)}[B(s')])\right] -\mathbb{E}_{z\sim\upsilon,s\sim \rho^{\pi_z}}[\log(1-D_{\psi}(s,z))]
\end{aligned}
L d i s c r i m i n a t o r ( ψ ) = = − E ( s , z ) ∼ p M [ log ( D ψ ( s , z ) ) ] − E ( s , z ) ∼ p π ϕ [ log ( 1 − D ψ ( s , z ) ) ] − E s ∼ M [ log ( D ψ ( s , E s ′ ∼ τ ( s ) [ B ( s ′ ) ] ) ] − E z ∼ υ , s ∼ ρ π z [ log ( 1 − D ψ ( s , z ) ) ]
上式存在理论最优解 D ∗ ( s , z ) = p M ( s , z ) p M ( s , z ) + p π z ( s , z ) D^*(s,z)=\frac{p_{M}(s,z)}{p_{M}(s,z)+p_{\pi_z}(s,z)} D ∗ ( s , z ) = p M ( s , z ) + p π z ( s , z ) p M ( s , z )
证明方法可以直接对 p M log D + p π z log ( 1 − D ) p_{M}\log D+p_{\pi_z}\log(1-D) p M log D + p π z log ( 1 − D ) D D D 0 0 0
π z = π ϕ ( ⋅ ∣ s , z ) \pi_z=\pi_{\phi}(\cdot|s,z) π z = π ϕ ( ⋅ ∣ s , z ) D D D
max r ( s , z ) = log p M ( s , z ) p π z ( s , z ) = log D ∗ 1 − D ∗ ≈ log D ψ 1 − D ψ \max r(s,z) = \log\frac{p_{M}(s,z)}{p_{\pi_z}(s,z)} = \log\frac{D^*}{1-D^*} \approx \log\frac{D_{\psi}}{1-D_{\psi}}
max r ( s , z ) = log p π z ( s , z ) p M ( s , z ) = log 1 − D ∗ D ∗ ≈ log 1 − D ψ D ψ
用TD方式对上述折后回报进行估计，令 Q η ( s , a ) : S × A → R Q_{\eta}(s,a):S\times A\to \mathbb{R} Q η ( s , a ) : S × A → R
L c r i t i c ( η ) = E ( s , a , s ′ ) ∼ D o n l i n e z ∼ υ , a ′ ∼ π z ( ⋅ ∣ s ′ ) [ ( Q η ( s , a , z ) − log D ψ ( s ′ , z ) 1 − D ψ ( s ′ , z ) − γ Q ˉ η ( s ′ , a ′ , z ) 2 ] \mathcal{L}_{critic}(\eta) = \mathbb{E}_{\substack{(s,a,s')\sim D_{online}\\z\sim \upsilon,a'\sim\pi_z(\cdot|s')}}\left[\left(Q_{\eta}(s,a,z)-\log\frac{D_{\psi}(s',z)}{1-D_{\psi}(s',z)}-\gamma \bar{Q}_{\eta}(s',a',z\right)^2\right]
L c r i t i c ( η ) = E ( s , a , s ′ ) ∼ D o n l i n e z ∼ υ , a ′ ∼ π z ( ⋅ ∣ s ′ ) [ ( Q η ( s , a , z ) − log 1 − D ψ ( s ′ , z ) D ψ ( s ′ , z ) − γ Q ˉ η ( s ′ , a ′ , z ) 2 ]
将模仿奖励 Q Q Q ( 3 ) (3) ( 3 )
L a c t o r ( ϕ ) = E s ∼ D , z ∼ υ a ∼ π ψ ( ⋅ ∣ s , z ) [ F θ ( s , a , z ) T z + α Q η ( s , a , z ) ] (4) \tag{4}\mathcal{L}_{actor}(\phi) = \mathbb{E}_{\substack{s\sim D,z\sim\upsilon\\a\sim\pi_{\psi}(\cdot|s,z)}}\left[F_{\theta}(s,a,z)^Tz+\alpha Q_{\eta}(s,a,z)\right]
L a c t o r ( ϕ ) = E s ∼ D , z ∼ υ a ∼ π ψ ( ⋅ ∣ s , z ) [ F θ ( s , a , z ) T z + α Q η ( s , a , z ) ] ( 4 )
综上，我们完成了4个主要损失函数的定义
L F B ( θ , ω ) \mathcal{L}_{FB}(\theta,\omega) L F B ( θ , ω ) F θ ( s , a , z ) , B ω ( s ) F_{\theta}(s,a,z),B_{\omega}(s) F θ ( s , a , z ) , B ω ( s ) L d i s c r i m i n a t o r ( ψ ) \mathcal{L}_{discriminator}(\psi) L d i s c r i m i n a t o r ( ψ ) D ψ ( s , z ) D_{\psi}(s,z) D ψ ( s , z ) L c r i t i c ( η ) \mathcal{L}_{critic}(\eta) L c r i t i c ( η ) Q η ( s , z ) Q_{\eta}(s,z) Q η ( s , z ) L a c t o r ( ϕ ) \mathcal{L}_{actor}(\phi) L a c t o r ( ϕ ) π ϕ ( ⋅ ∣ s , z ) \pi_{\phi}(\cdot|s,z) π ϕ ( ⋅ ∣ s , z )
假设我们维护了一个在线训练buffer D o n l i n e \mathcal{D}_{online} D o n l i n e M \mathcal{M} M τ o n l i n e , τ u n l a b l e d \tau_{online},\tau_{unlabled} τ o n l i n e , τ u n l a b l e d T T T
通过随机获取 z z z π ϕ ( ⋅ , z ) \pi_{\phi}(\cdot,z) π ϕ ( ⋅ , z ) D o n l i n e \mathcal{D}_{online} D o n l i n e z z z z z z
z = { B ( s ) , s ∼ D o n l i n e , 概率 τ o n l i n e , 1 T ∑ t = 1 T B ( s t ) , { s 1 , ⋯ , s T } ∼ M , 概率 τ u n l a b l e d , ∼ N ( 0 , I d ) , 概率 1 − τ o n l i n e − τ u n l a b l e d . z ← d z ∣ ∣ z ∣ ∣ 2 , 用 π ϕ ( ⋅ , z ) 与环境交互 T 步，将数据存储入 D o n l i n e z=\begin{cases}
B(s),&\quad s\sim \mathcal{D}_{online}, &\ \text{概率}\tau_{online},\\
\frac{1}{T}\sum_{t=1}^TB(s_t),&\quad \{s_1,\cdots,s_{T}\}\sim \mathcal{M},&\quad \text{概率}\tau_{unlabled},\\
\sim\mathcal{N}(0,I_d),&\quad &\quad \text{概率}1-\tau_{online}-\tau_{unlabled}.
\end{cases}\\
z\gets\sqrt{d}\frac{z}{||z||_2}, \text{用}\pi_{\phi}(\cdot,z)\text{与环境交互}T\text{步，将数据存储入} \mathcal{D}_{online}
z = ⎩ ⎪ ⎪ ⎨ ⎪ ⎪ ⎧ B ( s ) , T 1 ∑ t = 1 T B ( s t ) , ∼ N ( 0 , I d ) , s ∼ D o n l i n e , { s 1 , ⋯ , s T } ∼ M , 概率 τ o n l i n e , 概率 τ u n l a b l e d , 概率 1 − τ o n l i n e − τ u n l a b l e d . z ← d ∣ ∣ z ∣ ∣ 2 z , 用 π ϕ ( ⋅ , z ) 与环境交互 T 步，将数据存储入 D o n l i n e
