Most Compatible Reinforcement Learning Algorithm for Deep Brain Stimulation

Abstract

Tremors are a symptom of Parkinson’s disease that causes involuntary shaking movements in the hands and other parts of the body, which can disturb one’s quality of life. Tremors happen when malfunctioning neurons synchronize. Therefore, suppression and control of this collective synchronous activity are of great importance. Deep brain stimulation is where surgeons decide the amplitude and frequency of the stimulation to nullify collected signals from synchronous neurons in the brain according to their observation and expertise. A virtual Reinforcement Learning environment Krylov et al. created in 2020 can simulate this collective behavior of neurons and allows us to find suppression parameters for the environment of synthetic degenerate models of neurons. Although the newest generation of Deep Brain Stimulation technology does provide feedback functionality (they can be controlled using both traditional, physical controllers and machine algorithms), it is still challenging to decide which algorithm is most suitable for the task; the study by Krylov et al. applies Proximal Policy Optimization to their environment and successfully suppresses the synchronization in neuron activity. However, they do not test other types of algorithms. This paper expands upon their findings by systematically evaluating six reinforcement learning algorithms (A2C, DDPG, PPO, SAC, TD3, and TRPO). Our results indicate that Trust Region Policy Optimization (TRPO) is the most effective under conditions of low learning rate, moderate divergence between updates, and prioritizing long-term rewards.