This page introduces the ai projects I have made so far.
CartPole-v1
Before delving into AI exercises, I took the initiative to learn the concepts of artificial intelligence, including network architectures like CNN and algorithms such as PPO (Proximal Policy Optimization). Understanding the principles of artificial intelligence, I proceeded with practical implementation because I believed that hands-on experience would help me overcome learning gaps and avoid mere theoretical discussions.
When it came to selecting the environment, I didn't start by collecting and analyzing real-life data. Recognizing that I would be tackling the task on my own without external assistance, I realized that handling data processing, a complex task, might be challenging for me at that time. Moreover, troubleshooting issues could potentially prove difficult.
Therefore, I opted to use Cartpole, an environment provided by OpenAI, and began by practicing reinforcement learning. The process involved creating the environment, constructing a Q-table using NumPy arrays, implementing the Q-learning algorithm (including techniques like epsilon-greedy and replay buffer), and ultimately conducting training. I trained the model over 20,000 episodes to ensure that the pole would not fall.
Throughout the implementation, I encountered several challenges, with the most significant being version updates. At that time, I couldn't find tutorials addressing those specific issues. To overcome them, I carefully read the entire API documentation and conducted multiple trial-and-error attempts. Although solving a problem could take me nearly half a month (for instance, it took me 16 discontinuous days to solve a rendering issue), I persevered and successfully completed my first AI project.
PongNoFrameskip-v4 (DQN)
In my first implementation, I encountered numerous device-related issues, including compatibility problems and unsupported applications on the devices. These problems were often difficult to resolve directly. Therefore, for my second attempt, I chose to use Google Colab. The implementation process can be summarized into four main steps: environment setup, image preprocessing, algorithm (DQN) implementation, and the training and testing process.
Having already implemented Q-learning, I proceeded to try DQN (Deep Q-learning), which essentially replaces the Q-table in Q-learning with a convolutional neural network.
When the Q-values of all actions and states increase, the increase of the Q-table led to Q-learning failing to calculate all the states of Q-value. However, DQN can effectively solve this problem. Furthermore, DQN incorporates a replay memory, which stores data in a sequential manner; then collecting data in a random manner during training, thereby avoiding correlations.
BipedalWalker-v3 (PPO)
This environment has 24 state dimensions and 4 action dimensions, which is a significant increase compared to the previous attempts where there were only four. Due to the increased complexity, learning in this environment became relatively challenging, prompting me to use PPO (Proximal Policy Optimization). When implementing the construction of the policy network and value network, both networks have continuous outputs. To handle this, I utilized the Diagonal Gaussian Distribution Module. Essentially, this involved adding a bias, normal distribution with a given mean, and diagonal Gaussian distribution. I implemented these using PyTorch and referred to relevant APIs throughout the process.
Unlike the previous project, I didn't encounter the same level of difficulty in getting stuck or facing conflicts due to version updates. Perhaps it was because I had a better understanding of how to solve problems or because there were no conflicting issues caused by updates. I believe the most challenging aspect was maintaining the determination and perseverance to solve problems. Even a single function could take two to three days to complete, as it required not only staying on track but also comprehending the full usage of that function.
