zeynpakn/FrozenLake-Q-Learning

🐤🐸 FrozenLake — Q-Learning

Deterministic vs. Stochastic Environments

Two tabular Q-Learning agents trained on Gymnasium's FrozenLake-v1 (4×4) to explore how environmental uncertainty affects reinforcement learning.

This project extends the Hugging Face Deep RL Course — Unit 2 with comparative experiments and animated demonstrations.

Meet the Agents

	🐤 Ducky	🐸 Froggy
Environment	Non-slippery	Slippery
Transitions	Deterministic	Stochastic
Algorithm	Q-Learning	Q-Learning
Training Episodes	10,000	10,000
Evaluation Success Rate	100%	73.7%

Both agents use identical hyperparameters, so the only difference between them is the environment's transition dynamics.

🐤🐸 Watch the Agents Play

🐤 Ducky — Non-Slippery	🐸 Froggy — Slippery

Ducky follows the same predictable path every time, while Froggy has to deal with slippery ice that can push it in unintended directions. Both animations show successful episodes using the agents' greedy policies.

Training Configuration

Hyperparameter	Value
Learning Rate (α)	0.7
Discount Factor (γ)	0.95
Initial Epsilon	1.0
Minimum Epsilon	0.01
Epsilon Decay	0.999 (multiplicative, per episode)
Training Episodes	10,000
Max Steps per Episode	100
Random Seed	42

Each agent learns a Q-Table of 16 states × 4 actions using epsilon-greedy exploration.

Evaluation Results

Both agents were evaluated over 1,000 episodes with greedy action selection and no further Q-Table updates.

Metric	Ducky 🐤	Froggy 🐸
Success Rate	100%	73.7%
Mean Reward	1.000	0.737
Reward Standard Deviation	0.000	0.440
Average Steps per Episode	6.00	40.57
Average Steps in Successful Episodes	6.00	36.87
Success Rate (last 100 training episodes)	99%	63%

Training success rates are lower than evaluation success rates because the agents were still exploring (ε > 0) during training.

Ducky reaches the goal in the optimal 6 steps every time. Froggy cannot guarantee success even with a well-learned policy: on the slippery map, the intended move happens only one-third of the time. Its 73.7% success rate is close to the best achievable on this map within a 100-step limit, so the gap reflects the environment, not a failure to learn.

These results measure performance on the same map used during training, not generalization to unseen maps.

Key Takeaway

The same Q-Learning algorithm can produce substantially different policies and performance depending on the environment's transition dynamics.

In a deterministic world, the shortest path is the best path. In a stochastic one, the agent learns longer but safer routes that reduce the risk of slipping into a hole.

Usage

The repository contains two trained Q-Tables:

models/
├── ducky_q_table.npy
└── froggy_q_table.npy

Both are compatible with Gymnasium's default 4×4 FrozenLake-v1 state and action spaces.

import numpy as np
import gymnasium as gym
from huggingface_hub import hf_hub_download

repo_id = "zeynpakn/FrozenLake-Q-Learning"

q_table = np.load(hf_hub_download(repo_id=repo_id, filename="models/froggy_q_table.npy"))
env = gym.make("FrozenLake-v1", map_name="4x4", is_slippery=True)  # use is_slippery=False for Ducky

state, _ = env.reset(seed=42)
done = False
while not done:
    action = int(np.argmax(q_table[state]))  # greedy action
    state, reward, terminated, truncated, _ = env.step(action)
    done = terminated or truncated

print("Reached the goal!" if reward == 1 else "Fell into a hole or ran out of steps.")

---

🐤 Ducky follows the rules. Froggy negotiates with physics. 🐸

---

Built as an educational reinforcement learning experiment, inspired by the Hugging Face Deep RL Course.