This project provides tools for running experiments on various Markov Decision Process (MDP) types using different solvers for "MDP Geometry, Normalization and Reward Balancing Solvers" (link). It includes scripts for running experiments and plotting the results.

The project consists of two main scripts:

1. run_experiments.py: This script reads an experiment setup from a JSON file, runs the specified experiments on different MDP types using various solvers, and saves the results.

2. plot_results.py: This script reads the saved experiment results and creates plots to visualize the performance of different solvers across various MDP types and parameters.

The following packages are required to run the scripts:

• torch
• matplotlib
• numpy
• tqdm

Note: There are no specific version requirements for these packages. The latest versions should work fine.

To run the experiments:

1. Prepare the experiment setup JSON file with the desired parameters.
2. Run the run_experiments.py script with the JSON file as the argument:

   python run_experiments.py -s experiment_setup.json
   

To plot the results using the same setup file:

1. Run the plot_results.py script:

   python plot_results.py -s experiment_setup.json -m [metric]
   

This will generate plots in the 'figures' directory.

To reproduce the results from the paper, use the full_setup1.json file for the known MDP case, and setup_stochastic_random{2,3}.json for stochastic experiments. Use metric len for known MDP case and opt_policy_diff for stochastic experiments. For example:

python run_experiments.py -s full_setup1.json
python plot_results.py -s full_setup1.json -m len

This will generate plots in the 'figures' directory.

• MDP_Collection.py: Contains implementations of different MDP types.
• Solvers.py: Implements various MDP solvers.
• run_experiments.py: Main script for running experiments.
• plot_results.py: Script for plotting experiment results.
• experiment_setups/: Directory for storing experiment setups.
• experiment_setups/full_setup1.json: Setup file to reproduce paper results.
• experiment_setups/setup_stochastic_random{2,3}.json: Setup file for stochastic experiments.
• results/: Directory for storing experiment results.
• figures/: Directory for storing experiment plots.