Home

Welcome!

Process Control (pc-) gym is a set of benchmark chemical process control problems for reinforcement learning with integrated policy evaluation methods to aid the development of reinforcement learning algorithms.

The pc-gym was developed within the Sargent Centre for Process Systems Engineering and is published as an open-source package which welcomes contributions from the RL and PSE communities.

Note: this is a pre-release version of the documentation and may not reflect the current version of pc-gym

Quick start

Setup a CSTR environment with a setpoint change

import pcgym

# Simulation variables
nsteps = 100
T = 25

# Setpoint
SP = {'Ca': [0.85 for i in range(int(nsteps/2))] + [0.9 for i in range(int(nsteps/2))]} 

# Action and observation Space
action_space = {'low': np.array([295]), 'high': np.array([302])}
observation_space = {'low': np.array([0.7,300,0.8]),'high': np.array([1,350,0.9])}

# Construct the environment parameter dictionary
env_params = {
    'N': nsteps, # Number of time steps
    'tsim':T, # Simulation Time
    'SP' :SP, 
    'o_space' : observation_space, 
    'a_space' : action_space, 
    'x0': np.array([0.8, 330, 0.8]), # Initial conditions [Ca, T, Ca_SP]
    'model': 'cstr_ode', # Select the model
}

# Create environment
env = pcgym.make_env(env_params)

# Reset the environment
obs, state = env.reset()

# Sample a random action
action = env.action_space.sample()

# Perform a step in the environment
obs, rew, done, term, info = env.step(action)

Installation

The latest production pc-gym version can be installed from PyPI:

pip install pcgym

Alternatively, you can install the latest development version directly from GitHub:

pip install git+https://github.com/MaximilianB2/pc-gym

Examples

Example notebooks with training walkthroughs, implementing constraints, disturbances and the policy evaluation tool can be found here.

Environments

Currently there are six implemented process control environments, this will be expanded over time.

Environment	Category	Source Code	Description
CSTR	Reactor	Code	Documentation
First Order System	Simple Model	Code	Documentation
Multistage Extration	Column	Code	Documentation
Nonsmooth Control	Linear System	Code	Documentation
Crystallisation Reactor	Reactor	Code	Documentation
Four Tank System	Level Control	Code	Documentation
Fluidized Biofilm Sand Bed Reactor	Reactor	Code	Documentation
Photoproduction	Bioreactor	Code	Documentation

All environments use the following observation representation for $i$ states and $j$ disturbances: \begin{align} \nonumber o = [x_i,..., x_{i,sp}..., d_j,...] \end{align}

Future Features

The following features are being worked on to be added in the near future: - Observability Mask - More case studies - Custom reward functions

Citing `pc-gym`

If you use pc-gym in your research, please cite using the following

@software{pcgym2024,
  author = {Max Bloor and Jose Neto and Ilya Sandoval and Max Mowbray
            and Akhil Ahmed and Mehmet Mercangoz and Calvin Tsay and Antonio Del Rio-Chanona},
  title = {{pc-gym}: Reinforcement Learning Envionments for Process Control},
  url = {https://github.com/MaximilianB2/pc-gym},
  version = {0.0.4},
  year = {2024},
}

Other great gyms

Other works have built upon the OpenAI Gymnasium framework: