Code for the paper How Low Can We Go: Trading Memory for Error in Low-Precision Training (arXiv:2106.09686).
Low-precision arithmetic trains deep learning models using less energy, less memory and less time. However, we pay a price for the savings: lower precision may yield larger round-off error and hence larger prediction error. We view these precision choices as a hyperparameter tuning problem, and borrow ideas from meta-learning to learn the tradeoff between memory and error. We introduce Pareto Estimation to Pick the Perfect Precision (PEPPP): we use matrix factorization to find non-dominated precision configurations (the Pareto frontier) with a limited number of network evaluations. For any given memory budget, the precision that minimizes error is a point on this frontier.
peppp_code/
│
├── architectures/ # Low-precision neural network architectures
│ ├── resnet_lp.py # ResNet with quantized layers
│ ├── vgg_lp.py # VGG with quantized layers
│ ├── resnet_cifar.py # ResNet for CIFAR
│ ├── resnet_mnist.py # ResNet for MNIST
│ ├── resnet_stl.py # ResNet for STL-10
│ ├── resnext_cifar.py # ResNeXt for CIFAR
│ └── vgg_cifar.py # VGG for CIFAR
│
├── peppp/ # Core PEPPP algorithm
│ ├── experiment_design.py # D-optimal experiment design (greedy & convexification)
│ └── utils_qtorch.py # QPyTorch quantization helpers & LP training utilities
│
├── training/ # Training and evaluation scripts
│ ├── train.py # Main low-precision training loop
│ ├── evaluate.py # Model evaluation script
│ ├── train_func.py # Data loading, transforms, checkpoint utilities
│ ├── datasets.py # Custom dataset classes (iNaturalist, InMemoryDataLoader)
│ ├── cifar100_main.py # CIFAR-100 superclass training script
│ └── bram_main.py # DECA dataset training script
│
├── utils/ # General-purpose utilities
│ ├── utils.py # CSV logging, checkpoint saving, accuracy computation
│ └── convert_cifar100.py # Convert CIFAR-100 into superclass image folders
│
├── configs/ # Configuration files
│ └── arith_format.json # Arithmetic format specifications
│
├── notebooks/ # Jupyter notebooks for analysis & visualization
│ ├── pareto_frontier_plotting_meta_test.ipynb
│ ├── pareto_frontier_plotting_meta_training_new_violin.ipynb
│ ├── dataset_kendall_tau_correlation.ipynb
│ ├── memory_calculation.ipynb
│ ├── Experiment_Vision_Classification.ipynb
│ └── ... (additional analysis notebooks)
│
├── qtorch/ # Vendored QPyTorch library (low-precision quantization)
├── torchsummary/ # Vendored torchsummary library
├── scripts/ # SLURM job submission scripts (for cluster training)
├── requirements.txt # Python dependencies
└── .gitignore
git clone https://github.com/chengrunyang/peppp
cd peppp
pip install -r requirements.txtpython training/train.py \
--arch resnet18 \
--data cifar10 \
--epo 100 \
--bs 32 \
--lr 0.001 \
--mom 0.9 \
--wd 5e-4 \
--data_dir ./data/python training/evaluate.py \
--model_dir ./saved_models/<model_name> \
--data_dir ./data/The core PEPPP algorithm is in peppp/experiment_design.py. It provides two methods for selecting precision configurations to evaluate:
- Greedy D-optimal design (
greedy_stepwise_selection): iteratively selects the most informative configurations. - Convexification (
convexification_solve): solves a relaxed optimization to find optimal configurations.
| Component | Description |
|---|---|
peppp/experiment_design.py |
D-optimal experiment design for selecting which precision configurations to evaluate |
peppp/utils_qtorch.py |
Quantization utilities, LP model construction, and LP training functions |
training/train.py |
Main training script iterating over low/high precision format pairs |
training/train_func.py |
Architecture loading, dataset loading, data transforms, and checkpoint management |
architectures/resnet_lp.py |
ResNet with QPyTorch quantized layers |
architectures/vgg_lp.py |
VGG with QPyTorch quantized layers |
The training scripts sweep over combinations of low-precision and high-precision floating-point formats (defined by exponent and mantissa bit widths). Each configuration specifies:
- Low-precision format: used for forward/backward pass computations
- High-precision format: used for optimizer state (momentum, accumulation)
@article{yang2021how,
title={How Low Can We Go: Trading Memory for Error in Low-Precision Training},
author={Yang, Chengrun and Wu, Ziyang and Chee, Jerry and De Sa, Christopher and Udell, Madeleine},
journal={arXiv preprint arXiv:2106.09686},
year={2021}
}This project is released under the MIT License.