ABSTRACT
ACKNOWLEDGMENTS
1 INTRODUCTION
1.1 The Unreasonable Effectiveness of Neural Networks
1.2 Processors at the End of Dennard Scaling
1.3 Intelligent Devices: The Drive Towards On-Device Inference
1.4 Shrinking Neural Networks
1.5 Research Challenges
1.6 Proposed System Design Methodology
1.7 Key Contributions and Thesis Overview
2 PRELIMINARIES
2.1 Neural Networks
2.1.1 Inference
2.1.2 Learning through Backpropagation
2.1.3 Convolutional Neural Networks
2.2 Quantization
2.2.1 Fixed-Point and Floating-Point Representations
2.2.2 Post-Training Quantization and Quantization-Aware Training
2.3 Pruning
2.3.1 Sensitivity Analysis
2.3.2 Iterative Pruning and Fine-Tuning
2.4 Approximate Multiplications
2.4.1 Truncated Multiplier
2.4.2 Logarithmic Multiplier
3 APPROXIMATE MULTIPLICATIONS DURING NETWORK TRAINING
3.1 Introduction
3.2 Related Work
3.2.1 Approximate Multipliers
3.2.2 Approximate Neural Network Training
3.3 Granular Quantization for Approximate Neural Networks
3.3.1 Motivation
3.3.2 Approximate Per-Channel Quantization
3.4 From Fake-Quantization to Fake-Approximation
3.4.1 Rationale
3.4.2 Modeling Approximate Multiplications as a Composition of Transformations and Accurate Product Functions
3.4.3 Required Properties and Limitations
3.5 Analytic Implementation of Fake-Approximation
3.5.1 Modeling Logarithmic Approximate Multipliers
3.5.2 Performance Considerations for GPUs
3.6 Data-driven Implementation of Fake-Approximation
3.6.1 Modeling Truncated Multipliers
3.6.2 Determining Model Parameters through Linear Regression
3.7 Generalization of Fake-Approximation to Arbitrary Circuits
3.8 Design Considerations for the TorchApprox Software Toolkit
3.9 Accuracy Impact of Quantization Granularity
3.9.1 Aggregate Evaluation on a Population of Models
3.9.2 Impact on Individual Models
3.10 Fidelity and Throughput of the Proposed Floating-Point Models
3.10.1 Experimental Setup
3.10.2 Evaluation
3.11 Inference Throughput of Approximate Deep Learning Frameworks
3.11.1 Experimental Setup
3.11.2 Evaluation
3.11.3 Comparison with TFApprox
3.12 Conclusion
4 MODEL SENSITIVITY TO APPROXIMATE MULTIPLICATIONS
4.1 Introduction
4.2 Related Work
4.2.1 Retraining for Approximate Multiplications through Gaussian Noise
4.2.2 Heterogeneous Approximate Multiplier Assignment
4.3 Modeling Approximate Multiplication as Noise
4.3.1 Propagation of Small Numerical Errors in a Neuron
4.3.2 Handling of the Error Mean
5.7 Case Study: Large Sparse vs. Small Dense MobileNetV2 on CIFAR-10
5.8 Hybrid Compression for Image Classification Tasks
5.8.1 Accuracy and Sparsity
5.8.2 Extraction Throughput
5.9 Conclusion
6 ADAPTIVE APPROXIMATE COMPUTING AND SPARSITY
6.1 Introduction
6.2 Related Work
6.3 n-constrained Multiplier Selection
6.3.1 Rationale
6.3.2 Input Preparation and Clustering
6.3.3 Interpretation of Clustering Results
6.4 Extending to Multiple Approximate Operating Points
6.5 Model Size Reduction
6.5.1 Rationale
6.5.2 Importance of Error Mean
6.5.3 Fine-Tuning Scheme
6.5.4 Sparsity for Shared Weights
6.6 Optimization of a Single Operating Point
6.6.1 Training Setup
6.6.2 Reference Optimization Techniques
6.6.3 Achieved Accuracy
6.7 Impact of Multiple Operating Points
6.7.1 Training Setup
6.7.2 Accuracy and Model Size
6.7.3 Model Compression through Sparsity and Hybrid Encoding
6.8 Conclusion
7 CONCLUSION AND OUTLOOK
7.1 Conclusion
7.2 Outlook
BIBLIOGRAPHY
APPENDIX
A Use of Large Language Models in the Writing of This Thesis
B Approximate Multiplication Training Framework
B.1 Network Training Setup
B.2 Experimental Results
C Compressed Sparsity
C.1 Likelihood of Element Groups
C.2 Spacing of Elements
C.3 Impact of Group Occupancy θ on Partitioning
C.4 Binary Group Mask
C.5 Keyword Spotting Reference Architecture