1 Introduction
1.1 Semiconductor Fabrication
1.1.1 Die Stacking and Image Sensors
1.1.2 Technology Computer Aided Design and Machine Learning
1.1.3 Plenty Possibilities for reliable Machine Learning
1.2 Machine Learning in real-world Applications
1.3 The Goal of this Work
1.3.1 Research Questions
1.3.2 Contribution
1.4 Structure of this Thesis
2 Machine Learning in Industrial Environments
2.1 Regression and Classification
2.2 The Nomenclature of Data Science
2.3 The Requirements of real-world Industry Applications
2.4 Related Work and Literature Collection
2.4.1 Reliability and Uncertainty Estimates
Sensitivity Analysis
Variance of a Bagged Model
Local Cross Validation Reliability
Local Modeling of Prediction Error
Density-based reliability
Quantile Regression
Transduction
Probabilistic Approach
Conformal Prediction
Reliability Scoring
2.4.2 Concept Drift Detection Techniques
Process Variables
Difficult-to-measure Parameter
Combination of Multiple Factors
2.5 Shortcomings in the Available Approaches
2.6 Summary
3 Output Confidence in Machine Learning
3.1 Confidence for Model Outputs
3.2 Similarity between Training and Inference Data
3.2.1 Feature Knowledge
3.2.2 Concept Drift
3.2.3 Similarity Score
Tuning Parameters and Additional Techniques
3.2.4 Application and Calibration of the Similarity Score
3.3 Uncertainty for Predictions
3.4 The Confidence Estimation Architecture
3.5 Summary
4 Application with a Virtual Metrology Model
4.1 The Features
4.2 Uncertainty Estimation
4.3 Virtual Metrology with Reliability and Uncertainty
Aggregation to the Confidence Score
An interactive way to find a suited threshold
Evaluation of the Confidence Score
4.4 The Confidence Score and Prediction Error
4.5 The Confidence Score as an Indicator for a Model Update
4.6 Summary
5 Analytic Uncertainty Propagation in Neural Networks
5.1 Benefits and Limits of Uncertainty Forward Propagation
5.2 Analytic Uncertainty Propagation
5.2.1 Example - Uncertainty Propagation for Velocity
5.3 Uncertainty Propagation in Neural Networks
5.3.1 First to Second Layer
5.3.2 Second to Third Layer
5.3.3 Third to Fourth Layer
5.3.4 Generalization
5.3.5 Dimensionality
5.4 Application to Examples
5.4.1 Velocity Example
5.4.2 Eight Dimensional Synthetic Example
5.4.3 Boston Housing Data Set
5.5 Intrusive or Non-Intrusive?
5.6 Further Evaluation
5.6.1 Uncertainty as a Function of Training Epochs
5.6.2 Weight Approximation using analytical Uncertainty Propagation
5.6.3 Exact Calculation of Uncertainty induced by Approximated Weights
5.6.4 Application to Image Level Data
5.7 Summary
6 Building Reliably Accurate Models
6.1 Similarity to Design Space Exploration
6.2 The Concept
6.2.1 Iterative Approach
6.2.2 Generation of new Feature Vectors
6.2.3 Model Validation and Exit Criterion
6.3 Partitioned Model Evaluation
Model Performance Metric
1st Order Density - Gaps in Single Feature Distributions
2nd Order Density - Uniformities in Bivariate Feature Distributions
Scores for the Regions
6.3.1 Dimensionality Reduction and Co-Variance Exploitation
6.4 Model Build
6.5 Integration of Domain Knowledge
6.6 Application in Semiconductor Manufacturing
6.6.1 Technology Platform, Data Generation and Model Build
6.6.2 Model Application
6.7 Summary
7 Estimating Process Variations using Digital Twins
7.1 Variability Analysis and the Understanding of Process Variations
7.2 Extracting Information from the Shape of Point Clouds
7.2.1 Measuring the Point Cloud Overlap
7.2.2 Finding the Process Variability
7.3 Testing the Idea
7.4 Application to Real-World Data
7.5 Summary
8 Application to a Physics Informed Digital Twin
8.1 Western Electric Rules
8.2 Uncertainty Estimation
8.3 The Digital Twin with Reliability and Uncertainty
8.4 A Danger of Synthetic/Simulation Data
8.5 Summary
9 Conclusion and Outlook
9.1 Summary and Conclusion
9.2 Beyond Classical Regression Problems
9.3 Outlook