Suchen und Finden
Service
Computer Simulation Validation - Fundamental Concepts, Methodological Frameworks, and Philosophical Perspectives
Claus Beisbart, Nicole J. Saam
Verlag Springer-Verlag, 2019
ISBN 9783319707662 , 1056 Seiten
Format PDF, OL
Kopierschutz Wasserzeichen
Geräte
Mehr zum Inhalt
Computer Simulation Validation - Fundamental Concepts, Methodological Frameworks, and Philosophical Perspectives
Preface
6
Contents
8
Contributors
12
1 Introduction: Computer Simulation Validation
15
1.1 Introduction
15
1.2 Goals and Readership of this Handbook
17
1.3 Structure and Topics
19
1.3.1 Foundations (Parts I–II)
20
1.3.2 Methodology (Parts III–VI)
24
1.3.3 Validation at Work—Best Practice Examples (Part VII)
31
1.3.4 Challenges in Simulation Model Validation (Part VIII)
34
1.3.5 Reflecting on Simulation Validation: Philosophical Perspectives and Discussion Points (Part IX)
37
1.4 Outlook
41
References
43
Foundations—Basic Conceptions in Simulation Model Validation
46
2 What is Validation of Computer Simulations? Toward a Clarification of the Concept of Validation and of Related Notions
47
2.1 Introduction
48
2.2 Preliminaries
49
2.3 Influential Definitions of Validating Computer Simulations
52
2.4 Discussion of the Definitions
53
2.4.1 Commonalities
53
2.4.2 Difference 1: The Object of the Validation
54
2.4.3 Difference 2: The Standard of Evaluation
58
2.4.4 Difference 3: Type of Evaluation
68
2.4.5 Difference 4: Cogency (Degree of Credibility)
69
2.4.6 Difference 5: Empirical Methodology
71
2.5 Conclusions
76
References
77
3 Simulation Accuracy, Uncertainty, and Predictive Capability: A Physical Sciences Perspective
80
3.1 Introduction
81
3.2 Foundational Issues in Simulation Credibility
83
3.3 Verification Activities
86
3.3.1 Code Verification
86
3.3.2 Solution Verification
88
3.4 Validation, Calibration, and Prediction
90
3.4.1 Model Validation
90
3.4.2 Calibration and Predictive Capability
97
3.5 Concluding Remarks
104
References
105
4 Verification and Validation Principles from a Systems Perspective
109
4.1 Introduction
109
4.2 Approaches to Verification
114
4.3 Approaches to Validation
118
4.3.1 Quantitative Approaches to Validation
120
4.3.2 Qualitative Methods: Face Validation Approaches
122
4.3.3 Validation of Library Sub-models and Generic Models
123
4.4 Acceptance or Upgrading of Simulation Models
124
4.5 Discussion
125
References
127
5 Errors and Uncertainties: Their Sources and Treatment
129
5.1 Introduction
129
5.2 Verification-Related Errors/Uncertainties
131
5.2.1 Discrete Algorithm Choice and Software Programming
132
5.2.2 Numerical Approximation Errors
134
5.2.3 Conversion of Numerical Errors into Uncertainties
137
5.2.4 Estimating Total Numerical Uncertainty
138
5.3 Validation-Related Errors/Uncertainties
139
5.3.1 Experimental Measurement
139
5.3.2 Model Validation
140
5.3.3 Model Calibration
141
5.3.4 Extrapolation
142
5.4 Uncertainty Propagation-Related Uncertainties
143
5.4.1 Model Inputs
144
5.4.2 Model Parameters (i.e.., Parametric Uncertainty)
145
5.5 Total Prediction Uncertainty
146
5.6 Discussion
148
5.7 Conclusions
149
References
149
Foundations—Validation as a Scientific Method: Philosophical Frameworks for Thinking about Validation
152
6 Invalidation of Models and Fitness-for-Purpose: A Rejectionist Approach
153
6.1 Setting the Scene for Model Evaluation
154
6.2 The Falsification Framework of Karl Popper
156
6.3 Simulation Models, Invalidation and Falsification
158
6.4 Fitness-for-Purpose, Verisimilitude and Likelihood
160
6.5 If All Models May Be False, When Can They Be Considered Useful?
162
6.6 Defining Fitness-for-Purpose and Model Invalidation
165
6.6.1 Using Bayes Ratios to Differentiate Between Models
168
6.6.2 Use of Implausibility Measures to Differentiate Between Models
169
6.6.3 Use of Limits of Acceptability to Define Behavioural Models
170
6.7 Epistemic Uncertainties and Model Invalidation
171
6.8 The Model Advocacy Problem
172
6.9 Conclusions
174
References
175
7 Simulation Validation from a Bayesian Perspective
180
7.1 Introduction
180
7.2 The Fundamentals of Bayesian Epistemology
182
7.2.1 Basic Tenets of Bayesian Epistemology
182
7.2.2 A Brief Discussion of Bayesian Epistemology
187
7.3 Bayesian Epistemology and the Validation of Computer Simulations
189
7.3.1 Data-Driven Validation
190
7.3.2 The Problem of the Priors in Validation
194
7.4 Discussion
199
7.5 Conclusions
205
References
205
8 Validation of Computer Simulations from a Kuhnian Perspective
209
8.1 Introduction
210
8.2 Kuhn's Philosophy of Science
211
8.3 A Revolution, but not a Kuhnian Revolution: Computer Simulations in Science
214
8.4 Validation of Simulations from a Kuhnian Perspective
215
8.4.1 Do Computer Simulations Require a New Paradigm of Validation?
216
8.4.2 Validation of Simulations and the Duhem–Quine Thesis
219
8.4.3 Validation of Social Simulations
221
8.5 Summary and Conclusions
227
References
228
9 Understanding Simulation Validation—The Hermeneutic Perspective
231
9.1 Introduction
231
9.2 Hermeneutics in Versus Hermeneutics of Validation
233
9.3 Hermeneutics in Validation
234
9.3.1 Hermeneutics According to Kleindorfer, O’Neill and Ganeshan
235
9.3.2 A Reply to Kleindorfer, O’Neill and Ganeshan
237
9.3.3 Claim C-Open—A Second View
243
9.4 Hermeneutics of Validation
243
9.4.1 The Requirement of a Hermeneutic Situation
244
9.4.2 Hermeneutic Naiveté Versus Hermeneutic Consciousness
245
9.4.3 Interdisciplinary Dialogue
245
9.4.4 The Hermeneutic Tasks
247
9.5 Discussion
249
9.6 Conclusions
250
References
251
Methodology—Preparatory Steps
253
10 Assessing the Credibility of Conceptual Models
254
10.1 Introduction
255
10.2 Simulation-Based Knowledge, Verification, Validity, and the Function of Models
256
10.3 Taking the Notion of “Credibility” Seriously
261
10.4 The Credibility of Models: Lessons from Scientific Practice
263
10.5 Empirical Fit and Causal Understanding
266
10.6 Models and the Exploration of Credible Worlds
269
10.7 Summary
272
References
273
11 The Foundations of Verification in Modeling and Simulation
275
11.1 Verification in Modeling and Simulation
275
11.2 Code Verification
277
11.3 Types of Code Verification Problems and Associated Benchmarks
281
11.4 Solution Verification
285
11.5 Solution Verification for Complex Problems
289
11.6 Conclusion and Prospectus
291
References
296
12 The Method of Manufactured Solutions for Code Verification
298
12.1 Introduction
298
12.2 Broad Description of MMS
300
12.3 Three Example Problems in MMS
301
12.3.1 Example 1
301
12.3.2 Example 2
302
12.3.3 Example 3
303
12.3.4 Complex Problems
304
12.4 Application to Code Verification
304
12.5 Features and Examples of MMS Code Verification
307
12.5.1 Radiation Transport Codes
307
12.5.2 Nonhomogeneous and Nonlinear Boundary Conditions
308
12.5.3 Shocks, Partitioning, and “Glass-Box” Verification
308
12.5.4 Shocks, Multiphase Flows, and Discontinuous Properties
309
12.5.5 Verification of Boundary Conditions
310
12.5.6 Unsteady Flows and Divergence-Free MMS
310
12.5.7 Variable Density Flows; Combustion
311
12.6 Attributes of MMS Code Verification
311
12.6.1 Two Multidimensional Aspects
311
12.6.2 Blind Study
311
12.6.3 Burden of MMS and Option Combinations
312
12.6.4 Code Verification for Commercial Codes
312
12.6.5 Code Verification with a Strong Completion Point
313
12.6.6 Proof?
313
12.6.7 Mere Mathematics
314
12.6.8 Irrelevance of Solution Realism to Code Verification
315
12.7 Reasons for Solution Realism in MMS
315
12.7.1 Realistic MMS in Code Verification of Glacial Ice Flow Modeling
316
12.7.2 Realistic MMS in Solution Verifications and Turbulence Models
316
12.7.3 Realistic MMS in Singularity Studies
317
12.7.4 Other Uses and Generation Methods for Realistic MMS
317
12.8 Alternative Formulations and General References for MMS
318
12.9 Conclusion
318
References
319
13 Validation Metrics: A Case for Pattern-Based Methods
322
13.1 Introduction
322
13.2 Validation Metrics
324
13.2.1 Four Types of Measurement Scales
324
13.2.2 The Desirable Properties of a Validation Metric
325
13.3 Four Families of Validation Measures
326
13.3.1 Empirical Likelihood Measures
326
13.3.2 Stochastic Area Measures
327
13.3.3 Pattern-Based Measures I: Information-Theoretic Measures
328
13.3.4 Pattern-Based Measures II: Strategic State Measures
328
13.4 Measures of Closeness or of Information Loss
329
13.4.1 Kullback–Leibler Information Loss
329
13.4.2 The Generalized Subtracted L divergence (GSL-div)
330
13.5 The Example: Models and Data
331
13.6 The State Similarity Measure (SSM)
332
13.6.1 Results for the Models
333
13.6.2 Monte Carlo Simulations of the SSM
333
13.7 Classical Possibility Theory
334
13.7.1 The Generalized Hartley Measure (GHM) for Graded Possibilities
335
13.7.2 Applying U-Uncertainty to Our Data
336
13.8 Comparing the Distances Measured by SSM and GHM
338
13.9 Conclusions
339
References
340
14 Analysing Output from Stochastic Computer Simulations: An Overview
342
14.1 Introduction
342
14.2 Preliminaries
344
14.2.1 Definitions
344
14.2.2 Background Statistical Knowledge
344
14.2.3 Setting Up the Problem
346
14.3 Working with Terminating Simulations
346
14.4 Working with Non-terminating Simulations
347
14.4.1 Welch's Method
349
14.4.2 MSER-5
351
14.5 How Many Replications?
352
14.6 Making Comparisons
353
14.6.1 Comparing Two Systems
353
14.6.2 Comparing Many Systems
355
14.7 Conclusion
355
References
356
Methodology—Points of Reference and Related Techniques
357
15 The Use of Experimental Data in Simulation Model Validation
358
15.1 Introduction
358
15.2 Data Sets for Model Development and Testing
361
15.3 Comparison Methods for Model and Target System Data Sets
362
15.3.1 Graphical Methods for System and Model Data Comparisons
362
15.3.2 Some Quantitative Measures for System and Model Comparisons in the Time Domain
363
15.3.3 Frequency-Domain Measures and Comparisons
364
15.4 System Identification and Parameter Estimation in Model Validation
365
15.4.1 A Brief Overview of System Identification and Parameter Estimation
365
15.4.2 Issues of Identifiability
367
15.4.3 Applications of System Identification and Parameter Estimation to the Processes of Validation
369
15.5 Design of Experiments and Selection of Inputs for Model Testing
371
15.6 Model Structure Optimisation
373
15.7 Experimental Data for Validation: A Physiological Modelling Example
373
15.7.1 Experimental Constraints
377
15.7.2 Experimental Design and Test Signal
378
15.8 Discussion
380
15.9 Conclusions
381
References
382
16 How to Use and Derive Stylized Facts for Validating Simulation Models
384
16.1 Introduction
384
16.2 Epistemological Foundations of Stylized Facts
386
16.2.1 Development and Definition of the Stylized Facts Concept
386
16.2.2 Using Stylized Facts for Simulation Model Validation
388
16.3 Existing Approaches to Establish Stylized Facts
392
16.4 An Alternative Process to Derive Stylized Facts
396
16.5 Conclusion and Outlook
401
References
402
17 The Users’ Judgements—The Stakeholder Approach to Simulation Validation
405
17.1 Introduction
405
17.2 Action Research and the Use of Simulation Models
407
17.2.1 Meta-Theoretical Foundations of Action Research
408
17.2.2 The Validity of Action Research Knowledge
409
17.2.3 The Use of Simulation Models in Action Research and the Subject Matter of Their Validation
410
17.3 The Logical Empiricist Versus the Post-positivist Understanding of Validity
412
17.3.1 The Logical Empiricist Understanding of Validity
412
17.3.2 The Need for a Post-positivist Understanding of Validity
413
17.4 A General Definition of Simulation Validity
414
17.4.1 Definition
415
17.4.2 Application to Socio-Ecological Simulation Models in Action Research
416
17.5 Validation Techniques Related to the Stakeholder’s Judgements
422
17.5.1 Qualitative Interviewing
422
17.5.2 Focus Groups
423
17.5.3 Role-Playing Games
424
17.5.4 Inappropriate Techniques and Related Consequences
425
17.6 Discussion
425
17.7 Conclusions
428
17.8 Outlook
428
References
429
18 Validation Benchmarks and Related Metrics
432
18.1 Introduction
432
18.2 The Concept of Validation Benchmarks
434
18.2.1 Defining Validation Benchmarks
434
18.2.2 Motivations for Using Validation Benchmarks
435
18.2.3 Sources of Benchmarks
436
18.3 The Benchmarking Process
438
18.3.1 Types of Benchmarking
438
18.3.2 Criteria of Benchmark Selection
441
18.4 A Typology of Validation Benchmarks
443
18.4.1 Strong-Sense Benchmarks
443
18.4.2 Standard Benchmarks
445
18.4.3 Yardstick Benchmarks
446
18.4.4 Touchstone Benchmarks
446
18.5 Metrics Related to Benchmarking
447
18.5.1 Basic Concepts
448
18.5.2 Measures of Accuracy
449
18.5.3 Skill Scores
451
18.5.4 Murphy–Winkler Framework and Beyond
452
18.5.5 Holistic Measurement
452
18.6 Discussion
453
18.6.1 Normalizing Simulation Validation
453
18.6.2 The Social Character of Validation Benchmarks
453
18.6.3 Between Validation and Comparison—the Limitations of Benchmarking
455
18.6.4 The Price of Efficient Benchmarks
456
18.6.5 The Devaluation of Benchmarks Proper
456
18.7 Conclusions
457
References
459
Methodology—Mathematical Frameworks and Related Techniques
461
19 Testing Simulation Models Using Frequentist Statistics
462
19.1 Introduction
462
19.2 Frequentist Statistics
464
19.2.1 Important Background
464
19.2.2 Estimation
467
19.2.3 Models of Dependence
468
19.2.4 Null Hypothesis Significance Tests
468
19.3 Statistical Model Validation: Why and How?
471
19.3.1 Why Validate?
472
19.3.2 Estimating Goodness of Fit
472
19.3.3 Testing Goodness of Fit
473
19.3.4 Tests for Splitting and Tests for Lumping
474
19.3.5 Conceptual Entry Point: TOST
476
19.3.6 A Uniformly Most Powerful Invariant Test
478
19.3.7 More Descriptive: Test of Fidelity
478
19.3.8 Statistical Validation Overview
479
19.4 Examples
481
19.4.1 Fitness for Purpose
481
19.4.2 Validation of a Theoretical Model
484
19.5 Discussion
487
19.5.1 Generalizations
488
19.5.2 Significant and Important?
489
19.5.3 Nuisance Parameters
489
19.5.4 Bayesian or Frequentist Approach?
489
19.5.5 Conclusion
491
References
491
20 Validation Using Bayesian Methods
494
20.1 Introduction
494
20.2 Fundamentals
497
20.3 Bayesian Decision Rule
499
20.4 Bayesian Univariate Hypothesis Testing
501
20.5 Multivariate Bayesian Hypothesis Testing
502
20.6 A Bayesian Measure of Evidence
503
20.7 Bayes Network
505
20.8 Non-normal Data Transformation
506
20.9 Bayesian Model Validation Process
507
20.10 Numerical Application
510
20.10.1 Example 1: Bayesian Decision Rule
510
20.10.2 Example 2: Univariate Model Validation
514
20.10.3 Example 3: Multivariate Model Validation
516
20.11 Concluding Remarks
519
References
519
21 Imprecise Probabilities
522
21.1 Introduction
522
21.2 Basics
523
21.3 Examples
525
21.3.1 Unknown Parameters
525
21.3.2 The Challenge Problems
526
21.3.3 Nonprobabilistic Odds
527
21.4 Interpretations
528
21.4.1 One-Sided Betting
528
21.4.2 Indeterminate Belief
529
21.4.3 Robustness Analysis
530
21.4.4 Evidence Theory
530
21.5 Problems
531
21.5.1 Updating
531
21.5.2 Decision-Making
532
21.6 Validation and IP
533
21.6.1 Interpretations
534
21.6.2 Problems
534
21.7 Conclusion
535
References
535
22 Objective Uncertainty Quantification
538
22.1 Introduction
538
22.2 Gene Regulatory Networks
541
22.3 Optimal Operators
544
22.4 Optimal Intervention in Regulatory Networks
545
22.5 Intrinsically Bayesian Robust Operators
547
22.6 IBR Intervention in Regulatory Networks
550
22.7 Objective Cost of Uncertainty
550
22.8 Optimal Experimental Design for Regulatory Networks
552
22.9 Discussion
554
22.10 Conclusion
555
References
556
Methodology—The Organization and Management of Simulation Validation
558
23 Standards for Evaluation of Atmospheric Models in Environmental Meteorology
559
23.1 Introduction
560
23.2 Definitions Used
561
23.2.1 Specifics of an Atmospheric Model
561
23.2.2 Modeling
562
23.2.3 Guideline
563
23.2.4 Standard
563
23.2.5 Verification
563
23.2.6 Validation
564
23.2.7 Evaluation
564
23.2.8 Model Quality Indicator
565
23.2.9 Reference Data
567
23.3 From Guidelines to Standards
567
23.3.1 Historical Background
567
23.3.2 How to Achieve a Standard
569
23.4 Generic Structure of an Evaluation Guideline
570
23.4.1 Specification of Application Area
571
23.4.2 Evaluation Steps to be Performed by the Model Developer
571
23.4.3 Evaluation Steps to be Performed by the Model User
573
23.5 Examples for Standards
574
23.5.1 Comparing Application Areas of Two Standards
574
23.5.2 Detailed Specification of an Application Area
575
23.5.3 Some Detailed Evaluation Steps to be Performed by the Model Developer
576
23.5.4 Some Detailed Evaluation Steps to be Performed by the Model User
578
23.6 Conclusions
579
References
580
24 The Management of Simulation Validation
583
24.1 Introduction
583
24.2 Simulation Terminology
585
24.3 Principles of Simulation Validation
586
24.4 Management of Simulation V&V: A Framework
588
24.5 Process-Oriented Simulation V&V Management
590
24.5.1 Simulation Validation Steps
591
24.5.2 Simulation Verification Steps
593
24.6 Draw up an Optimized V&V Scheme
594
24.7 Quantify Simulation V&V Results
596
24.8 Computer Aided Management of Simulation V&V
597
24.8.1 Management Platform
597
24.8.2 Other Validation Tools
598
24.9 Discussion
599
24.10 Conclusions
599
References
600
25 Valid and Reproducible Simulation Studies—Making It Explicit
603
25.1 Introduction
603
25.2 Example: A Model of the Decision to Migrate
606
25.3 Managing the Model: Domain-Specific Modeling Languages
608
25.4 Managing an Experiment: Experiment Specification Languages
611
25.5 Managing a Simulation Study: Provenance Models
613
25.6 Discussion
619
25.7 Conclusion
619
References
620
Validation at Work—Best Practice-Examples
624
26 Validation of Particle Physics Simulation
625
26.1 Introduction
625
26.2 What Particle Physics is About: Example LHC
626
26.2.1 The Status of the Standard Model
626
26.2.2 The Forefront Experiment: LHC
627
26.3 Data Analysis and the Use of Simulations
628
26.3.1 From Data to Physics
628
26.3.2 The Role of Simulation for Data Analysis
629
26.4 Modeling the LHC Processes
630
26.4.1 The Matrix Element of the Hard Collision
631
26.4.2 Parton Distribution Functions: Dressing the Initial State
632
26.4.3 Dressing of the Outgoing Partons
632
26.5 Detector Simulation
633
26.6 Principles of Validation and Uncertainties
636
26.6.1 Factorization of Migration
637
26.6.2 Is Factorization Correct?
638
26.7 General Procedures of Validation in Particle Physics
638
26.8 Validation of the Physics Generators
639
26.8.1 pdfs
640
26.8.2 Pile-up in pp Scattering
642
26.9 Validation of Detector Simulation
643
26.9.1 Testing the Detector Geometry
643
26.9.2 Validation of Electron Simulation
644
26.10 How Simulation is Applied in Data Analysis
646
26.10.1 Measurement of the Higgs Cross Section
646
26.10.2 Search for a Stop Quark
647
26.11 Discussion
650
26.12 Summary and Conclusion
651
References
652
27 Validation in Fluid Dynamics and Related Fields
655
27.1 Fluid Dynamics and Related Fields
655
27.1.1 Weak Models, Strong Models, and RANS Turbulence Models
656
27.2 Separation of Verification and Validation
657
27.3 Errors and Uncertainties
657
27.4 Validation—What Does It Mean?
658
27.4.1 Issue #1. Acceptability (Pass/Fail) Criteria
659
27.4.2 Issue #2. Necessity for Experimental Data
660
27.4.3 Issue #3. Intended Use
660
27.4.4 Issue #4. The Prediction Issue
661
27.5 Validation Methodology Based on ASME V&V 20-2009
661
27.5.1 ASME V&V 20-2009 Background, Motivation, and Philosophy
662
27.5.2 Validation Metrics
662
27.5.3 Defining Validation Uncertainty Uval
663
27.5.4 Estimating Validation Uncertainty
664
27.5.5 Interpretation of Validation Results and Caveats
665
27.5.6 Observations
667
27.5.7 Importance of Case 2
668
27.5.8 Model Quality Versus Validation Quality
668
27.5.9 Forthcoming Addenda to V&V 20-2009
669
27.6 Model Form Errors Versus Parameter Errors
669
27.7 Model Form Uncertainty and Probability Distribution Functions
670
27.8 Weakest Link in Validation Practice
671
27.9 New Paradigm of Experiments Designed Specifically for Validation
672
27.10 Unrealistic Expectations Placed on Experimentalists
672
27.11 Can Models be Validated? A Discussion of Falsificationism Versus Validation
673
27.11.1 Truth Versus Accuracy
674
27.11.2 Summary of Falsificationism Versus Validation
675
References
676
28 Astrophysical Validation
678
28.1 Introduction
678
28.2 Approach to Verification and Validation
679
28.3 Simulation Instruments
681
28.3.1 The Flash Code
682
28.3.2 The Postprocessing Toolkit
684
28.3.3 Simulating Reactive Flow
684
28.4 Validation Examples
685
28.4.1 Overview of Flash Problems
686
28.4.2 Shocks and Fluid Instabilities
687
28.4.3 Computation of Reaction Products in Large Eddy Simulations Of Supernovae
693
28.5 Discussion
697
28.6 Conclusions
699
References
699
29 Validation in Weather Forecasting
703
29.1 Introduction
703
29.2 Setting the Scene
705
29.2.1 The Atmospheric Model: State of the Art
705
29.2.2 Intended Use of the Simulation Output
709
29.3 Validation Concepts
710
29.3.1 Idealized Tests for the Verification of the Dynamical Core
711
29.3.2 Validation of Parameterizations
717
29.3.3 Comparison to Observations
718
29.4 Uncertainty Estimation via Ensemble Forecasting
723
29.5 Discussion and Summary
724
References
726
30 Validation of Climate Models: An Essential Practice
729
30.1 Introduction
729
30.2 Climate Model Validation: Emergence of Definition and Community Practice
731
30.3 Definition of Terms
734
30.4 Model Construction, Observations, Assimilation: Roles in Validation
737
30.5 Validation of Climate Models in Practice
739
30.5.1 Independence, Transparency, and Objectivity: Basic Values of Verification and Validation
740
30.5.2 Identification of Independent Observational Data
741
30.5.3 Deliberative Validation and Expert Judgment
742
30.5.4 Quantitative Evaluation
745
30.6 Discussion
750
30.7 Conclusion
751
References
752
31 Validation of Agent-Based Models in Economics and Finance
755
31.1 Introduction
756
31.2 Agent-Based Computational Economics: Common Practices
756
31.2.1 The Development of a Typical Agent-Based Model
757
31.2.2 Inputs of Agent-Based Models
759
31.2.3 Outputs of Agent-Based Models
760
31.2.4 Relation Between Input and Output
761
31.3 Agent-Based Model Validation: Theoretical Framework
761
31.4 Agent-Based Model Validation: Literature Review
763
31.4.1 Calibration and Estimation
765
31.4.2 Validation
767
31.5 A New Wave of Validation Approaches
768
31.5.1 Validation As Replication of Time Series Dynamics
769
31.5.2 Validation as Matching of Causation
770
31.5.3 Global Sensitivity Analysis via Kriging Meta-Modeling
771
31.5.4 Parameter Space Exploration and Calibration via Machine-Learning Surrogates
772
31.6 Conclusions
773
References
774
Challenges in Simulation Model Validation
780
32 Validation and Equifinality
781
32.1 Introduction
781
32.2 The Origins of Equifinality Concepts
783
32.3 Equifinality as an Empirical Result
783
32.4 Equifinality in Model Calibration in the Inexact Sciences
787
32.5 Equifinality as Behavioural Model Ensembles
788
32.6 Defining a Model Likelihood
791
32.7 Equifinality and Model Validation in the Inexact Sciences
794
32.8 Discussion
795
References
796
33 Validation and Over-Parameterization—Experiences from Hydrological Modeling
800
33.1 Introduction
800
33.1.1 Over-Parameterization Terminology
801
33.1.2 Main Types of Hydrological Models
803
33.1.3 Peculiarities of Hydrological Models
806
33.2 Types of Validation in Hydrological Modeling
807
33.2.1 Validation Based on Independent Time Periods
807
33.2.2 Validation Based on Independent Catchments
808
33.2.3 Validation Based on Independent Variables
808
33.3 Conclusions—All Models Are Wrong, but Which Are Useful?
813
Textbox: Short Description of Catchment Hydrology
815
References
816
34 Uncertainty Quantification Using Multiple Models—Prospects and Challenges
824
34.1 Introduction
824
34.2 Challenges for Uncertainty Quantification in Climate Modeling
826
34.3 Uncertainty Quantification Using Model Ensembles
828
34.4 Problems with Model Democracy
830
34.5 Beyond Model Democracy
832
34.6 Illustration of Model Weighting for Arctic Sea Ice
834
34.7 Discussion and Open Issues
836
34.8 Conclusion
840
References
841
35 Challenges to Simulation Validation in the Social Sciences. A Critical Rationalist Perspective
845
35.1 Introduction
845
35.2 Illustrative Example: Models of Social Influence
849
35.3 Challenges to Model Validation
852
35.3.1 Obscure Concepts
852
35.3.2 Abundance of Latent Concepts
854
35.3.3 Representation of Time
856
35.3.4 Interplay of Multiple Processes
857
35.3.5 Context Characteristics Matters
859
35.4 Discussion
861
35.4.1 Compare Models and Identify Critical Assumptions!
861
35.4.2 Defend Your Assumptions!
862
35.4.3 Explore Model Scope and Its Boundaries!
863
35.4.4 More Validation!
863
References
864
36 Validation and the Uniqueness of Historical Events
868
36.1 A Brief History of Simulation’s Semantics
870
36.2 History
871
36.3 Challenges
873
36.4 Uses and Potentials of Simulations in History
877
36.4.1 Big-Data and Longue Durée History
877
36.4.2 Microhistorical Research and Simulation
878
36.4.3 Digital Games and Simulation Games
879
36.5 Conclusion and Outlook
880
References
882
Reflecting on Simulation Validation: Philosophical Perspectives and Discussion Points
885
37 What is a Computer Simulation and What does this Mean for Simulation Validation?
886
37.1 Introduction
887
37.2 Preliminaries
888
37.3 Computer Simulations and Experiments
890
37.3.1 Computer Simulations as Experiments
890
37.3.2 Computer Simulations as Modeled Experiments
892
37.4 Computer Simulations, Thought Experiments and Argumentation
895
37.5 Models and Simulations
900
37.6 Conclusions
904
References
906
38 How Do the Validations of Simulations and Experiments Compare?
909
38.1 Introduction
909
38.2 Epistemology and Methodology of Validation
911
38.2.1 The Concept of Validation
911
38.2.2 Epistemology and Methodology
916
38.3 Illustration: Validation of Experiments and Simulations in the Field of Evolution
920
38.4 Discussion and Conclusion
925
References
925
39 How Does Holism Challenge the Validation of Computer Simulation?
927
39.1 Introduction
927
39.2 Holism and Modularity—Two Counteracting Concepts
929
39.2.1 Modularity—The Rational Picture
929
39.2.2 Holism—A Multifaceted Challenge
932
39.3 The Challenge Arising from Parameterization and Tuning
933
39.4 The Challenge from Kluging
937
39.5 The Limits of Validation
941
References
943
40 What Types of Values Enter Simulation Validation and What Are Their Roles?
945
40.1 Introduction
945
40.2 The Framework
946
40.3 A Defense of Epistemic Values that Assess the Credibility of Simulation Results
949
40.4 Roles of Cognitive and Social Values in Assessing the Credibility of Simulation Results
951
40.4.1 Assistance in the Assessment of Performance in Terms of Epistemic Values
951
40.4.2 Determining Minimal Probabilities for Accepting or Rejecting a Hypothesis
952
40.5 Roles of Cognitive and Social Values in Assessments of the Usefulness of Simulation Models
954
40.5.1 Accounting for the Practicability of Simulation Models
955
40.5.2 Accounting for the Relevance of Simulation Models
956
40.6 Simulation Validation as a Multi-criteria Assessment
958
40.7 Summary and Conclusion
959
References
961
41 Calibration, Validation, and Confirmation
964
41.1 Introduction
964
41.2 Computer Simulations, and Calibration
965
41.2.1 Calibration, Verification, and Validation
965
41.2.2 Adequacy for Purpose
970
41.3 Predictivism
972
41.3.1 The Paradox of Predictivism
972
41.3.2 Bayesian Confirmation Theory V and the Problem of Old Evidence
974
41.3.3 Validation and Confirmation
977
41.4 The Problem of Old Evidence and Model Calibration
977
41.4.1 The Static Problem of Old Evidence
977
41.4.2 The Dynamic Problem of Old Evidence
978
41.4.3 An Argument for Predictivism
979
41.4.4 A Novel Bayesian Argument for Predictivism
981
41.5 Conclusion
983
Appendix
984
References
985
42 Should Validation and Verification be Separated Strictly?
988
42.1 Introduction
989
42.2 Preliminaries
990
42.2.1 Scientific Methods
990
42.2.2 Verification and Validation
991
42.3 The Distinction Between Verification and Validation
994
42.3.1 Verification as a Means of Validation of the Computational Model?
996
42.3.2 Verification as Means for Validation of the Conceptual Model?
999
42.4 Arguments Against a Clean Separation Between Verification and Validation
1003
42.4.1 The Separation Between Verification and Validation
1005
42.4.2 Verification and Mathematics
1008
42.5 Conclusions
1009
References
1010
43 The Multidimensional Epistemology of Computer Simulations: Novel Issues and the Need to Avoid the Drunkard’s Search Fallacy
1012
43.1 Introduction: Computer Simulations, a Revolutionary Epistemology?
1013
43.2 Methodological and Conceptual Preliminaries
1014
43.3 Dimensions of Computational Inquiries, or Where Things Can Go Wrong Epistemically
1017
43.3.1 The Production of Computational Results: Can We Control the Beast?
1018
43.3.2 The Reception and Post Hoc Assessment of Computational Results
1026
43.4 Should Epistemologists of Science Bother, After All?
1030
43.4.1 Target Models, Actually Investigated Models, and Failure
1030
43.4.2 The Valuable Redundancy Argument
1031
43.4.3 The Procrustean Objection
1031
43.4.4 The Absence of Data Argument and the Ostrich Strategy
1033
43.5 Conclusion and Moral
1034
References
1036
Index
1039
Shop
Kopierschutz
= Kopierschutzfrei
= Wasserzeichen
= DRM Kopierschutz
