Evaluation of uncertainty Simulation („virtual experiment ... · 3 3 The use of Monte Carlo...

1

NMIJ - BIPM workshop on theImpact of Information Technology in Metrology

18 -20 May 2005, AIST Tsukuba, Japan

Forward calculation

Cause EffectCause Effect

Inverse problem

Model

Measurand ObservablesMeasurand Observables

SampleSampleGas target

xx

zz

D(d,n) X(n,n')X(n,n')

DetectorDetector

ϑn,n'ϑn,n'

Simulation („virtual experiment“) 0 0

5 5

1010

1515kgkg

0 0

5 5

1010

1515kgkg

0 0

5 5

1010

1515kgkg

Measurement 0 0

5 5

1010

1515kgkg

MeasurementMM

M MM

Evaluation of uncertainty

The use of Monte Carlo techniques for metrology- evaluation of uncertainties and simulation of experiments –

Bernd R.L. Siebert*, Dieter Sibold* and Klaus-Dieter Sommer+

* Physikalisch-Technische Bundesanstalt in Braunschweig und Berlin (PTB)+ Landesamt für Mess- und Eichwesen Thüringen (LMET)

The presentation does not discuss new results, but is to demonstrate thato Monte Carlo integration allows to treat any physically sound

model for the evaluation of uncertainty and thereby providesadded value to the Guide on the Expression of Uncertainty inMeasurement (GUM) as it is not restricted to linear models and

o Monte Carlo simulation is a versatile tool to analyse and evaluatecomplex measurements.

MC integration and MC simulation are not generally used terms. However, it can be shown that uncertainty evaluation using given models according to GUM (Guide for the expression of uncertainty in measurement) can be transformed to a simple integration problem whereas “virtual” experiments often require a “translation” of the model into a step-by-step procedure, especially so if the underlying physics is described by (a system of) integral equations. The term “simulate” means then to “mock-up” cause-and-effect chains by a chain of physical interactions. Here, the model itself is also under study.Two simple concurrent examples serve to demonstrate this.

2

2The use of Monte Carlo techniques for metrology

- evaluation of uncertainties and simulation of experiments -

Content

IntroductionCause-and-Effect Chain and Model, Integration ↔ SimulationDeterministic and Stochastic Models

1 Summary and Conclusions5 Evaluation of UncertaintyGUM concept, Markov Formula & Standard Procedure

3 Monte CarloIntroduction, Random numbers, Sampling, Sorting & „Statistics“

24 Simulation („virtual experiment“)Model, Inference, “Validation” & Uncertainty

Main message:Inference from given data (effects) to possible causesis a major task of metrology,Monte Carlo techniques provideversatile, powerful and easy to handle toolsto that end. Introduction

Cause-and-Effect Chain and Model, Integration ↔ SimulationDeterministic and Stochastic Models

1 IntroductionCause-and-Effect Chain and Model, Integration ↔ SimulationDeterministic and Stochastic Models

11

Summary and Conclusions5 Summary and Conclusions55

Evaluation of UncertaintyGUM concept, Markov Formula & Standard Procedure

3 Evaluation of UncertaintyGUM concept, Markov Formula & Standard Procedure

33

Monte CarloIntroduction, Random numbers, Sampling, Sorting & „Statistics“


22



Complete result of ameasurement (calculation):y & u(y) or y & U.Model is given.

Interprets, supplements oreven replaces an experiment.Model is under study!

Content

IntroductionCause-and-Effect Chain and Model, Integration ↔ SimulationDeterministic and Stochastic Models


11 Summary and Conclusions5 Summary and Conclusions55 Evaluation of UncertaintyGUM concept, Markov Formula & Standard Procedure






Main message:Inference from given data (effects) to possible causesis a major task of metrology,Monte Carlo techniques provideversatile, powerful and easy to handle toolsto that end. Introduction

Cause-and-Effect Chain and Model, Integration ↔ SimulationDeterministic and Stochastic Models


11

Summary and Conclusions5 Summary and Conclusions55

Evaluation of UncertaintyGUM concept, Markov Formula & Standard Procedure


33

Monte CarloIntroduction, Random numbers, Sampling, Sorting & „Statistics“


22



Complete result of ameasurement (calculation):y & u(y) or y & U.Model is given.

Interprets, supplements oreven replaces an experiment.Model is under study!

We use two concurrent examples:o Measurement of a mass with or calibration of a spring scale to

demonstrate Monte Carlo integration ando Determination of a differential neutron cross section with a

time-of-flight experiment to demonstrate Monte Carlo simulation.

Clearly, simple spring scales are not used much in practice, but theyare ideal for demonstration, as they require, although simplified, tomodel effects that are of importance with many types of

measurements.

The time-of-flight experiment is still running at PTB. The material used here serves for demonstration only and it is taken from a MonteCarlo code developed some years ago (1975-1995) for theanalysis of this experiment. This problem was selected since itexemplifies an evaluation of a measurement that does require a very detailed simulation of a long cause-and-effect (Markov) chains. References see bottom of viewgraph 19.

PDF-Dateiöffnen

We use two concurrent examples:o Measurement of a mass with or calibration of a spring scale todemonstrate Monte Carlo integration and

o Determination of a differential neutron cross section with atime-of-flight experiment to demonstrate Monte Carlo simulation.

Clearly, simple spring scales are not used much in practice, but theyare ideal for demonstration, as they require, although simplified, tomodel effects that are of importance with many types of measurements.

The time-of-flight experiment is still running at PTB. The material used here serves for demonstration only and it is taken from a MonteCarlo code developed some years ago (1975-1995) for theanalysis of this experiment. This problem was selected since itexemplifies an evaluation of a measurement that does require a very detailed simulation of a long cause-and-effect (Markov) chains.

3



Introduction Cause-and-Effect Chain and Model (1/4)

1

Cause Effect

Measurand Observables

Yes, but

0 0

5 5

1010

1515kgkgM

0 0

5 5

1010

1515kgkg

0 0

5 5

1010

1515kgkg

MeasurementM M

Can one infer the valuem of the mass M andu(m) from mIND= 5 kg?

one needs a model,to link the measurandand the observed data


xx

zz


DetectorDetector

ϑn,n'ϑn,n'

Cyc

lotro

n:Em

ittan

ce

Time of flightspectra

Can one infer thedifferential crosssection value σ (ϑn,n’)from TOF-spectra?

Inference requires a model

We use the term model for a description of a cause action chain asconstrued for a defined purpose.

The purpose is to predict a consequent (posterior) state Xpost

if one is given the antecedent (prior) state Xante.A model is not “true” but “reasonable” or “believable”. The verb “construe” is to mean: deduce by inference from given information, i.e. data in the widest meaning. For instance, the fact that many physical relations are locally linear is a datum. Basis for construing a model can be a theory or a phenomenological relation, validated by experiments.

NOTE 1: A given effect (observed data) can result from different causes, e.g. thelength of a spring is affected by an applied force or a change intemperature. A cause, e.g. “a neutron passes through matter”, can resultin a distribution of effects, here the path length to the location of thenext interaction, that may be interpreted as PDF.

NOTE 2: Occam's razor* should be used for guidance, e.g. separate linear modelsare sufficient for small changes of Xante , i.e. ∆F and ∆T.*Here: The simplest explanation that fits all data given is the best one.

4



Introduction Integration ↔ Simulation (2/4)

1

A given effect canresult from differentcauses,

Inference from effect is an an inverse problem that may be ill-posed.

Cause Effect


Cause Effect


Cause Effect


A model allows a forward calculation from cause to effect.

e.g. a decrease in temperatureor an increase of exerted forcedecrease the length of a spring.

A given cause canresult in differentobservable effects,

e.g. a neutron in matter:the path length between tworeactions is distributed.

Forward calculation

Inverse problem

Model

The model for the evaluation of uncertainty allows to infer the value m of themass M and u(m) from mIND= 5 kg via forward calculation, here integration.

The model for the simulation of the time-of-flight experiment (TOF) allows toinfer σ (ϑn,n’) from TOF-spectra via forward calculation, here simulation.

The term model is here used as introduced on viewgraph 3. However,a cause-and-effect chain may be very long. In this case we use sub-models to describe effects due to “separable” causes. This separation is helpful for the later computation, but one encounters two problems:o generally, it is impossible to identify all possible causes ando the description of the identified cause-and-effect chain is often only

an approximation, either for lack of knowledge or due to mathe-matical or computational problems in formulating them exactly.

See also the citation on viewgraph 6 on deterministic and stochastic models.For more clarity, we use the termo Model for evaluation of uncertainty for the relation that allows to

infer from a given set of values of the input quantities determini-stically the value of the output quantity (or values of the output quantities) (this leads to an integration problem) and

o Model for simulation for the set of steps or procedures and theparameters used to realise a Markov chain for the entity understudy, here, in the concurrent example, deuterons and neutrons.NOTE: Not all simulation problems lead to Markov chains. For example, the

use of point detectors in transport calculations may violate Markovconditions if used not only for immediate tallying.

5



0 0

5 5

1010

1515kgkg

M

Introduction Deterministic model (3/4)

Example: change of the length of a spring:

lspring lengthXpost

∆Traise temperature

∆Fapply force

l0spring lengthXante

( )Tll T∆10 α+=

( )Fll F∆10 α+= ( )( )TFll TF ∆1∆10 αα ++=

Simplification for small values of

∆F and ∆T !

1

Basis: Hooke‘s law and temperature expansion

( )TllT,lT ∂

∂= −1

00α( )FllF,lF ∂

∂= −1

00α

The model allows to “propagate” the PDFs for the force F and the temperature T. Example: The information on temperature can be simple, e.g. one knows that the values obey the relation tmin < t <tmax → via PME gT(t) rectangular. Alas, it may also be appropriate to assign a U-shaped PDF to account for sinusoidal control.

-continued form notices on bottom of page 4 –Aristotle, if he lived today, might say:The purpose of the model is its causa finalis for making it and the model itself is an algorithms that reflects our knowledge of the causa efficiensunder study. He might add that thoughts are the causa materialis, and he might identify the algorithm as causa formalis.----------------------------------------------------------------------------------Although Hooke’s law and temperature expansion can be understood microscopically, they are in practice merely used as phenomenological relation with experimentally determined values of parameters, usually termed “coefficients”, i.e. thermal length (or volume) expansion coefficient.This is representative of a widely used modelling technique: From experiments, or, if available, preferentially from theories, one expects a relation and determines parameters that cannot be computed (either not at all or with undue expenditure only) by measurements.On viewgraph 15 is a more correct model for treating the effect of exerting a force. However, this model is already so “complex” that approximations are needed to obtain solutions with “justifiable expenditure. However, what is “justified” is determined be the aimed at smallest uncertainty contribution.

6



Introduction Stochastic model (4/4)

1

One obtains a PDF for path length to next reaction:

( ) ( )[ ] ( )mattern

matternexp

ntot

ntot

,,E,,E-

g X ΣξΣ

ξ =

Example: path length between two reactions of a neutron in matter:Basis: Attenuation of neutron fluence in matter

( ) ( ) ( )( )[ ]0ntotn0n matternexpnn ξξΣξΦξΦ −= ,,E-,E,,E,

n: spin, magnetic field,... matter: nuclei, temperature,

cross sections ...

internal states ?

Whether or not a stochastic model could be transformed to a deterministic oneif one were to know all internal states or so-called hidden variables may be disputed,in practice, however, the information given corresponds to incomplete knowledge!

Therefore, we consider as a PDF that reflects our (incomplete) knowledgeabout the quantity path length between two reactions of a neutron in matter.

( )ξXg

The prior knowledge about the interaction site is expressed by a PDF,but there will be exactly one site where the next interaction occurs!

Stochastic models can be used with Monte Carlo codes and deterministic codes. See for example: Hamid Tagziria, H. (editor): Review of Monte Carlo and Deterministic Codes in Radiation Protection and Dosimetry, NPL-report, Teddington, UK, 2000, ISBN: 0 946754 32. We cite from the introduction of that report:Modelling a physical system can be carried out either stochastically or deterministically. An example of the former method is the Monte Carlo technique in which statistically approximate methods are applied to exact models. No transport equation is solved as individual particles are simulated and some specific aspect (tally) of their average behaviour is recorded. The average behaviour of the physical system is then inferred using the central limit theorem. In contrast, deterministic codes use mathematically exact methods that are applied to approximate models to solve the transport equation for the average particle behaviour. The physical system is subdivided in boxes in the phase-space system and particles are followed from one box to the next. The smaller the boxes the better the approximations become.Note on sub-models: It can be shown, that most models can be broken down into few types of sub-models. Sub-models are easier to test and they simplify the encoding of models in higher computing languages, such as FORTRAN.

7



Step by step simulation of possible paths.

Monte Carlo Method Introduction (1/7)

1 2

Law of Large Numbers: If the expectation value EX exists, then for any εE > 0 exists a M < ∞.

( ) ( ) E1

1-max

min

d E ε+⋅== ∑∫ mm

M

m=gMVg ξξξ ξξξ

x

x

( ) ( ) (volume) minmaxT

minmax xxxx −−=V

Why Monte Carlo? Analytical or othernumerical methods

Validation?

Convergence?

If the variance exists, then one can compute u( EX )

[ ] ( )( )∫ −=max

min

d EVar 2x

x

ξ ξξξξ g

Variance:

• Integrals• Integral equations• Random walk• Bootstrap• ........xmin, xmax , ξ and ξm are vectors of dimension N, M denote the number of samples and

m a sample, the nth element of ξm is given by ξm,n = (xmax,n − xmin,n ) ρm,n and ρm,n is arandom number, ρm,n ∈[0,1].

The Monte Carlo method (MCM) is concerned with the application of random sampling to problems of applied mathematics. A short and well-coined description has been given by H. Kahn in. Applications of Monte Carlo, Report AECU-3259 (U.S. Atomic Energy Commission, The Rand Corporation, Santa Monica, CF /USA), 1956) :The expected score of a player in any reasonable game of chance,however complicated, can in principle be estimated by averaging the results of a large number of plays of the game. Such estimation can be rendered more efficient by various devices, which replace the original game with another known to have the same expected score. The new game may lead to a more efficient estimate by being less erratic, that is, having a score of lower variance or by being cheaper to play with the equipment at hand.The MCM is used in a wide range of applications in physics, e.g. Binder, K. (editor): Monte Carlo Methods in Statistical Physics. Topics in Current Physics, Vol. 7 [36], (Berlin Heidelberg New York 1979 [1984], ISBN 3-540-09018-5 [3-540-12764-X].The internet provides ample information on history, theory and present applications of MCM to natural and other sciences. In this section we discuss only general principles. The concurrent ex-amples are discussed in section 3 (integration) and 4 (simulation).

8



Monte Carlo Method Random numbers (2/7)

0 20 40 60 80 100 120 140ξ

10-9

10-8

10-7

10-6

10-5

10-4

10-3

10-2

G(ξ

)

TheoryTheory 2.1092.109

2.1062.106

16 digits are given16 digits are given

ξ = sum over their valuesξ = sum over their values

For both, integration and simulation,one needs many random numbers.(About 500 to thermalise a fast neutron)One is to rely on random numbersprovided by an algorithm. It is notpossible to verify such random numbergenerators!But one can validate them bycomputing problems for which oneknows the answer exactly.For instance, the frequency distribution of the sum over all digits.

1 2

If one has a random number generator that producesuniformly distributed values in the interval [0,1]one can reproduce any probability distribution function,be it continuous or discrete.

For instance, we use the portable random number generator that is suggested in the supplement* in a 16 digit representation and tested the generator by computing:o the frequency distribution of the sum over all digits (see Figure),o the frequency that the number 0,… or 9 occurs in a random once, twice …16 times,o the frequency that the number 0,… or 9 occurs in any of the 16 digits ando the frequency of chains of the numbers 0,… or 9 of length n =1,2,…8 starting at

the (17 - n)th-digit.For the four questions one can compute the exact answers. Furthermore, we tested for correlation and computed analytically known expectation values, variances and higher moments. For all values computed, we calculated in addition their variance by repeating the calculations many times. The tests did not exhibit any significant deviation from the expectations based on analytical calculations or logical reasoning. Therefore, we assume that the random generator used is sufficiently ideal for our applications; however, this is not a proof!

* See paper WS-35 by M. G. Cox, P. M. Harris and I. M. Smith:Supplements to the guide to the expression of uncertainty. in measurement.

9



Monte Carlo Method Sampling(1/2) (3/7)

( )( ) x.xu +−= 503 ρξ

If a reliable random number generator is given one obtains:

a sample from an uniform (rectangular) PDF from

where, using GUM nomenclature, x is the expectation value and u(x)=σa uniform PDF is given if one only knows the limits x− and x+of the values that might be reasonably attributed to that quantity:(Theory: Principle of Maximum (information) Entropy.

Hand waving: Any other shape would pretend not given information!)

a sample from a triangular PDF can be obtained from a sum of two samplesfrom uniform PDFs of equal half width,a sample from a trapezoidal PDF from a sum of two samples fromuniform PDFs of different half width anda sample from a U-shaped PDF is obtained from

( )

( ) ( )xuxxa

xxx

32121

=−=

+=

−+

−+

( )( ) ( )xu. 250sin −= ρπξ

1 2

To simulate throwing a fair dice with n faces, labeled 1,...i,..none would use: , but use n if n+1 is sampled![ ] 1 += ρni

σ µ

The central limit theorem [(A.M. Ljapunoff (1857 - 1918)]. is used for two entirely different aspects of the problems studied here.(1) The PDF for an output quantity that is linearly combined from many input quantities tends to be a Gaussian according to the central limit theorem. Indeed, for instance the linear combination of a few rectangular PDFs with approximately the same variances can in practice not be dis-tinguished from a Gaussian. This is the basis for the often made statement“for a linear model with many input quantities one obtains a GaussianPDF for the measurand.”

(2) The distribution Z of repeatedly computed expectation values of any PDF Y tends to be a Gaussian if the number of samples, M, used for computing the expectation values becomes large. The variance of Z is according to the central limit theorem given by M−1 Var[Y]. If Y is a Gaussian then Z is a t-distribution of degree of freedom ν = M−1:

If Y is not a Gaussian one can, if needed, compute Z by Monte Carlo techniques.

( ) ( )xux

ν

NNg ,T−

=)2Γ(π

⎟⎠⎞

⎜⎝⎛ +

Γ=⎥

⎦

⎤⎢⎣

⎡+=

+−

ξτνννττ ν

ν

νν and 21

where12

12

10



Monte Carlo Method Sampling (2/2) (4/7)

1 2

There are various sampling techniques. Analogue sampling works always:

-3 -2 -1 0 1 2 3ξ / u(x)

0

0.25

0.5

0.75

1

G (ξ

)

Cumulative distribution function1st step: G(ξ) = ρ

2nd step: ξ = G-1(ρ )

3rd step: ξ i,s = ξi u(xi)+ x

( ) ( )

( ) ( )

( ) ρξξξρξζζ

ξ

ρξζζ

ξ

ξ

ddddd

dd

d

=⇒

==

==

∫

∫

∞−

∞−

X

XX

XX

g

gg

Gg

If G(ξ) can not be expressed ana-lytically, one can use a discrete table: ⇒dice with n faces+ interpolation.

For many PDFs, e.g. a Gaussian or a t-distribution algorithms exist that produce a sample, e.g. based on rejection technique.

Correlation: Joint PDFs Choleskyfactorisation orauxiliary quantities.

Variance reduction: StratifiedImportance..........

Annotations:

Use standard forms, σ =1 and EX =0 ⇒ ξs= u(x) ξs,n + x

- continued from notices on bottom of page 9-In computing integrals and in doing step by step simulations we have the situation as described under (2) above, i.e. we can consider a computed expectation value. This allows then the simple statement: If the variance exists one can reach convergence and the uncertainty associated with the once computed expectation value of Y , u2(EYcomputed) = M−1 Var[Ycomputed].-----------------------------------------------------------------------------------------------------------------------------

Rejection technique (von Neumann):Consider a probability density function, g(ξ ), which can be surrounded by a box (or any function come to that) between xmin and xmax and havingheight gmax.The following algorithm yields - for a large sample number- g(ξ):(1) Create ξ = xmin+ (xmax − xmin ) ρ1

(2) Create g = gmax. ρ2

(3) If g < g(ξ), then accept ξ. If not, reject and go to (1).For transformation method see next page and for correlation and variance reduction see bottom of pages 12 and 13, respectively.

11



Monte Carlo Method Sorting & statistics (1/3) (5/7)

1 2

Evaluation of uncertainty: Generation of a possible value of the measurand.

Model for the evaluation, Y=f(X1,...,XN), combines possible values of the Xi tothe possible values of the measurand: ( ) sample.current ; 1 == m,...,f m,Nm,m ξξη

We do not need to consider εE and εV since we can use the central limit theorem:

( ) .MyMMuM

mm

M

mm ∞→−⇒⎟⎠

⎞⎜⎝

⎛ ∑∑=

−

=

−

1

22

1

12 for ηηThe ηm can be sorted, this yieldsa frequency distribution thatapproximates the PDF for Y.Details see Section 3.

Caveat: Check conditions for central limit theorem.

This procedure is equivalent to solving integrals (see Sect. 3):

NNN ξξξfξg...y ...dd ),...,(),...,( 111 ξξ∫∫∞

∞−

∞

∞−

= X ( ) ( )( ) .ξξy,...,ξξfξg...yu NNN ...dd),...,( 12

112 −= ∫∫

∞

∞−

∞

∞−

ξX

( ) [ ] ( )∑∑=

−

=

− +−−==+==M

mm

M

mm yMYyuMYy

1Var

212E

1

1 .1)(Var and E εηεηFor large samplenumber M

-continued from notices on bottom of page 10 -Transformation methodConsider the general transformation: and introduce:

The Jacobian determinant can be computed and one obtains:

Take gR (ρ) is uniform for both, ρ1 and ρ1, one obtains 2 independent Gaussians in standard from this transformation. This is the Box-Muller method (Ann. Math. Staist.,29 (1958) p. 610.611.). It requires the calculation of sinus and cosines functions. This can be avoided by using rejection techniques, for detail see Press, W.H., Flannery, B.P., Teukolsky, S.A. and Vetterling, W.T.: Numerical Recipes -The Art of Scientific Computing-. Cambridge, Cambridge University Press (1989)).Correlation and variance reduction see next pages, 12 and 13,respectively.

( ) ( ) ( ) ( )and 2sinlog2 and 2coslog2 21e221e1 πρρξπρρξ −=−=

( ) ( ) ( )( ) ξρξ RX d

21

21

ξξρρ

,,gg

∂∂

=

( ) .- ⎟⎟⎠

⎞⎜⎜⎝

⎛=⎥⎦

⎤⎢⎣⎡ +=

2

12

22

211 acrtan

21 and

21exp

ξξ

πρξξρ

( )( ) .

,,

⎥⎦⎤

⎢⎣⎡−⎥⎦

⎤⎢⎣⎡−=

∂∂

2exp21

2exp21 2

22

1

21

21 ξπ

ξπξξ

ρρ

12



Monte Carlo Method Sorting & Statistics (2/3) (6/7)

1 2

Step by step simulation: Generation of a history (1/2).

Step by step procedure yields a possible history of the object under study.

Random walk problem: Transition probabilities from one step to the next one.(interaction probability).

Associate a state vector with object under study.Number of elements depends on the purpose of the calculation.

Concurrent example:probability, time type of particles, energy, location, direction of flight and flags.

k is e.g. deuteron or neutron, flags indicate history

and via r the material and geometry are known. ( )flags,,,E,k,t,p Ωrs =T

Markov chain: Transition probabilities from a previous state to next statedepend only on the previous state vector.

E.g. D(d,n) happened in r ±∆ r and n was x-times scattered and ...

Properties of spectators⇒ modelling, optimising

“possible” ≡ (synonym) reasonably expected due to my (incomplete) knowledge.

CorrelationIn the case that the information about correlated input quantities is given by joint Gaussian distribution one can use the Cholesky procedure, which decom-poses the uncertainty matrix Ux for the values of the input quantities such that Ux = RTUx = RTR; where R is a matrix for which all elements below its diagonal are zero. The product of RT with values sampled from the PDF in standard form yields the sample values needed. For an illustration of the formalism, we discuss briefly the case of two correlated input quantities:

One obtains:

where we set r(x1,x2)=sinα and the ξstd,i,m are independent samples from a Gaussian in standard form, i.e. σi =1 and EXi = 0.If the correlation coefficient is given and if the shape of the PDF for the quantity that is the logical cause of correlation is known, then one can introduce it as auxiliary variable and consider Xi as an appropriately weighted sum or difference of its uncorrelated part Xi,0 and the quantity Xc, that causes the correlation.

( ) ( ) ( ) ( )( ) ( ) ( ) ( ) ⎟

⎟

⎠

⎞

⎜⎜

⎝

⎛=

22

2211

221112

xuxux,xrxu

xux,xrxuxuXU

( )( ) ( ) ⎟⎟

⎠

⎞⎜⎜⎝

⎛⎟⎟⎠

⎞⎜⎜⎝

⎛+⎟⎟

⎠

⎞⎜⎜⎝

⎛=⎟⎟

⎠

⎞⎜⎜⎝

⎛

m,

m,,

m,

m,

xuxuxu

xx

2std,

1std

22

1

2

1

2

1

cos sin0

ξξ

ααξξ

13



( )flags,,,E,k,t,p Ωrs =T

A history is manipulated to yield useful information; this is the reason for p in s.

Monte Carlo Method Sorting & Statistics (3/3) (7/7)

1 2

Step by step simulation: Generation of a history (2/2).

Concurrent example: every• started deuteron is to produce a neutron• neutron is to reach the sample and• detector is to be reached by scattered neutrons.

Enforcing is “paid” for bymultiplying the probability with

Generally, in a virtual experiment one needs methods to reduce variance.


xx

zz


DetectorDetector

ϑn,n'ϑn,n'

A site r X(n,n’) for a scattering event , the nucleus and the scattering reaction be given.Select a point on the surface of the detector r Detector that is far away.

reduce ⇒ redistribute

X

Caveat: Method is in principle simple, but there are pitfalls.

( )2

Detector Sample

Detector

Reaction

ReactionDetector Sample 4cos

cos

i

ii r

A,Epi

→→ ∂

∂=

πϑσϑσ

ϑ

Stratified sampling (as one example of Variance reduction techniques)

The path length between two reactions of a neutron in matter, (viewgraph 6) leads to an exponential expression of the form:

λ is termed mean free path. We consider the path length of a neutron normally incident on slab with a thickness lmax. The cumulative distribution function can be written in closed form:

The inverse exists and we obtain a representative sample of the path length: ξ = −λ loge(1-ρ). However, for any ρ > 1−exp(−λ−1 lmax) the neutron would pass through the slab with any interaction. To avoid this waste of CPU time, one enforces an interaction in the slab. To that purpose we re-normalise the PDF:

and give a sample the “statistical weight” w= 1−exp(−λ−1 lmax). This can be generalised to selecting different path-length intervals with differentweights, large weights there, where the function varies strongly.

( ) ( ) ( );,,E-g X mattern whereexp ntot11 Σ== −− λξλλξ

( ) ( ) ( ).--GX ξλξξλλξξ

1

0

1 exp1 dexp −− −=′′= ∫

( ) ( ) ( )

if 0

if expexp1

1 11

⎪⎩

⎪⎨

⎧

>

≤−=

−−

max

maxmaxX

l

l-l-g

ξ

ξξλλλξ

14



The discussion in recent years and the forthcoming additional guidance by theJoint Committee Guides for Metrology (JCGM) and the definition of uncertainty

allow now to the concept of the GUM quite briefly:

Evaluation of Uncertainty GUM concept (1/3) (1/x)

1 2 3

Uncertainty of measurement (VIM Definition)parameter, associated with the result of a measurement, that characterises the dispersion of the values that could reasonably be attributed to the measurand.

See WS-35 by M. G. Cox, P. M. Harris and I. M. Smith: Supplements to the GUM.

additional guidance definition of uncertainty

Modelof the measurement:

PDFs (probability density function)

for the input quantities:

reflects the knowledgeabout the interrelationof the input quantities Xi with the output quantity Y

( )NX,...,XfY 1= free form

( )N,...,f ξξη 1=uniquely

but fromgiven valuesfor the Xi

reflects our degree of belief in reasonably possible values, based on given information.

Model and PDF

reflects the knowledgeabout the interrelationof the input quantities Xi with the output quantity Y

physical,chemical,biological,…

Result is determined, once model and PDFs are given, it is calculated, not estimated!

GUM concept (brief):The knowledge given on possible values ξi of the input quantities Xi is expressed by PDFs and the knowledge given on their interrelation with the values η of the output quantity (measurand) Y is expressed by the model function f. The values η can be expressed via the model as a function of the values ξi. The form of the model is not prescribed. However, the model must reflect all influencing quantities, including corrections. Otherwise one could loose traceability! A closed theory for modelling is not known.The PME and the Bayes theorem are used to derive a PDF from data. It is important to realise, that a well defined measurand + has one value, but our knowledge is in principal incomplete, whence we can,if we are honest, only express the degree of believe in possible values based on given data* and never state one and only one value.+It is hard to define a measurand well. Consider as simple example the length ofan infinitesimal thin, but perfectly straight, rigid and incompressible wire at a temperature zero Kelvin in absolutely free space!

*Data pl. of datum (Latin: given, cited from Webster): something given or admitted especially as a basis for reasoning or inference.

15



M

TMTM

TAirTAir

mIND

F=MgF=Mg

F→LF→L

ρMρM

ρAirρAir

∆L∆L

δLδL

........bouancybouancy couplingcoupling deviationsdeviations(transfer)(transfer)

0 0

5 5

1010

1515kgkg

( )MINReading,MAXReading,0 mml −=


1 2 3

“Model” demonstrates:even simple models, that neglectmany influences, need furthersimplification and are not linear.

Just for demonstration!

mgBF =

⎟⎟⎠

⎞⎜⎜⎝

⎛−=

m

Bρρa1

INDINDScaleIND δ∆ mmmm ++=Approximation:CF→L = l0 α (F/2)

& F ≅ mIND E(gB)

LFFCm →=Scale

Model: example

( ) ⎟⎟⎠

⎞⎜⎜⎝

⎛′′+= ∫

+

→ FT,FlCFF

FLF d1

0

0

0 α

It is good practice to break down complicated models into sub-models.

( ) ( ) E2

δ∆ 1

IND0INDINDIND

−

⎟⎟⎠

⎞⎜⎜⎝

⎛⎟⎟⎠

⎞⎜⎜⎝

⎛−−=

gBmgBlmmmm α

Measurand is m:

δ∆ INDIND mgBCmmm LFS −−= →

Calibration:mIND & E(gB) are const., need PDFs for others!

buoyancy

Clearly, the above “model” is contrived, since one would turn to an alternative measurement principle and measurement method, if one were to aim at such small uncertainties that all of the discussed influences become significant.But, it is hopefully didactical, as all discussed effects can be understood in basic physical terms.Stated without proof, Monte Carlo integration could even tackle

where F is force and T is temperature.

Admittedly, it would be hard work. However, a detailed study on the above used approximation could be carried out using Monte Carlo. Such a study could result in in a parameterisation of a PDF for CF→L

as a function of F, or , if needed of F and T.There may be many cases in practice, where it could be necessary, to use Monte Carlo techniques for studying whether or not made approximations in modelling are justified.

( ) ⎟⎟⎠

⎞⎜⎜⎝

⎛′′+= ∫

+

→ FT,FlCFF

FLF d1

0

0

0 α

16




Consider a model and PDFs for the input quantities Xi are given:(Information of TYPE A and TYPE B leads via PME and Bayes theorem to PDFs.)

Combining PDFs

1 2 3

How can one determinepossible values of Y, η ?

( ) ξξξ dE ∫∞

∞−

==iXii gXx ( ) ( )( ) ξξξ dVar 22 ∫

∞

∞−

−== xgXxuiXii( )321 X,X,XfY =

ξ1

ξ2

ξ3

ηm = f (ξ1,m, ξ2,m, ξ3,m )

ξ1,m ξ2,m ξ3,m

Representative: e.g. analogue sampling(see viewgraph 10)Fo

rwar

d ca

lcul

atio

n

* Reminder: Knowledge about values is expressed by PDF

*For determining the coverage interval we need gY(η).

By representative combinationsof all possible values of the inputquantities, according to the model !

Markov formula: NNNY ξξξfξgg ...d))d,...,((),...,()( 111 ξηδξη −= ∫∞

∞−X

If the dice have an increasing number of faces, then the resulting PDF for the sum will tend to a triangular PDF. If one uses dices with different numbers of faces one obtains a trapezoidal PDF.

00 11r=(2π)-1r=(2π)-1

+

)(f 21 ξξη +=

Possible values of Y and

do depend on + – . or /, but the number of combinations does not!

Probabilities for values

2211 33 44 55 66 2211 33 44 55 66

22 33 44 55 66

2211 33 44 55 66

77 88 99 1010 1111 1212

361

362

363

364

365

366

365

364

363

362

361

2211 33 44 55 66 2211 33 44 55 66

22 33 44 55 66

2211 33 44 55 66

77 88 99 1010 1111 1212

361

362

363

364

365

366

365

364

363

362

361

361

362

363

364

365

366

365

364

363

362

361

361

362

363

364

365

366

365

364

363

362

361

Consider two fair dice with six faces each

Assume now a continuous fair dice:all values between 0 and 1 are equally probable, this leads to a uniform PDF and a triangular PDF for Y.

[ ]10,i ∈ξ

Illustration of combining possiblevalues from two PDFs, here discrete.Try it!

17

L

p

BB b

h

QQ BB b

Construction causes a raise Construction causes a raise of water level (Bernoulli).of water level (Bernoulli).

23

23

32 hbCCgQ Dv⎟⎠⎞

⎜⎝⎛=National Engineering Laboratory, UK

Flow rate in a cooling water channel

23

0030100601 ⎟⎠⎞

⎜⎝⎛ −⎟⎠⎞

⎜⎝⎛ −=

hL.

bL.CD01

332

322

22

=+−⎟⎟⎠

⎞⎜⎜⎝

⎛+⎟

⎟⎠

⎞⎜⎜⎝

⎛vv CC

phh

Bb

? =∂∂

BQ



Evaluation of Uncertainty Markov Formula & Standard Procedure (4/x)

1 2 3

-2 0 20

0.2

0.4

g X i( ξ i

)

-1 0 1ξ i / u( xi)

-2 0 20

0.5

1

GX

i (ξ i

)

0 100 200 300Bin k

0

5

10

g Y (k

)

0

103

← M=1000← M=1000 M=1000000 →M=1000000 →

Markov formula: NNNY ξξξfξgg ...d))d,...,((),...,()( 111 ξηδξη −= ∫∞

∞−XMarkov formula: NNNY ξξξfξgg ...d))d,...,((),...,()( 111 ξηδξη −= ∫

∞

∞−X

ξξξX d )()( fgy ∫∞

∞−

=

( ) ( )[ ] ξξξX d)( 22 -yfgyu ∫∞

∞−

=

Integration over η:

( ) ( )∑=

−+=N

iiii xcf

1 ξη x

In general y ≠ f (x),but for linear model:

since E (ξi− xi) = 0

vector

Linear model in Markov formula ⇒ u2(y) = cT UX c, i.e. GUM standard in matrix form

cT=(c1,...,cN)uX,i,j= u(xi,xj)

Frequency distribution⇒ PDF for measurand

η =ηoffset + ck→η k

An example of a problem, where the numerical calculation of at least onesensitivity coefficient is fairly difficult.The example was originally provided by the National EngineeringLaboratory, UK.An obstruction in a cooling water channel (supply for a power station) leads to a measurable raise of the water level. x x

18



Evaluation of Uncertainty Advantages of MCI ? (5/x)

1 2 3

Monte Carlo integration:

Key comparisons

Model is non linear

PDF for Y is not a Gaussian

More than one output quantity

Sensitivity coefficients.

Real world is not alwaysGaussian

( ) 950dmax

min

.gy

yY =∫ ηη

Find min ymax− ymin

Dominating uniform or t-PDF,Asymmetric PDF.

E.g. compute median or pooled data.

Straightforward, but problem to represent U.

1nwi =

( )nY,.....,Y,YfZ 21≡ ∑ ⋅=n

iii ywz

∑ ⋅=n

ijiij w ,ηζ

( )

( )∑′

′−

−

⋅= n

ii

ii

yu

yuw2

2

Least squares

( ) odd if 121 NNi*

sort +⋅=

121 2

1 sortsort +⋅=⋅= +− NiNi

Median: sort ζ i and take

Detailed variance analysis and search formodel simplification

k=U/u(y)?

⇒ U− and U+

Supplement:

Monte Carlo sampling allows to compute many interesting feature, such as the variance or the relative weights of the participants: Count the number mPi of samples where Participant i is the median or (even number of participants) contributes to the median: mPi /M is thentermed weight; M= total number of samples.Note: The median for correlated input quantities has peculiar features.

Also being discussed in literature is the median

( )

⎪⎭

⎪⎬⎫

⎪⎩

⎪⎨⎧

+⋅=⋅=⎟⎠⎞⎜

⎝⎛ +⋅

+⋅==

+−+

− even if 121 and 2

1 ; 21

odd if 121 ;

nnini

nni

sortsortii

*sort*i

sortisortηη

ηζ

The values of the ηi obtained in the mth given historyare increasingly ordered, e.g.

η2 ≤ η3 ≤ η1 ≤ η7 ≤ η4 ≤ η6 ≤ η5 n = 7 (odd)

The mth value for ζ is the given by:

η2 ≤ η3 ≤ η1 ≤ η7 ≤ η4 ≤ η6 ≤ η5 ≤ η8 n = 8 (even)

19



Simulation („virtual experiment“) Model (1/4)

1 2 3 4

Monitor

Monitor

Cyclotron

Beam-lineTarget

Sample

D(d,n)D

etec

tors

DetectorDetermination of differential neutron cross section

via a Time of flight (TOF) experiment

“realistic view”

(n, n’)

Target

SKIP 20

Collimator

Details on the simulation of the TOF-experimentA cyclotron is used to accelerate deuterium ions (D+) . The extracted beam is directed to a deuterium gas target, where neutrons are produced. The neutrons are emitted in all directions, some will hit a sample, say consisting of polyethylene or other materials of interest. The neutrons interact with that material and are scattered. Some of these neutrons are detected by detectors under different angles. A collimator assures that the detectors “see” only the sample and not the target. A calibratedmonitor, shielded such as to “see” only the target, is used to link the measurement to neutron fluence. The cyclotron, together with beam line and target, can be turned around the sample, such that the set of five detectors allows scattering angles from about 120 to 1650 to be obtained.The simulation starts by selecting a deuteron with a given time (of “birth”), energy and location. These data are sampled from PDFs that are derived from emittance measurements. The deuterons are transported to the target, they loose some energy in the gas and are forced to generate a neutron in an spatial angle that encompasses the sample. The neutrons are scattered in the sample and then transported to the detectors.The important variable is the time (of flight) that elapses between thebirth of a deuteron and the detection of a neutron in a detector.(see also bottom of page 22)

20



xx

z

y

ϑiϑi

MonitorDetector i

Detector j

ϑjϑj

SampleSample

Gas target

Simulation („virtual experiment“) Model (2/4)

1 2 3 4

d passes through fixed plane

td,0 , Ed,0 , rd,0, cd,0

d is slowed down in foil andgas prior to possible reaction

td,1 , Ed,1 , rd,1, cd,1

x

d reacts: D(d,n)

tn,1 , En,1 , rn,1, cn,1

Enforce n to react in monitorEnforce n to interact in sample

Enforce n‘ to interact in detector 1

Enforce n‘ to interact in detector n...............Iterate

Compute TOF spectra:

Total, Mul 1,2,..mDiscriminate reaction typesand compute spectators.

Gas target

Determination of differential neutron cross section via a Time of flight (TOF) experiment

steps for computation(overview only)

Sort event in spectra

Details on the simulation of the TOF-experiment(continued from page 19)Details on treating the gas targetThe entrance window of the gas target (needed since the beam line is evacuated) is very thin and bent out toward the vacuum.The deuterons are slowed down in that entrance window and there is not only a loss in energy, but also energy and angular straggling.Very few deuterons react in the gas target. This allows to approximate the loci of interaction by an rectangular PDF and to use a simple energy loss model.The reaction D(d,n) is not isotropic. The code samples a location on a circle (normal coincident with beam line) that is circumscribed about the sample. This fixes the direction of the newly “born” neutron and the spatial angle. Since the sample is a cylinder, not all neutrons hit it. They need to be rejected and this leads to a correction of the spatial angle that is used for computing the statistical weight of the neutron.Details on treating the sampleNeutrons, that hit the sample are forced to react. Special care is needed for neutrons that intersect the top or the bottom of the sample cylinder. Furthermore, some samples are liquids and then one needs to also account for scattering in the flask (closed thin metallic cylinder).

21



Simulation („virtual experiment“) Inference (3/4)

1 2 3 4

( ) ΩΩσΩΩσ ′⋅′⋅′′⋅′′< ∫∫rrrr

dd)( = >)( prior r,Simulationexp,Simulationexp,r E,E,Eg,E

K(ϑi) is obtained from fittingthe simulated spectra to theexperimental ones.

Simplified: The sum of themeasured events in the peakspertaining to the reaction understudy is normalised to 1.

The sum of the simulatedevents in the peaks pertaining tothe same reaction is normalisedto 1.

( ) Literatur""reactionneureaction )( )( iii ,EK,E ϑσϑϑσ ⋅=

Determination of differential neutron cross section via a Time of flight (TOF) experimentTOF

Cou

nts

per c

hann

el

C(n,n’) H(n,n’)

Edge is used to validatecyclotron position! (“intrinsic-calibration”)

GOTO ENDE

Details on the simulation of the TOF-experiment(continued from page 20, Details on treating the sample)How an intersection site is generated is explained on page 13 (bottom).If the interaction site is selected, a nucleus is to be selected as reaction partner.If the nucleus is selected, a possible reaction is to be selected, for instance elastic scattering or inelastic scattering; the latter can have different Q-values (for different excitation levels).Details on treating the detectorsFrom each interaction site (nucleus and reaction are selected) the code computes then the probability to reach a detector for this neutron. This is done for up to fifty detectors under different angles. The neutrons are sorted in tallies according to their time of flight.Then multiple scattering is enforced in the sample and the probability of detection by the detectors is again computed, This iteration is ended by various cut-off conditions.The needed PDFs are generated from cross section data found in evaluated neutron data file (ENDF) and stopping power tables. The geo-metry data are taken from the construction sheet, the uncertainties are not treated, but there influence was examined.

22



Simulation („virtual experiment“) Validation and uncertainty (4/4)

1 2 3 4Determination of differential neutron cross section via a Time of flight (TOF) experiment

Literature, start valuesFinal values

Reproduction of measurements validates code Sources of uncertainty in a simulation:algorithmstochasticestimatorsapproximations.....

parameters:geometrymaterial compositionsstopping powerscross sections

......

Brute force,correlated samplingand perturbationtechniques

Advantages: Optimisation of the experiment, “spectators”calculation of not measurable coefficients,...

Costs: Computing times, development, but: well-maintained and quite flexible code packages exist, e.g. MCNP for radiation transport.

Edge is used tovalidatecyclotronposition!

Annotations to spectators for optimisationThe accumulated tallies, i.e. the time of flight spectra for each detector, are then the basis for the evaluation as briefly outline on viewgraph 21.The main advantage of this simulation method is, that one can compute many so-called spectators that provide valuable information, which couldnot be obtained from the real experiment.Example are:

extra tallies for total spectra 1st scattering for each detector and

…………… for each reactionmth scattering.

tallies that discriminate against geometry, regions in targetregions in sample.

tallies for the statistical weight distribution in time of flightwindows to assess uncertainties

An additional power of such a simulation is the possibility to use it to optimise the measurement, for instance by searching for the optimal detector positions, optimal target-sample distance and sample form.

23



Summary and Conclusions

51 2 3 4

Thanks for your attention.

Conclusion:Monte Carlo techniquesadd value to metrology sincethey open up new ways formodelling and “exact” solutions.

0 0

5 5

1010

1515kgkg

0 0

5 5

1010

1515kgkg

0 0

5 5

1010

1515kgkg

Measurement 0 0

5 5

1010

1515kgkg

MeasurementMM

M MM

Evaluation of uncertainty 0 0

5 5

1010

1515kgkg

0 0

5 5

1010

1515kgkg

0 0

5 5

1010

1515kgkg

Measurement 0 0

5 5

1010

1515kgkg

MeasurementMM

M MM

Evaluation of uncertainty


xx

zz


DetectorDetector

ϑn,n'ϑn,n'

Simulation („virtual experiment“)

Monte Carlo techniques are ideal tools forcomputing the PDF for the measurand(Markov formula: “propagation” of PDFs)

simulating complicated experiment(Necessary for evaluation)

in depth uncertainty analysismodel simplifications

Approximation: CF→L = l0 α (F/2)and F≅ mIND gB

Final remarkThe method can also be used to optimise the model for the evaluation of uncertainty, An indication was given on bottom of page 15. This might be even of more interest with respect to model for key comparisons. Also, if the PDFs for the participants’ measurand are not Gaussian, or if there is correlation, Monte Carlo techniques could help to find the best model and solutions.Even more important, the method does not need to rely on statistics theorem that are often only valid for certain PDFs, mostly Gaussians, and very large numbers. One can simply reiterate the entire calculation in order to get numerical representations. For instance: What is the correct uncertainty if you have correlated few repeated measurement, drawn from an uniform PDF? Simply draw a couple of hundred million time such samples and sort them (takes only a few hours on a modern PC). The extension to more than one output quantity is trivial.Above all these advantage, the method is also intellectually pleasing, as it is simple and implements Bayesian inference in direct way with given priors (see Introduction).Nevertheless, do not use Monte Carlo, if you can get analytic solutions.But, use always (partial) analytic solutions to validate your code and assess the uncertainties contributed by the algorithms used. [email protected]

Evaluation of uncertainty Simulation („virtual experiment ... · 3 3 The use of Monte Carlo...

Documents

Transcript of Evaluation of uncertainty Simulation („virtual experiment ... · 3 3 The use of Monte Carlo...