Chapter 8 · Φ = E(rrT ) = GE(εεT )GT = GVGT. Basic Concepts in Data Reconciliation ......

Basic Concepts in Data ReconciliationBasic Concepts in Data Reconciliation

© Universidad de Guanajuato, Mexico © University of Ottawa, Canada, 2004

Chapter 8:Gross Error Detection



8.1 Global Test

In the previous seven chapters, it was assumed that only random, normally distributed errors (with zero mean and known variance) were present in the data.

However, real process data can often include other types of errors, due mainly to instrument malfunction and process leaks. These gross errors are non-random and completely invalidate the statistical basis of data reconciliation techniques.

This is why it is necessary to have methods to check a data set for gross errors. If any data is found to contain gross errors, it must be removed prior to the reconciliation.

CHAPTER 8

Gross Error Detection



8.1 Global Test

Of course, there are many possible ways to detect gross errors, but the methods discussed in this chapter will be based on how well statistically the data satisfies the balance equations.

The first method discussed, the global test, is fast and useful. As its name implies, the global test can determine whether or not a data set contains gross error(s), but will not indicate precisely where.

Therefore, it is a good preliminary analysis, in order to see iffurther action will be required.

CHAPTER 8




8.1 Global Test

As shown earlier in the module, the reconciled measurement vector (without gross errors) can be written as:

(8.1)

where y is the vector of redundant raw measurements, and is the vector of random errors. Also, it must be assumed that:

1) Expected value of , E( ) = 02) Covariance matrix, V, is known.

CHAPTER 8


ε+= yy

εε

ε



8.1 Global Test

Obviously, gross errors can only be checked for in data that has been measured, so unmeasured data, z, and its corresponding balance equations, Az, need not be considered in the analysis. Therefore, the only balances to be reconciled correspond to:

(8.2)

where G in this case (and for the remainder of Chapter 8) corresponds to the reduced subset of redundant equations(G=Q2

TAy) and y corresponds to the redundant measured variables.

Due to random error, there is bound to be residua, r, leftover in the balances, such that:

(8.3)

CHAPTER 8


0Gy =

εεε GGGy)G(yr =+=+=



8.1 Global Test

The global test is based on statistical hypothesis testing, where the null hypothesis, H0 (no gross error is present), is tested against the data set. Under this hypothesis, if the random errors are normally distributed, then Equation 8.3implies that the residua are also normally distributed, and:

(8.4)

Also, the propagation of this error, the covariance matrix of residuum, can be shown by:

(8.5)

CHAPTER 8


0)GE()E(GE(r) === εε

TTTT GVG)GGE()E(rr ===Φ εε



8.1 Global Test

The global test function, then, is formulated as:

(8.6)

If G has full row rank equal to m, then has a chi-squared distribution with m (# of equations) degrees of freedom. Therefore, at some specified level of significance, (usually0.05 or 0.10):

(8.7)

For a specific , a critical value of , , can be calculated. If for a data set is larger than this critical , then the data set has a (1- )% chance of containing gross error.

CHAPTER 8

Gross Error Detection rr -1T Φ=τ

τ

τ cττ

α

αcτ

α

αχτ α =≥ − m))(P( 12



8.1 Global Test

For example, consider the following redundant measured flow rates, mass balance equations, and covariance matrix:

CHAPTER 8


⎥⎥⎥⎥

⎦

⎤

⎢⎢⎢⎢

⎣

⎡

=

3.88001.22954.79350.1858

y⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

−−−−−−

=2.06.03.01.01.02.01.08.07.02.06.01.0

G

⎥⎥⎥⎥

⎦

⎤

⎢⎢⎢⎢

⎣

⎡

=

0.0400000.00057600000.002500000.000289

V



8.1 Global Test

Calculating the covariance matrix of residuum, and using the measured data in the three mass balance equations to find the residuals, results in:

Using Equation 8.6:

CHAPTER 8


⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡=

0.0571-0.0059-0.0672-

r⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡=Φ

0020.00010.00061.00010.00006.00030.00061.00030.00205.0

8.5347rr -1T =Φ=τ



8.1 Global Test

The rank of G is equal to three, so the critical value with 3 degrees of freedom and = 0.05 is:

Since > , there is a 95% chance that the data set contains gross errors in one or more of the measurements.

CHAPTER 8


τ

cτα

cτ

7.815 (3)0.952

c == χτ



8.2 Serial Elimination

Once a gross error has been detected in the data set, the next step is to determine where (which measurements) it has occurred.

Serial elimination is a technique which can test one or more redundant raw measurements and determine if gross errors are present. In this section, it will be shown how to use serial elimination to detect gross errors by going through the data setsequentially, testing one measurement at a time.

Since a different method to test multiple measurements for gross errors will be discussed in the next section, readers interested in using serial elimination for this purpose are encouraged to consult the Suggested Readings.

CHAPTER 8





Assuming that c measurements have gross errors and that g-c do not, the matrix G may be divided as follows:

(8.8)

Where Ag contains the columns of the measurements without suspected gross errors and Ac contains the column(s) of the measurements assumed to have gross errors.

If there is no correlation between the measurements, then the covariance matrix can be divided in the same manner:

(8.9)

CHAPTER 8


[ ]cg AAG =

⎥⎦

⎤⎢⎣

⎡Δ+

=VV0

0VV

c

gn




where is the increase in the variance of the c measurements (very large) due to gross error.

The new covariance matrix of residuum, , can be calculated as follows:

(8.10)

The best way to calculate the inverse of this, , is to use the following matrix inversion lemma:

CHAPTER 8


VΔ

nΦ

-1nΦ

Tn n

T Tg g g c c c

Tc c

GV GA V A A (V V)A

A ( V)A

Φ =

= + + Δ

= Φ + Δ




Let Y and Z be nonsingular matrices of order (nxn) and (pxp) respectively, and U a (nxp) matrix.

If (8.11)

Then (8.12)

Applying this lemma to find the inverse of Equation 8.10results in:

(8.13)

CHAPTER 8


TUZUYX ±=

-1T-1-1T-1-1-1-1 YUU]YUU[ZYYX ±= m

}A]AAV)[(A-{I -1Tc

-1c

-1Tc

-1c

-1-1n ΦΦ+ΔΦ=Φ




Since is assumed to be large, its inverse in Equation 8.13can be assumed to be zero, resulting in this modification:

(8.14)

Finally, the value of the least squares objective function, J, can be calculated as shown:

(8.15)

CHAPTER 8


}A]A[AA-{I -1Tc

-1c

-1Tcc

-1-1n ΦΦΦ=Φ

VΔ

T -1nJ r r= Φ




Now, it can be determined which measurements contain gross errors by following these steps:

1) Remove measurement i from the data set, and use its corresponding column from G to make the Ac matrix.

2) Calculate using this Ac matrix as shown in Equation 8.14.

3) Calculate the objective function that results from removing measurement i, Ji.

4) Repeat Steps 1-3 for measurements i+1, i+2, etc.5) If Ji is very small (and smaller than in particular), then

measurement i contains gross errors (obtaining a low value for an objective function indicates less error, thus the measurement removed to obtain a low objective function must contain the error).

CHAPTER 8


-1nΦ

τ




To illustrate the use of serial elimination, consider the example shown in the last section. Using the global test, it was found that there was a gross error in the data set. Now, by followingthe steps shown on the previous slide, it can be found which measurement contains the gross error.

Step 1:

Removing y1 from the data set results in this Ac matrix:

CHAPTER 8


⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡=

0.10.80.1

Ac




Step 2:

Step 3:

CHAPTER 8


⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡=

ΦΦΦ=Φ

5057.6452.1-1440.7-452.1-41.2122.5

1440.7-122.5460.9}A]A[AA-{I -1T

c-1

c-1

cc-1-1

n

7.2953rrJ -1

nT

1

=Φ=




Step 4:

Carrying through Steps 1-3 on the remaining 3 measurements results in these 3 objective functions:

Step 5:

Since J2 is the smallest objective function, we can conclude that it is measurement y2 that contains the gross error. It is also possible that measurement y3 may contain a gross error, as its objective function is smaller than as well.

CHAPTER 8


0.9636J2 = 1.5702J3 = 8.4374J4 =

τ



8.3 The Combined Procedure

Although serial elimination is a very effective method for finding the location of gross errors, there is a more efficient approach that combines serial elimination (the sequential treatment of measurements) together with the sequential processing of constraints. For this reason, it is called the combined procedure.

First of all, consider the overall problem consisting of m balances and divide it into m sub problems. That is, split the constraint matrix of redundant measured variables, G, into m individual row matrices, A1, A2, A3, etc. This is done so that one equation may be processed at a time.

CHAPTER 8





For the first equation, A1, the corresponding covariance matrix, V1, is given by:

(8.16)

and the estimate of the measurement errors is:

(8.17)

Also, by replacing the redundant measurement matrix, y, in Equation 8.17 with the covariance matrix, V1, we can obtain the covariance of the estimation error:

(8.18)

CHAPTER 8


VV1 =

yA)AV(AAV 1-1T

111T

111 =ε

11-1T

111T

11 VA)AV(AAVV1

=ε




However, when processing the second equation, adjustments must be made by using the estimates of the measurement errors found while processing the first equation. In this way, the estimates for the measurement errors are constantly updated using new information and will more closely approach their true values.

The covariance matrix corresponding to A2 is given by:

(8.19)

or more generally by:

(8.20)

CHAPTER 8


1V-VV 12 ε=

1-iV-VV 1-ii ε=




Similarly, the error estimates and their covariance for A2 can be shown by:

(8.21)

(8.22)

Or more generally by:

(8.23)

(8.24)

CHAPTER 8


)-(yA)AV(AAV 12-1T

222T

222 εε =

22-1T

222T

22 VA)AV(AAVV2

=ε

)-(yA)AV(AAV 1-ii-1T

iiiT

iii εε =

ii-1T

iiiT

ii VA)AV(AAVVi

=ε




For each equation processed, its least squares objective function can be calculated:

(8.25)

Assuming a normal distribution of the errors, this objective function can be combined with a -test, indicating one or more gross errors in the equation if:

(8.26)

CHAPTER 8


i-1T

iiiT

ii r)AV(ArJ =

2χ

(i)J -12

i αχ>




Once it has been determined that a constraint contains a gross error, serial elimination can be used among its measurements to find precisely where. The measurement that has a much lower objective function (from Equation 8.15) will contain the gross error.

CHAPTER 8




8.4 Estimation of Gross Error Magnitude

Now that a gross error has been detected in a measurement, it is a simple matter to correct it. The measurement can be removed from the data set, the remaining data reconciled, and the faulty measurement estimated from the reconciled data.

However, this is not always desirable or even possible. This section, then, will discuss how to estimate the bias in the faulty measurement and incorporate it into the reconciliation.

CHAPTER 8





Since it is assumed that there are one or more measurements containing gross errors, a constant bias of magnitude mb is added to the redundant raw measurement vector, y. The measurement equation can now be written in the form:

(8.27)

where B is a (gxs) matrix of ones and zeroes (ones corresponding to faulty measurements and zeroes corresponding to unbiased measurements), s is the number of gross errors in the data, and g is the number of measurements.

CHAPTER 8


bBmyy ++= ε




Therefore, the data reconciliation problem becomes:

minimizing(8.28)

subject to

Solving this optimization problem by using Lagrange multipliers results in:

(8.29)

CHAPTER 8


)y-(yV)y-(y)yJ( -1T=

0)Bm--yG( b =ε

]mP-[Gy)(GVGVG-yy bb-1TT=




where(8.30)

and(8.31)

While Equation 8.29 can be used to find new estimates for the reconciled values, it is perhaps easier to simply adjust the suspected measurement by its bias and redo the reconciliation.

CHAPTER 8

Gross Error Detection Gy)(GVGP]P)(GVG[Pm -1TT

b-1

b-1TT

bb =

BAP yb =



8.5 Quiz

Question 1:Gross errors have(a) zero mean and a large variance.(b) non-zero mean and a large variance.(c) non-zero mean and known variance.(d) non-zero mean and unknown variance.

Question 2:When checking a data set for gross errors(a) only measured variables are tested.(b) only redundant measured variables are tested.(c) all measured variables and observable unmeasured variables are tested.(d) all variables are tested.

CHAPTER 8




8.5 Quiz

Question 3:The most efficient method to test for the presence of gross errors is(a) the global test.(b) serial elimination.(c) the combined procedure.(d) (b) or (c).

Question 4:Calculation of a low value for an objective function indicates(a) a gross error if using serial elimination.(b) a gross error if sequentially processing constraints.(c) no gross errors if using serial elimination.(d) (b) or (c).

CHAPTER 8




8.6 Suggested Readings

Romagnoli, J.A. and Sanchez, M.C. (2000). “Data Processing and Reconciliation for Chemical Process Operations”. Academic Press, San Diego.

CHAPTER 8


Chapter 8 · Φ = E(rrT ) = GE(εεT )GT = GVGT. Basic Concepts in Data Reconciliation ......

Documents

Transcript of Chapter 8 · Φ = E(rrT ) = GE(εεT )GT = GVGT. Basic Concepts in Data Reconciliation ......