1 Experts are sometimes used to rank risks associated with accident scenarios, or to
rank the frequency or severity of hazards. One example is the ranking that takes place
at the end of FSA Step 1 – Hazard Identification. This is a subjective ranking, where
each expert may develop a ranked list of accident scenarios, starting with the most
severe. To enhance the transparency in the result, the resulting ranking should be
accompanied by a concordance coefficient, indicating the level of agreement between the
experts.
Calculation of concordance coefficient
2 Assume that a number of experts (J experts in total) have been tasked to rank a number
of accident scenarios (I scenarios), using the natural numbers (1, 2, 3, .. , I). Expert
"j" has thereby assigned rank xij to scenario "I". The concordance
coefficient "W" may then be calculated by the following formula:
3 The coefficient W varies from 0 to 1. W=0 indicates that there is no agreement between
the experts as to how the scenarios are ranked. W=1 means that all experts rank
scenarios equally by the given attribute.
Examples
4 The following three tables are examples. In each example there are 6 experts (J=6)
that are ranking 10 scenarios (I=10). In order to show the role of the concordance
coefficient, the final combination by Σxij constructed by the importance of
hazards 1- 10 for all three groups. From tables 1 to 3 it is quite evident how various
degrees of concordance have been formed.
5 Assessment of significance of the concordance coefficient is determined by parameter
Z:
which has the Fischer distribution with degrees of freedom
ν1 = I - 1 - 
and ν2 = (J -
1)ν1 . If I > 7 Pearson's criteria χ2 may be
used. The value of J(I - 1)W has a χ2 -distribution with
v = I - 1 degrees of freedom.
| Table 1: Group of experts with high degree of
agreement
|
|
|
Hazards
|
1footnote
|
2
|
3
|
4
|
5
|
6
|
7
|
8
|
9
|
10
|
| Experts
|
| 1
|
1
|
3
|
4
|
2
|
5
|
6
|
8
|
10
|
7
|
9
|
| 2
|
2
|
3
|
1
|
5
|
4
|
6
|
7
|
8
|
9
|
10
|
| 3
|
1
|
2
|
3
|
4
|
5
|
6
|
7
|
8
|
9
|
10
|
| 4
|
2
|
1
|
4
|
3
|
6
|
5
|
7
|
8
|
10
|
9
|
| 5
|
2
|
3
|
1
|
4
|
5
|
6
|
8
|
10
|
9
|
7
|
| 6
|
1
|
2
|
4
|
3
|
5
|
7
|
6
|
8
|
9
|
10
|
| Σxij
|
9
|
14
|
17
|
21
|
30
|
36
|
43
|
52
|
53
|
55
|
-
Calculations based on Table 1 result in W = 0,909; χ2 =
J(I - 1)W = 47,5 ; confidence level of probability ɑ =
0,999 .
| Table 2: Group of experts with medium degree of
agreement
|
|
|
Hazards
|
1
|
2
|
3
|
4
|
5
|
6
|
7
|
8
|
9
|
10
|
| Experts
|
| 1
|
1
|
6
|
8
|
4
|
2
|
3
|
5
|
7
|
9
|
10
|
| 2
|
2
|
3
|
1
|
5
|
6
|
4
|
7
|
8
|
10
|
9
|
| 3
|
3
|
4
|
1
|
2
|
5
|
8
|
9
|
10
|
6
|
7
|
| 4
|
4
|
5
|
6
|
1
|
8
|
2
|
3
|
10
|
7
|
9
|
| 5
|
4
|
3
|
1
|
9
|
2
|
5
|
7
|
10
|
6
|
8
|
| 6
|
5
|
1
|
7
|
4
|
3
|
9
|
8
|
2
|
10
|
6
|
| Σxij
|
19
|
23
|
24
|
25
|
26
|
31
|
39
|
47
|
48
|
49
|
-
Calculations based on the ranking in Table 2 result in W = 0,413; χ2 =
25.4 ; ɑ = 0,995 , where ɑ is the confidence level of probability.
|
|
|
|
|
|
Table 3: Group of experts with low degree of agreement
|
|
|
|
Hazards
|
1
|
2
|
3
|
4
|
5
|
6
|
7
|
8
|
9
|
10
|
| Experts
|
| 1
|
5
|
9
|
3
|
8
|
2
|
1
|
7
|
10
|
6
|
4
|
| 2
|
1
|
5
|
7
|
4
|
8
|
9
|
3
|
6
|
2
|
10
|
| 3
|
6
|
2
|
8
|
3
|
9
|
10
|
4
|
1
|
5
|
7
|
| 4
|
1
|
4
|
3
|
2
|
7
|
5
|
9
|
6
|
10
|
8
|
| 5
|
6
|
1
|
3
|
5
|
2
|
8
|
4
|
9
|
7
|
10
|
| 6
|
3
|
7
|
5
|
8
|
4
|
2
|
10
|
6
|
9
|
1
|
| Σxij
|
22
|
28
|
29
|
30
|
32
|
35
|
37
|
38
|
39
|
40
|
6 The level of agreement is characterized in table 4:
-
| Table 4: Concordance
coefficients
|
| W
|
> 0.7
|
Good agreement
|
| W
|
0.5 – 0.7
|
Medium agreement
|
| W
|
< 0.5
|
Poor agreement
|
Other use
7 The method described can be used in all cases where a group of experts are asked to
rank object according to one attribute using the natural numbers [1,I].
8 Generalizations of the method may be used when experts assign values to parameters,
when pair comparison methods are used, etc. David (1969), Kendall (1970). An FSA
application is published by Paliy et al. (2000).
References for further reading
1 David, H.A. The method of Paired Comparisons. Griffin and Co, London, 1969.
2 Kendall, M. Rank Correlation Methods. Griffin and Co, London, 1970.
3 Paliy, O., E. Litonov, V.I. Evenko. Formal Safety Assessment for Marine Drilling
Platforms. Proceedings Ice Tech' 2000, Saint Petersburg, 2000.