Scenarios for Using Raters or Observers

There are different times where either the raters or the observers should be used to ensure that they gather any useful data that can be used to do constructive studies. The paper provides several examples of where this can be completely done. The first scenario is that the calculation to be performed comprises primarily of various groups. Here, the raters will be compelled to look at the divisions where each and every observation made will go. In order to thoroughly gauge this subject, it must be possible to approximate the percentage of the agreement between the various haters by means of estimates (Burda et al.,2017). A good example is a situation where two raters are to do a rating for almost one hundred observations.

In another scenario, a measure is used in continuous nature. Under this circumstance, one will be forced to calculate the correlations between the ratings that will be provided by the two witnesses. One good example is the case of a classroom where all the levels of activities taking place are rated on the scale of 1 to 10. At the same time, they may decide to give their rating at a constant interval of let say every 20 seconds. By establishing the correlations between the scores of the two raters, consistency will be fully gauged, as well as the level of data reliability (Burda et al., 2017).

The final scenario is where the observers are being calibrated by the reliability. This involves the whole process of encouraging the rate at which the observers are reliable. This is possible even without doing an estimation of the value. For instance, those working in the psychiatric unit where the nurses are assigned with the task of rating the patients who are the units (Baylis et al.,2015). Therefore, it is believed that each and every nurse will be able to give ratings which can be compared.

Statistical Procedures Involved in Evaluating Inter-Rater Reliability

Various methods can be used when it comes to ensuring the reliability of inter-rater. In this section, I will have a look at these three important procedures one by one. The first procedure is the use of joint probability of the agreement between these raters. Here, the process is very simple and experiences robust measurement (Baylis et al., 2015). It is done regarding the numbers of times the ratings have been assigned by every individual rater. This will then be divided by the sum of all the ratings done during that period. In this procedure, data that is to be collected is presumed to be nominal in nature.

The other procedure is the Kappa statistical method. This procedure works for two raters involving any number of raters available so long as it is a fixed number (Baylis et al., 2015). This process will take into account the amount or total summation of the agreements that is expected to arise in the case of chance.

Finally, the correlation coefficient procedure involves the utilization of coefficients that were first engineered by Spearman, Kendall, and Pearson. These can be used to measure pairwise correlation among various raters through the use of ordered scale. According to Burda et al. (2017), a scale used in rating is usually continuous. The other two believes and takes the assumption that the scale is ordinal. The real result will, therefore, be obtained concerning average agreement level.

References

Baylis, A., Chapman, K., Whitehill, T. L., & Group, T. A. S. (2015). Validity and reliability of visual analog scaling for assessment of hyper nasality and audible nasal emission in children with repaired cleft palate. The Cleft Palate-Craniofacial Journal, 52(6), 660-670.

Burda, B. U., O'Connor, E. A., Webber, E. M., Redmond, N., & Perdue, L. A. (2017). Estimating data from figures with a Web‐based program: Considerations for a systematic review. Research Synthesis Methods.