Preliminary Evaluation of CASP8 Quality Assessment (QA) Predictors on 120 Targets

The automated assessment is for informational purpose only. The results may be different from the official CASP8 results. The other issue is that different kinds of QA methods (clustering-based methods or single-model methods) are not distinguished. In general, clustering-based methods performed better than single-model methods during CASP8.

QA predictors are ranked by three measures: average correlation across all the targets (mean per-target correlation), the total GDT-TS scores of top 1 ranked models, and the overall correlation on all the models (global correlation). All the predictors except MULTICOM are server predictors. MULTICOM is an automated human QA predictor.


1. QA predictors ranked by the average correlation on 120 CASP8 targets. The correlation of a target for a QA predictor is the Pearson correlation between predicted quality scores and true GDT-TS scores of the models associated with the target.

RANKPredictorAverage CorrelationNum of Targets
1Pcons_Pcons0.922115
2ModFOLDclust0.914113
3QMEANclust0.899118
4MULTICOM0.899119
5LEE-SERVER0.89459
6MULTICOM-CLUSTER0.885120
7GS-MetaMQAPconsI0.88117
8LEE-SERVER0.87874
9LEE-SERVER0.87774
10GS-MetaMQAPconsII0.857116
11selfQMEAN0.848114
12FAMSD0.842118
13MULTICOM0.8110
14Mariner20.75489
15MULTICOM-CMFR0.742120
16MULTICOM-REFINE0.724120
17QMEAN0.723114
18circle0.7120
19GS-MetaMQAP0.699119
20QMEANfamily0.697102
21MULTICOM-RANK0.684120
22SIFT_consensus0.679107
23Pcons_ProQ0.671115
24MUFOLD-QA0.646113
25SIFT_SA0.638102
26SELECTpro0.627113
27ModFOLD0.609112
28Fiser-QA0.544111
29Fiser-QA-FA0.528110
30Fiser-QA-FA0.504111
31Fiser-QA-COMB0.496111
32MODCHECK-HD0.299112
33ProtAnG_s0.139119
34qa-ms-torda-server0.012107

 

 

2. QA predictors ranked by the sum of the GDT-TS scores of the top 1 ranked models on 120 CASP8 targets. The ranking is not fair because the number of targets for a QA predictor is different.

RANKPredictorSum of GDT-TS scoresNum of TargetsAverage GDT-TS Score
1MULTICOM75.5061190.635
2MULTICOM-CLUSTER74.12421200.618
3FAMSD73.89771180.626
4QMEANclust73.71191180.625
5MULTICOM-RANK73.2621200.611
6MULTICOM-CMFR73.20811200.61
7Pcons_Pcons72.85431150.634
8MULTICOM-REFINE72.04941200.6
9ModFOLDclust72.0131130.637
10GS-MetaMQAPconsI71.91741170.615
11GS-MetaMQAPconsII70.83021160.611
12circle70.11771200.584
13selfQMEAN69.91941140.613
14QMEAN69.82241150.607
15MULTICOM69.13161100.628
16ProtAnG_s66.55491190.559
17GS-MetaMQAP66.48131190.559
18MUFOLD-QA65.78551130.582
19Pcons_ProQ65.69941150.571
20SIFT_consensus63.63131070.595
21ModFOLD62.3711120.557
22SELECTpro61.74921130.546
23MODCHECK-HD61.36161120.548
24QMEANfamily61.17521020.6
25SIFT_SA59.73281020.586
26Fiser-QA56.91591110.513
27Fiser-QA-FA56.09171100.51
28Fiser-QA-FA56.0861110.505
29Fiser-QA-COMB52.62741110.474
30LEE-SERVER51.0011740.689
31LEE-SERVER50.6673740.685
32Mariner248.3429890.543
33LEE-SERVER40.2993590.683
34qa-ms-torda-server20.16851070.188

 

3. QA predictors ranked by the overal correlation on all the models associated with 120 targets.

RANKPredictorOveral CorrelationNum of Targets
1ModFOLDclust0.919113
2QMEANclust0.918118
3Pcons_Pcons0.917115
4MULTICOM0.916119
5MULTICOM-CLUSTER0.903120
6selfQMEAN0.893114
7GS-MetaMQAPconsI0.887117
8Mariner20.87789
9LEE-SERVER0.86559
10GS-MetaMQAPconsII0.861116
11MULTICOM0.857110
12LEE-SERVER0.82374
13LEE-SERVER0.8274
14GS-MetaMQAP0.797119
15MULTICOM-REFINE0.794120
16QMEAN0.772115
17MULTICOM-CMFR0.757120
18QMEANfamily0.755102
19MULTICOM-RANK0.727120
20ModFOLD0.708112
21SIFT_consensus0.702107
22Pcons_ProQ0.686115
23FAMSD0.681118
24circle0.677120
25MUFOLD-QA0.593113
26Fiser-QA0.529111
27MODCHECK-HD0.509112
28Fiser-QA-COMB0.499111
29SIFT_SA0.488102
30SELECTpro0.474113
31Fiser-QA-FA0.321110
32Fiser-QA-FA0.296111
33ProtAnG_s0.097119
34qa-ms-torda-server0.095107

 

 

4. QA predictors ranked by the average loss (loss is defined as the difference between the GDT-TS score of the best model and the GDT-TS score of the topped ranked model.) based on 120 CASP8 targets.

RANKPredictorAverage LossNum of Targets
1MULTICOM0.0483890756302521119
2ModFOLDclust0.0520814159292035113
3Pcons_Pcons0.0528417391304348115
4MULTICOM0.0538790909090909110
5FAMSD0.0574974576271186118
6QMEANclust0.0594491525423729118
7MULTICOM-CLUSTER0.0618525120
8LEE-SERVER0.062643243243243374
9LEE-SERVER0.067044067796610259
10LEE-SERVER0.067154054054054174
11selfQMEAN0.0679587719298246114
12MULTICOM-RANK0.068175120
13GS-MetaMQAPconsI0.0689589743589744117
14MULTICOM-CMFR0.0730833333333333120
15GS-MetaMQAPconsII0.0731784482758621116
16QMEANfamily0.078656862745098102
17QMEAN0.0795833333333333114
18MULTICOM-REFINE0.0835691666666667120
19SIFT_consensus0.0949644859813084107
20circle0.09524120
21SIFT_SA0.100549019607843102
22MUFOLD-QA0.10719203539823113
23Pcons_ProQ0.114913913043478115
24Mariner20.12109325842696689
25GS-MetaMQAP0.126606722689076119
26ProtAnG_s0.127576470588235119
27ModFOLD0.131373214285714112
28SELECTpro0.137299115044248113
29MODCHECK-HD0.140385714285714112
30Fiser-QA0.176516216216216111
31Fiser-QA-FA0.177332727272727110
32Fiser-QA-FA0.183992792792793111
33Fiser-QA-COMB0.215151351351351111
34qa-ms-torda-server0.492553271028037107

 

If you have any comments or questions, please contact Dr. Jianlin Cheng at chengji@missouri.edu or Zheng Wang at zwyw6@mizzou.edu.

Last modified on Sep 29th, 2008.