CASP8 QA Evaluation

Preliminary Evaluation of CASP8 Quality Assessment (QA) Predictors on 120 Targets

The automated assessment is for informational purpose only. The results may be different from the official CASP8 results. The other issue is that different kinds of QA methods (clustering-based methods or single-model methods) are not distinguished. In general, clustering-based methods performed better than single-model methods during CASP8.

QA predictors are ranked by three measures: average correlation across all the targets (mean per-target correlation), the total GDT-TS scores of top 1 ranked models, and the overall correlation on all the models (global correlation). All the predictors except MULTICOM are server predictors. MULTICOM is an automated human QA predictor.

1. QA predictors ranked by the average correlation on 120 CASP8 targets. The correlation of a target for a QA predictor is the Pearson correlation between predicted quality scores and true GDT-TS scores of the models associated with the target.

RANK	Predictor	Average Correlation	Num of Targets
1	Pcons_Pcons	0.922	115
2	ModFOLDclust	0.914	113
3	QMEANclust	0.899	118
4	MULTICOM	0.899	119
5	LEE-SERVER	0.894	59
6	MULTICOM-CLUSTER	0.885	120
7	GS-MetaMQAPconsI	0.88	117
8	LEE-SERVER	0.878	74
9	LEE-SERVER	0.877	74
10	GS-MetaMQAPconsII	0.857	116
11	selfQMEAN	0.848	114
12	FAMSD	0.842	118
13	MULTICOM	0.8	110
14	Mariner2	0.754	89
15	MULTICOM-CMFR	0.742	120
16	MULTICOM-REFINE	0.724	120
17	QMEAN	0.723	114
18	circle	0.7	120
19	GS-MetaMQAP	0.699	119
20	QMEANfamily	0.697	102
21	MULTICOM-RANK	0.684	120
22	SIFT_consensus	0.679	107
23	Pcons_ProQ	0.671	115
24	MUFOLD-QA	0.646	113
25	SIFT_SA	0.638	102
26	SELECTpro	0.627	113
27	ModFOLD	0.609	112
28	Fiser-QA	0.544	111
29	Fiser-QA-FA	0.528	110
30	Fiser-QA-FA	0.504	111
31	Fiser-QA-COMB	0.496	111
32	MODCHECK-HD	0.299	112
33	ProtAnG_s	0.139	119
34	qa-ms-torda-server	0.012	107

2. QA predictors ranked by the sum of the GDT-TS scores of the top 1 ranked models on 120 CASP8 targets. The ranking is not fair because the number of targets for a QA predictor is different.

RANK	Predictor	Sum of GDT-TS scores	Num of Targets	Average GDT-TS Score
1	MULTICOM	75.506	119	0.635
2	MULTICOM-CLUSTER	74.1242	120	0.618
3	FAMSD	73.8977	118	0.626
4	QMEANclust	73.7119	118	0.625
5	MULTICOM-RANK	73.262	120	0.611
6	MULTICOM-CMFR	73.2081	120	0.61
7	Pcons_Pcons	72.8543	115	0.634
8	MULTICOM-REFINE	72.0494	120	0.6
9	ModFOLDclust	72.013	113	0.637
10	GS-MetaMQAPconsI	71.9174	117	0.615
11	GS-MetaMQAPconsII	70.8302	116	0.611
12	circle	70.1177	120	0.584
13	selfQMEAN	69.9194	114	0.613
14	QMEAN	69.8224	115	0.607
15	MULTICOM	69.1316	110	0.628
16	ProtAnG_s	66.5549	119	0.559
17	GS-MetaMQAP	66.4813	119	0.559
18	MUFOLD-QA	65.7855	113	0.582
19	Pcons_ProQ	65.6994	115	0.571
20	SIFT_consensus	63.6313	107	0.595
21	ModFOLD	62.371	112	0.557
22	SELECTpro	61.7492	113	0.546
23	MODCHECK-HD	61.3616	112	0.548
24	QMEANfamily	61.1752	102	0.6
25	SIFT_SA	59.7328	102	0.586
26	Fiser-QA	56.9159	111	0.513
27	Fiser-QA-FA	56.0917	110	0.51
28	Fiser-QA-FA	56.086	111	0.505
29	Fiser-QA-COMB	52.6274	111	0.474
30	LEE-SERVER	51.0011	74	0.689
31	LEE-SERVER	50.6673	74	0.685
32	Mariner2	48.3429	89	0.543
33	LEE-SERVER	40.2993	59	0.683
34	qa-ms-torda-server	20.1685	107	0.188

3. QA predictors ranked by the overal correlation on all the models associated with 120 targets.

RANK	Predictor	Overal Correlation	Num of Targets
1	ModFOLDclust	0.919	113
2	QMEANclust	0.918	118
3	Pcons_Pcons	0.917	115
4	MULTICOM	0.916	119
5	MULTICOM-CLUSTER	0.903	120
6	selfQMEAN	0.893	114
7	GS-MetaMQAPconsI	0.887	117
8	Mariner2	0.877	89
9	LEE-SERVER	0.865	59
10	GS-MetaMQAPconsII	0.861	116
11	MULTICOM	0.857	110
12	LEE-SERVER	0.823	74
13	LEE-SERVER	0.82	74
14	GS-MetaMQAP	0.797	119
15	MULTICOM-REFINE	0.794	120
16	QMEAN	0.772	115
17	MULTICOM-CMFR	0.757	120
18	QMEANfamily	0.755	102
19	MULTICOM-RANK	0.727	120
20	ModFOLD	0.708	112
21	SIFT_consensus	0.702	107
22	Pcons_ProQ	0.686	115
23	FAMSD	0.681	118
24	circle	0.677	120
25	MUFOLD-QA	0.593	113
26	Fiser-QA	0.529	111
27	MODCHECK-HD	0.509	112
28	Fiser-QA-COMB	0.499	111
29	SIFT_SA	0.488	102
30	SELECTpro	0.474	113
31	Fiser-QA-FA	0.321	110
32	Fiser-QA-FA	0.296	111
33	ProtAnG_s	0.097	119
34	qa-ms-torda-server	0.095	107

4. QA predictors ranked by the average loss (loss is defined as the difference between the GDT-TS score of the best model and the GDT-TS score of the topped ranked model.) based on 120 CASP8 targets.

RANK	Predictor	Average Loss	Num of Targets
1	MULTICOM	0.0483890756302521	119
2	ModFOLDclust	0.0520814159292035	113
3	Pcons_Pcons	0.0528417391304348	115
4	MULTICOM	0.0538790909090909	110
5	FAMSD	0.0574974576271186	118
6	QMEANclust	0.0594491525423729	118
7	MULTICOM-CLUSTER	0.0618525	120
8	LEE-SERVER	0.0626432432432433	74
9	LEE-SERVER	0.0670440677966102	59
10	LEE-SERVER	0.0671540540540541	74
11	selfQMEAN	0.0679587719298246	114
12	MULTICOM-RANK	0.068175	120
13	GS-MetaMQAPconsI	0.0689589743589744	117
14	MULTICOM-CMFR	0.0730833333333333	120
15	GS-MetaMQAPconsII	0.0731784482758621	116
16	QMEANfamily	0.078656862745098	102
17	QMEAN	0.0795833333333333	114
18	MULTICOM-REFINE	0.0835691666666667	120
19	SIFT_consensus	0.0949644859813084	107
20	circle	0.09524	120
21	SIFT_SA	0.100549019607843	102
22	MUFOLD-QA	0.10719203539823	113
23	Pcons_ProQ	0.114913913043478	115
24	Mariner2	0.121093258426966	89
25	GS-MetaMQAP	0.126606722689076	119
26	ProtAnG_s	0.127576470588235	119
27	ModFOLD	0.131373214285714	112
28	SELECTpro	0.137299115044248	113
29	MODCHECK-HD	0.140385714285714	112
30	Fiser-QA	0.176516216216216	111
31	Fiser-QA-FA	0.177332727272727	110
32	Fiser-QA-FA	0.183992792792793	111
33	Fiser-QA-COMB	0.215151351351351	111
34	qa-ms-torda-server	0.492553271028037	107

If you have any comments or questions, please contact Dr. Jianlin Cheng at chengji@missouri.edu or Zheng Wang at zwyw6@mizzou.edu.

Last modified on Sep 29th, 2008.