Training a Threshold Binary Classification Algorithm

Using Lowest Average Cost Per Event as the Cost Function

Training Set	base rate:	32%		Test Set	base rate	34%
1	0.16	0		101	0.73	0
2	0.67	0		102	0.45	0
3	0.41	0		103	0.85	1
4	0.51	1		104	0.95	0
5	0.90	1		105	0.01	0
6	0.93	1		106	0.20	0
7	0.27	0		107	0.04	0
8	0.39	1		108	0.65	1
9	0.25	0		109	0.31	0
10	0.05	0		110	0.85	0
11	0.34	1		111	0.70	0
12	0.27	0		112	0.08	0
13	0.54	0		113	0.77	1
14	0.87	1		114	0.33	0
15	0.21	0		115	0.56	0
16	0.11	0		116	0.88	1
17	0.31	0		117	0.83	0
18	0.56	0		118	0.30	0
19	0.53	1		119	0.08	0
20	0.06	0		120	0.05	0
21	0.00	0		121	0.13	0
22	0.88	1		122	0.30	0
23	0.97	1		123	0.62	1
24	0.56	1		124	0.62	1
25	0.36	0		125	0.80	1
26	0.02	0		126	0.07	0
27	0.79	0		127	0.99	1
28	0.95	1		128	0.73	1
29	0.15	0		129	0.98	0
30	0.10	0		130	0.76	0
31	0.29	0		131	0.71	0
32	0.84	0		132	0.84	1
33	0.48	0		133	0.56	0
34	0.68	1		134	0.43	1
35	0.78	1		135	0.75	1
36	0.69	1		136	0.07	0
37	0.05	0		137	0.85	0
38	0.74	0		138	0.86	1
39	0.22	0		139	0.48	0
40	0.94	1		140	0.28	0
41	0.84	1		141	0.73	0
42	0.48	0		142	0.76	0
43	0.54	0		143	0.00	1
44	0.71	1		144	0.23	1
45	0.86	0		145	0.85	1
46	0.45	0		146	0.26	0
47	0.50	1		147	0.54	1
48	0.27	0		148	0.62	0
49	0.65	1		149	0.76	0
50	0.92	0		150	0.04	0
51	0.90	1		151	0.77	0
52	0.97	0		152	0.76	1
53	0.96	1		153	0.62	1
54	0.07	0		154	0.72	1
55	0.10	1		155	0.89	1
56	0.20	1		156	0.44	0
57	0.46	0		157	0.28	0
58	0.89	0		158	0.93	0
59	0.38	0		159	0.12	0
60	0.51	0		160	0.70	0
61	0.36	0		161	0.01	0
62	0.52	1		162	0.83	1
63	0.05	0		163	0.51	0
64	0.98	0		164	0.93	1
65	0.26	1		165	0.46	0
66	0.65	0		166	0.58	0
67	0.70	0		167	0.61	1
68	0.17	0		168	0.15	0
69	0.35	0		169	0.20	0
70	0.63	1		170	0.70	1
71	0.29	0		171	0.35	0
72	0.18	0		172	0.68	1
73	0.28	0		173	0.23	0
74	0.20	0		174	0.44	1
75	0.53	0		175	0.05	1
76	0.08	0		176	0.27	0
77	0.95	0		177	0.99	1
78	0.50	0		178	0.66	0
79	0.17	0		179	0.30	0
80	0.38	1		180	0.58	0
81	0.38	0		181	0.47	0
82	0.07	0		182	0.21	0
83	0.35	1		183	0.58	1
84	0.82	0		184	0.41	0
85	0.91	1		185	0.68	0
86	0.39	0		186	0.16	0
87	0.44	0		187	0.85	0
88	0.25	0		188	0.61	0
89	0.19	0		189	0.43	0
90	0.21	0		190	0.22	0
91	0.98	0		191	0.82	0
92	0.24	0		192	0.90	1
93	0.72	0		193	0.44	0
94	0.40	0		194	0.47	0
95	0.79	1		195	0.99	1
96	0.73	1		196	0.46	0
97	0.11	0		197	0.92	1
98	0.11	0		198	0.51	0
99	0.80	1		199	0.84	0
100	0.06	0		200	1.00	1

Assume the diagnostic threshold cannot be changed, but new data suggests a better estimate for costs is $10,000 per False Negative and $2,000 per False Positive.

15. What is the new Cost Per Event at the old threshold with the new cost function on the Test Data?

16. Assume that you can set a new threshold based on the new cost function. What is the new optimal threshold on the Training data? What is the cost per event for training data?

17. Apply the new Threshold to the test Data. What is the new Cost Per Event?

Assume the diagnostic threshold cannot be changed, but new data suggests a better estimate for costs is $10,000 per False Negative and $2,000 per False Positive. Q15 What is the new Cost Per Event at the old threshold with the new cost function on the Test Data? Q16 Assume that you can set a new threshold based on the new cost function. What is the new optimal threshold on the Training data? Cost per event on Training Data Q17 Apply the new Threshold to the test Data. What is the new Cost Per Event