Generate Key Plots and Statistics Needed to Judge Model Effectiveness

Topic: Evaluate Objective: to generate key plots and statistics needed to judge a model's effectiveness • Lift Chart • Predicted vs Actual • Number of Bins (10) • Data Source: Validation, Cross-Validation, or Hold-out • Sort bins low to high • Y_label: Average target value • X_label: Bins based on predicted value • ROC Curve • ROC Curve Chart • X_label: False Positive Rate (Fallout) • Y_label: True Positive Rate (Sensitivity) • Ideal cut point should be shown • AUC statistic should be shown on chart • KS statistic should be shown on chart • Prediction Distribution (should be for both Density and Distribution) • Y_label: Probability Density • X_label: Provability of Event (range: 0.0 – 1.0) • Ideal threshold marked (range: 0.0 – 1.0) • Y_label: Probability Frequency • X_label: Provability of Event (range: 0.0 – 1.0) • Ideal threshold marked (range: 0.0 – 1.0) • Statistics • F1 Score • True Positive Rate (Sensitivity) • False Positive Rate (Fallout) • True Negative Rate (Specificity) • Positive Predictive Value (Precision) • Negative Predictive Value • Accuracy • Matthew’s Correlation Coefficient • Confusion Matrix • Chart of Actual against Predicted • Actual should be on left side of quadrant • Predicted should be on top I will send images to help make the above more clear.