Data Science Methodology - From Understanding to Preparation and From Modeling to Evaluation (2)

Data Science Methodology

From Modeling to Evaluation

Modeling - Concepts

From Modeling to Evaluation

- Modeling : In what way can the data be visualized to get to the answer that is required?

- Evaluation : Does the model used really answer the initial question or does it need to be adjusted?

 

Data Modeling : Using Predictive or Descriptive?

 

Data Modeling : Using training/test sets

 

Understand the Question

1. understand the question at hand

2. select an analytic approach or method to solve the problem

3. obtain, understand, prepare, and model the data

 

Modeling - Case Study

Case Study : Analyzing the first model

- Initial decision tree classification model : low accuracy on "yes" outcome

 

Case Study : How to improve the model?

 

Case Study : Analyzing the second model

- second model : high accuracy on "yes" but poor on "No"

 

Case Study : Analyzing the third model

- third model : better balance on "yes" and "no" accuracy

 

Evaluation

When and not to adjust the model?

 

Case Study : Applying the concepts

misclassification cost tuning :

- tune the relative misclassificaion costs

- balance true-positive rate and false-positive rate for best model

 

Case study : Relative costs

 

Case Study : true-positives vs false-positives

 

Case Study : Using the ROC curve (ROC : Receiver Operating Characteristic. 적중확률 : 오경보확률)

Diagnostic tool for classification model evaluation:

- classification model performance

- True-Positive Rate vs False-Positive Rate

- optimal model at maximum separation

 

Practice Quiz Lesson 2 : From Modeling to Evaluation

Q. Which statement best describes the Modeling Stage of the data science methodology?

A. Modeling always requires testing multiple algorithms and parameters

 

Q. The Evaluation stage, or Modeling Evaluation, takes place before sharing the model with stakeholders and other users.

A. True

 

Q. Select the three correct statements about the Evaluation stage of the data science methodology.

A. Model Evaluaton ensures the data are correctly handled and interpreted.

    Model Evaluation includes validating that the model is designed as intended.

    Evaluating the data model includes ensuring that the model processes the data as intended.