Lido Blog: 機器學習百日馬拉松 - Day 5. EDA 資料分佈

Day5 作業:可參考 my Github - Day_005_HW.ipynb

Day 5 主要是練習數據的可視化，在大數據中，不可能一筆一筆資料去讀，可以的話，將資料畫成有用圖表，對分析資料結構會有更快的了解，看圖說故事也是我們最擅長的(竊笑)。

python 常用的數據可視化工具 : matplotlib & seaborn















# Import 需要的套件
import os
import numpy as np
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
%matplotlib inline
# 設定 data_path
dir_data = './data/'
f_app_train = os.path.join(dir_data, 'application_train.csv')
app_train = pd.read_csv(f_app_train)

這次作業主要是要練習 Pandas 的數值計算 ex. max(),min(),mean(),median()…等等，所以我們挑出資料中 float64 的資料型態的資料來練習數值計算的操作，最後把圖畫出來看看。






#挑出 datatype 為 float 的資料
app_train.select_dtypes(['float']).columns

Index(['AMT_INCOME_TOTAL', 'AMT_CREDIT', 'AMT_ANNUITY', 'AMT_GOODS_PRICE',
       'REGION_POPULATION_RELATIVE', 'DAYS_REGISTRATION', 'OWN_CAR_AGE',
       'CNT_FAM_MEMBERS', 'EXT_SOURCE_1', 'EXT_SOURCE_2', 'EXT_SOURCE_3',
       'APARTMENTS_AVG', 'BASEMENTAREA_AVG', 'YEARS_BEGINEXPLUATATION_AVG',
       'YEARS_BUILD_AVG', 'COMMONAREA_AVG', 'ELEVATORS_AVG', 'ENTRANCES_AVG',
       'FLOORSMAX_AVG', 'FLOORSMIN_AVG', 'LANDAREA_AVG',
       'LIVINGAPARTMENTS_AVG', 'LIVINGAREA_AVG', 'NONLIVINGAPARTMENTS_AVG',
       'NONLIVINGAREA_AVG', 'APARTMENTS_MODE', 'BASEMENTAREA_MODE',
       'YEARS_BEGINEXPLUATATION_MODE', 'YEARS_BUILD_MODE', 'COMMONAREA_MODE',
       'ELEVATORS_MODE', 'ENTRANCES_MODE', 'FLOORSMAX_MODE', 'FLOORSMIN_MODE',
       'LANDAREA_MODE', 'LIVINGAPARTMENTS_MODE', 'LIVINGAREA_MODE',
       'NONLIVINGAPARTMENTS_MODE', 'NONLIVINGAREA_MODE', 'APARTMENTS_MEDI',
       'BASEMENTAREA_MEDI', 'YEARS_BEGINEXPLUATATION_MEDI', 'YEARS_BUILD_MEDI',
       'COMMONAREA_MEDI', 'ELEVATORS_MEDI', 'ENTRANCES_MEDI', 'FLOORSMAX_MEDI',
       'FLOORSMIN_MEDI', 'LANDAREA_MEDI', 'LIVINGAPARTMENTS_MEDI',
       'LIVINGAREA_MEDI', 'NONLIVINGAPARTMENTS_MEDI', 'NONLIVINGAREA_MEDI',
       'TOTALAREA_MODE', 'OBS_30_CNT_SOCIAL_CIRCLE',
       'DEF_30_CNT_SOCIAL_CIRCLE', 'OBS_60_CNT_SOCIAL_CIRCLE',
       'DEF_60_CNT_SOCIAL_CIRCLE', 'DAYS_LAST_PHONE_CHANGE',
       'AMT_REQ_CREDIT_BUREAU_HOUR', 'AMT_REQ_CREDIT_BUREAU_DAY',
       'AMT_REQ_CREDIT_BUREAU_WEEK', 'AMT_REQ_CREDIT_BUREAU_MON',
       'AMT_REQ_CREDIT_BUREAU_QRT', 'AMT_REQ_CREDIT_BUREAU_YEAR'],
      dtype='object')

將這些資料逐個打印出來看看，挑選覺得有興趣的資料來做計算。





app_train[app_train.columns[0:20]].select_dtypes(['float']).head()





app_train[app_train.columns[20:50]].select_dtypes(['float']).head()





app_train[app_train.columns[50:70]].select_dtypes(['float']).head()

將覺得有趣的 columns 挑出，建立新的 df，有下列三種方法，參考 Pandas user guide














#方法一
cap_data = []
cap_data.append(app_train['TARGET'])
cap_data.append(app_train['AMT_INCOME_TOTAL'])
cap_data.append(app_train['AMT_CREDIT'])
cap_data.append(app_train['AMT_ANNUITY'])
cap_data.append(app_train['AMT_GOODS_PRICE'])
cap_data.append(app_train['REGION_POPULATION_RELATIVE'])
cap_data.append(app_train['DAYS_REGISTRATION'])
new_app_train=pd.DataFrame(cap_data).T












#方法二
new_app_train_ = pd.concat([app_train['TARGET'],
                            app_train['AMT_INCOME_TOTAL'],
                            app_train['AMT_CREDIT'],
                            app_train['AMT_ANNUITY'],
                            app_train['AMT_GOODS_PRICE'],
                            app_train['REGION_POPULATION_RELATIVE'],
                            app_train['DAYS_REGISTRATION']],axis=1)











#方法三
new_app_train__ = pd.DataFrame(app_train['TARGET']).join(app_train['AMT_INCOME_TOTAL'])
new_app_train__ = new_app_train__.join(app_train['AMT_CREDIT'])
new_app_train__ = new_app_train__.join(app_train['AMT_ANNUITY'])
new_app_train__ = new_app_train__.join(app_train['AMT_GOODS_PRICE'])
new_app_train__ = new_app_train__.join(app_train['REGION_POPULATION_RELATIVE'])
new_app_train__ = new_app_train__.join(app_train['DAYS_REGISTRATION'])

建立新的 df 後，挑其中一項看一下內容前五列：





new_app_train__['AMT_INCOME_TOTAL'].head()

0    202500.0
1    270000.0
2     67500.0
3    135000.0
4    121500.0
Name: AMT_INCOME_TOTAL, dtype: float64

計算挑出欄位的平均數及標準差









for column in new_app_train__.columns:
    print("mean of "+str(column)+" : ",new_app_train[column].mean())

for column in new_app_train__.columns:
    print("standard deviation of  "+str(column)+" : ",new_app_train[column].std())

mean of TARGET :  0.08072881945686496
mean of AMT_INCOME_TOTAL :  168797.9192969845
mean of AMT_CREDIT :  599025.9997057016
mean of AMT_ANNUITY :  27108.573909183444
mean of AMT_GOODS_PRICE :  538396.2074288895
mean of REGION_POPULATION_RELATIVE :  0.020868112057780042
mean of DAYS_REGISTRATION :  -4986.120327538419
standard deviation of  TARGET :  0.27241864564839396
standard deviation of  AMT_INCOME_TOTAL :  237123.14627885626
standard deviation of  AMT_CREDIT :  402490.77699585486
standard deviation of  AMT_ANNUITY :  14493.737315118333
standard deviation of  AMT_GOODS_PRICE :  369446.46054005757
standard deviation of  REGION_POPULATION_RELATIVE :  0.0138312801227047
standard deviation of  DAYS_REGISTRATION :  3522.8863209630713

將這些欄位畫出直方圖：









for column in new_app_train__.columns:
    plt.figure(figsize=(10,6))
    new_app_train__[column].hist(bins=40,)
    plt.title(column)
    plt.show()