Tabular data preprocessing¶
Overview¶
This package contains the basic class to define a transformation for preprocessing dataframes of tabular data, as well as basic TabularProc. Preprocessing includes things like
- replacing non-numerical variables by categories, then their ids,
- filling missing values,
- normalizing continuous variables.
In all those steps we have to be careful to use the correspondence we decide on our training set (which id we give to each category, what is the value we put for missing data, or how the mean/std we use to normalize) on our validation or test set. To deal with this, we use a special class called TabularProc.
The data used in this document page is a subset of the adult dataset. It gives a certain amount of data on individuals to train a model to predict whether their salary is greater than \$50k or not.
path = untar_data(URLs.ADULT_SAMPLE)
df = pd.read_csv(path/'adult.csv')
train_df, valid_df = df.iloc[:800].copy(), df.iloc[800:1000].copy()
train_df.head()
We see it contains numerical variables (like age or education-num) as well as categorical ones (like workclass or relationship). The original dataset is clean, but we removed a few values to give examples of dealing with missing variables.
cat_names = ['workclass', 'education', 'marital-status', 'occupation', 'relationship', 'race', 'sex', 'native-country']
cont_names = ['age', 'fnlwgt', 'education-num', 'capital-gain', 'capital-loss', 'hours-per-week']
Transforms for tabular data¶
Base class for creating transforms for dataframes with categorical variables cat_names and continuous variables cont_names. Note that any column not in one of those lists won't be touched.
The following TabularProc are implemented in the fastai library. Note that the replacement from categories to codes as well as the normalization of continuous variables are automatically done in a TabularDataBunch.
Variables in cont_names aren't affected.
tfm = Categorify(cat_names, cont_names)
tfm(train_df)
tfm(valid_df, test=True)
Since we haven't changed the categories by their codes, nothing visible has changed in the dataframe yet, but we can check that the variables are now categorical and view their corresponding codes.
train_df['workclass'].cat.categories
The test set will be given the same category codes as the training set.
valid_df['workclass'].cat.categories
cat_names variables are left untouched (their missing value will be replaced by code 0 in the TabularDataBunch). fill_strategy is adopted to replace those nans and if add_col is True, whenever a column c has missing values, a column named c_nan is added and flags the line where the value was missing.
Fills the missing values in the cont_names columns with the ones picked during train.
train_df[cont_names].head()
tfm = FillMissing(cat_names, cont_names)
tfm(train_df)
tfm(valid_df, test=True)
train_df[cont_names].head()
Values missing in the education-num column are replaced by 10, which is the median of the column in train_df. Categorical variables are not changed, since nan is simply used as another category.
valid_df[cont_names].head()
norm = Normalize(cat_names, cont_names)
norm.apply_train(train_df)
train_df[cont_names].head()
norm.apply_test(valid_df)
valid_df[cont_names].head()
Treating date columns¶
Will drop the column in df if the flag is True. The time flag decides if we go down to the time parts or stick to the date parts.
df = pd.DataFrame({'col1': ['02/03/2017', '02/04/2017', '02/05/2017'], 'col2': ['a', 'b', 'a']})
add_datepart(df, 'col1') # inplace
df.head()
show_doc(add_cyclic_datepart)
df = pd.DataFrame({'col1': ['02/03/2017', '02/04/2017', '02/05/2017'], 'col2': ['a', 'b', 'a']})
df = add_cyclic_datepart(df, 'col1') # returns a dataframe
df.head()
Splitting data into cat and cont¶
Parameters:
- df: A pandas data frame.
- max_card: Maximum cardinality of a numerical categorical variable.
- dep_var: A dependent variable.
Return:
- cont_names: A list of names of continuous variables.
- cat_names: A list of names of categorical variables.
df = pd.DataFrame({'col1': [1, 2, 3], 'col2': ['a', 'b', 'a'], 'col3': [0.5, 1.2, 7.5], 'col4': ['ab', 'o', 'o']})
df
cont_list, cat_list = cont_cat_split(df=df, max_card=20, dep_var='col4')
cont_list, cat_list