R: Data Management Show Missing Patterns

Summary

In this lecture, the speaker explores the handling of missing data in data analysis using R. They demonstrate several packages, such as 'foreign' for reading data and 'Dix tools' for visualizing missing data patterns. The speaker explains functions like 'plot_missing' and 'count_comp_cases' to identify and analyze missing data. They also introduce the 'Mi' package for multiple imputation and explore its functions, including missing data patterns and variable types. The overall focus is on effectively identifying and managing missing data for improved analysis.

Takeaways

• 📊 Missing data is very common in real-world data analysis and should be explicitly explored before modeling.
• 📦 R packages such as **foreign**, **distools**, and **mi** provide powerful tools for reading, inspecting, and visualizing missing data.
• 📥 The `read.dta()` function from the **foreign** package allows analysts to import external Stata data files into R.
• ✂️ Subsetting data (e.g., selecting specific variables and the first 50 cases) can simplify demonstrations and exploratory analysis.
• 👀 Visualizing missingness with `plot_miss()` helps identify which variables and cases contain missing values and how frequent they are.
• 📈 Summary statistics alongside plots reveal both the count and percentage of missing values for each variable.
• ✅ Functions like `complete_cols()` quickly identify variables that have no missing values at all.
• 🧮 The `count_complete_cases()` function provides detailed insights into how many observations remain under listwise deletion.
• ⚠️ Listwise deletion can substantially reduce sample size, depending on the pattern and amount of missing data.
• 🔄 Dropping different variables affects the number of complete cases differently, highlighting trade-offs in data preparation.
• 🧠 The **mi** package (by Andrew Gelman and colleagues) offers advanced tools for examining missing data structures and preparing for imputation.
• 🖼️ Visual tools like `image()` in the **mi** package cluster observations by missingness patterns, making complex structures easier to see.

Q & A

What is the purpose of the 'foreign' package in this script?

The 'foreign' package is used to import data from external formats, specifically to read a Stata data file using the `read.dta()` function.

What does the `subset()` function do in the context of the script?

The `subset()` function is used to select specific cases and variables from the dataset. In this script, it is used to create a subset containing the first 50 cases for easier exploration.

What is the purpose of the `plotMissing()` function from the VIM package?

The `plotMissing()` function is used to visualize missing data patterns. It creates a plot with variables on the y-axis and case numbers on the x-axis, showing which cases are missing for each variable.

What can we infer from the 'happy' variable's missing data in the plot?

The 'happy' variable has 16 missing cases, which account for 32% of the total data for that variable, as seen in the plot.

How does the `complete.cases()` function help with missing data?

The `complete.cases()` function identifies which columns (variables) do not have any missing data, making it easy to find variables with complete data.

What does the `countCompCases()` function tell us?

The `countCompCases()` function provides detailed information about missing data patterns. It shows how many complete cases (rows without missing values) are present and how the missing data is distributed across different variables.

How does the dropping of variables affect the number of complete cases?

When a variable with missing data is dropped, the number of complete cases increases. For example, dropping the 'happy' variable increases the complete cases from 33 to 49, as missing data is excluded.

What is the significance of the 'mi' package in this script?

The 'mi' package, developed by Andrew Gelman and colleagues, is used to explore and handle missing data. In this script, it helps generate a `missing.data.frame` object, which provides insights into the patterns of missing data for each variable.

What does the `image()` function do in the context of the 'mi' package?

The `image()` function in the 'mi' package visualizes missing data patterns by clustering observations based on missingness, allowing for a clearer understanding of how missing data is distributed across variables.

What is recommended for users when exploring missing data using these functions?

It is recommended to explore the documentation for each function and experiment with them. Trying out different options will provide deeper insights into the missing data patterns and help make informed decisions on handling missing data.