Hand out on Data Cleaning
Data Cleaning
Introduction to Data Cleaning
When data is described as "dirty," the dataset contains errors, inconsistencies, inaccuracies, or other issues that can hinder its
usability and reliability for analysis, reporting, or other purposes.
Data cleaning is the process of detecting and correcting (or removing) corrupt or inaccurate records from a dataset, ensuring the
quality of data.
Importance of Data Cleaning
1. Vital for accurate data analysis.
2. Prevents misleading results and conclusions.
3. Essential for operational efficiency and strategic decision-making
Common issues and challenges with data cleaning
1. Time-intensive: Requires significant effort and attention to detail.
2. Skill Requirement: Necessitates expertise in data handling and domain knowledge.
3. Risk of Distortion: Improper cleaning can lead to loss of data integrity.
Characteristics of Dirty Data
1. Incomplete Entries: Missing values or incomplete records within the dataset.
2. Inconsistent Formatting
Inconsistent formatting refers to variations or discrepancies in how data is presented or formatted within a dataset. It can occur
when data entries are not uniformly structured or formatted according to a predefined standard or convention. Inconsistent
formatting can manifest in several ways:
1. Different Date Formats: Dates may be represented in various formats (e.g., MM/DD/YYYY, DD/MM/YYYY, YYYY-MM-
DD), making it challenging to interpret and compare them accurately.
2. Numeric Formatting: Numeric values may have different decimal separators (e.g., periods or commas) or use different
units of measurement (e.g., currency symbols or percentages) inconsistently throughout the dataset.
3. Text Case: Textual data may have inconsistent capitalization (e.g., "Customer Name" vs. "customer name" vs. "CUSTOMER
NAME"), affecting searchability and readability.
4. Abbreviations and Acronyms: Abbreviations or acronyms may be used inconsistently or interchangeably within the
dataset, leading to confusion and misinterpretation.
5. Spacing and Punctuation: Inconsistent use of spaces, punctuation marks, or special characters within text fields can disrupt
data consistency and uniformity.
6. Categorical Variables: Categorical variables may have variations in spelling or naming conventions, leading to ambiguity
and difficulty in grouping or categorizing data.
3. Duplicate Entries: Multiple occurrences of the same data within the dataset.
4. Incorrect Data Types: Data stored in inappropriate formats or data types.
5. Inaccurate Values: Data entries that contain inaccuracies, typographical errors, or outdated information.
6. Outliers: Data points that significantly deviate from the norm or expected range.
7. Non-standardized Entries: Lack of standardization across data entries, making it challenging to analyze or compare. E.g. mixed
data types, inconsistent units, varied date format, mixed case formatting, etc.
8. Misaligned Data: Data that is not properly aligned or organized within the dataset, making it difficult to interpret e.g. data entry
error, merger, and centre.
Overall, dirty data poses challenges for data analysis, visualization, and interpretation, it can lead to incorrect conclusions, flawed
insights, and unreliable decision-making. It is crucial to address and clean dirty data to ensure its quality, integrity, and usability for
various applications.
FUNCTIONS AND FORMULAS FOR DATA CLEANING
Excel Functions for Data Cleaning
1. TRIM for removing extra spaces.
2. CLEAN for non-printable characters
3. CONCATENATE and TEXT TO COLUMNS for organizing data.
4. Find and Replace- to search for specific text strings and replace them with other values.
5. IFERROR or ISERROR - to identify and handle errors in data entries.
6. VALUE, DATEVALUE, or TEXT- Convert data types (e.g., text to numbers, dates to proper date format)
7. SUBSTITUTE or REPLACE -to remove unwanted characters.
8. ISBLANK Function: Determines if a cell is empty.
Function Uses Example
TRIM Removes extra spaces from text, except for single spaces =TRIM(A1)
between words.
LOWER /UPPER / PROPER Converts text to lowercase / uppercase / proper case. Lowercase: =LOWER(A1)
Uppercase: =UPPER(A1)
Proper Case: =PROPER(A1)
LEFT / RIGHT / MID Extracts characters from a text string. Left: =LEFT(A1, 5) (Extracts the
first 5 characters)
Right: =RIGHT(A1, 5) (Extracts the
last 5 characters)
Mid: =MID(A1, 3, 5) (Extracts 5
characters starting from the 3rd
position)
CONCATENATE / CONCAT Joins multiple text strings into one. =CONCATENATE(A1, " ", B1)
=CONCAT(A1, " ", B1)
FIND/SEARCH Finds one text value within another (casesensitive / =FIND("text", A1)
caseinsensitive).
=SEARCH("text", A1)
LEN Returns the number of characters in a text string. =LEN(A1)
ISNUMBER Checks whether a value is a number. =ISNUMBER(A1)
IFERROR Returns a value you specify if a formula evaluates to an =IFERROR(A1, "Error")
error; otherwise, returns the result of the formula.
SUBSTITUTE Replaces existing text with new text in a text string. =SUBSTITUTE(A1, "old", "new")
CLEAN Removes all nonprintable characters from text. =CLEAN(A1)
VALUE Converts a text argument to a number. =VALUE(A1)
Text to Columns Splits data in a single column into multiple columns. Go to 'Data' > 'Text to Columns'.
Remove Duplicate Removes duplicate values from a range of cells. Go to 'Data' > 'Remove Duplicates'
Conditional Formatting Highlight duplicate or unique values, cells that contain Go to 'Home' > 'Conditional
specific words, etc. Formatting'.
Filter Filters data inplace within the range. Go to 'Data' > 'Filter'.
Sorts Sorts the data in a range of cells. Go to 'Data' > 'Sort'.
VLOOKUP Searches for a value in the first column of a table array and =VLOOKUP(A1, Table, 2, FALSE)
returns a value in the same row from another column you
specify.
INDEX and MATCH A more flexible and powerful alternative to VLOOKUP. =INDEX(ColumnToReturn,
MATCH(LookupValue,
LookupColumn, 0))
ISBLANK Checks whether a reference is to an empty cell. =ISBLANK(A1)
SUBSTITUTE Removes all spaces from a string. =SUBSTITUTE(A1, " ", "")
UNIQUE Returns a list of unique values in a range. =UNIQUE(A1:A10)
Advanced Tools
Power Query for complex cleaning tasks
Pivot Tables for summarizing and analyzing.
Managing Missing Values
IF Function: Replaces missing values with a specified value or text.
Syntax: =IF(condition, value_if_true, value_if_false)
Example: =IF(ISBLANK(A1), Missing, A1) replaces empty cells in A1 with the text Missing.
IFERROR Function: Catches errors resulting from calculations and replaces them with a specified value.
1. Syntax: =IFERROR(value, value_if_error)
2. Example: =IFERROR(1/A1, Missing) returns Missing if dividing 1 by A1 results in an error (e.g., if A1 is blank).
FILTER Function: Excludes missing values from a range or array.
1. Syntax: =FILTER(array, include, [if_empty])
2. Example: =FILTER(A1:A10, NOT(ISBLANK(A1:A10)), All Missing) returns all non-blank values from A1 to A10, and All
Missing if all values are blank.
Best Practices
Regular checks and maintenance
Documentation of cleaning processes
