Need help with janitor?
Click the “chat” button below for chat support from the developer who created it, or find similar developers for support.

About the developer

sfirke
991 Stars 100 Forks Other 876 Commits 41 Opened issues

Description

simple tools for data cleaning in R

Services available

!
?

Need anything else?

Contributors list

Data scientists, according to interviews and expert estimates, spend from 50 percent to 80 percent of their time mired in this more mundane labor of collecting and preparing unruly digital data, before it can be explored for useful nuggets.

“For Big-Data Scientists, ‘Janitor Work’ Is Key Hurdle to Insight” (New York Times, 2014)

janitor


R build
status Coverage
Status lifecycle CRAN\_Status\_Badge \!Monthly Downloads \!Downloads <!-- badges: end -->

janitor has simple functions for examining and cleaning dirty data. It was built with beginning and intermediate R users in mind and is optimized for user-friendliness. Advanced R users can already do everything covered here, but with janitor they can do it faster and save their thinking for the fun stuff.

The main janitor functions:

  • perfectly format data.frame column names;
  • create and format frequency tables of one, two, or three variables - think an improved
    table()
    ; and
  • provide other tools for cleaning and examining data.frames.

The tabulate-and-report functions approximate popular features of SPSS and Microsoft Excel.

janitor is a #tidyverse-oriented package. Specifically, it plays nicely with the

%>%
pipe and is optimized for cleaning data brought in with the readr and readxl packages.

Installation

You can install:

  • the most recent officially-released version from CRAN with

    install.packages("janitor")
    
  • the latest development version from GitHub with

    install.packages("devtools")
    devtools::install_github("sfirke/janitor")
    

Using janitor

A full description of each function, organized by topic, can be found in janitor’s catalog of functions vignette. There you will find functions not mentioned in this README, like

compare_df_cols()
which provides a summary of differences in column names and types when given a set of data.frames.

Below are quick examples of how janitor tools are commonly used.

Cleaning dirty data

Take this roster of teachers at a fictional American high school, stored in the Microsoft Excel file dirty_data.xlsx: All kinds of dirty.

Dirtiness includes:

  • A header at the top
  • Dreadful column names
  • Rows and columns containing Excel formatting but no data
  • Dates in two different formats in a single column (MM/DD/YYYY and numbers)
  • Values spread inconsistently over the “Certification” columns

Here’s that data after being read in to R:

library(readxl); library(janitor); library(dplyr); library(here)

roster_raw Rows: 14 #> Columns: 11 #> $ Data most recently refreshed on: "First Name", "Jason", "Jason", "Alicia", "Ada", "Desus", "Chi... #> $ ...2 "Last Name", "Bourne", "Bourne", "Keys", "Lovelace", "Nice", "... #> $ ...3 "Employee Status", "Teacher", "Teacher", "Teacher", "Teacher",... #> $ Dec-27 2020 "Subject", "PE", "Drafting", "Music", NA, "Dean", "Physics", "... #> $ ...5 "Hire Date", "39690", "43479", "37118", "38572", "42791", "110... #> $ ...6 "% Allocated", "0.75", "0.25", "1", "1", "1", "0.5", "0.5", NA... #> $ ...7 "Full time?", "Yes", "Yes", "Yes", "Yes", "Yes", "Yes", "Yes",... #> $ ...8 "do not edit! --->", NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, N... #> $ ...9 "Certification", "Physical ed", "Physical ed", "Instr. music",... #> $ ...10 "Certification", "Theater", "Theater", "Vocal music", "Compute... #> $ ...11 "Active?", "YES", "YES", "YES", "YES", "YES", "YES", "YES", NA...

Now, to clean it up, starting with the column names.

Name cleaning comes in two flavors.

make_clean_names()
operates on character vectors and can be used during data import:
roster_raw_cleaner  Rows: 13
#> Columns: 11
#> $ first_name         "Jason", "Jason", "Alicia", "Ada", "Desus", "Chien-Shiung", "Chien-Shiung", NA,...
#> $ last_name          "Bourne", "Bourne", "Keys", "Lovelace", "Nice", "Wu", "Wu", NA, "Joyce", "Lamar...
#> $ employee_status    "Teacher", "Teacher", "Teacher", "Teacher", "Administration", "Teacher", "Teach...
#> $ subject            "PE", "Drafting", "Music", NA, "Dean", "Physics", "Chemistry", NA, "English", "...
#> $ hire_date          39690, 43479, 37118, 38572, 42791, 11037, 11037, NA, 36423, 27919, 42221, 34700...
#> $ percent_allocated  0.75, 0.25, 1.00, 1.00, 1.00, 0.50, 0.50, NA, 0.50, 0.50, NA, NA, 0.80
#> $ full_time          "Yes", "Yes", "Yes", "Yes", "Yes", "Yes", "Yes", NA, "No", "No", "No", "No", "No"
#> $ do_not_edit        NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA
#> $ certification      "Physical ed", "Physical ed", "Instr. music", "PENDING", "PENDING", "Science 6-...
#> $ certification_2    "Theater", "Theater", "Vocal music", "Computers", NA, "Physics", "Physics", NA,...
#> $ active             "YES", "YES", "YES", "YES", "YES", "YES", "YES", NA, "YES", "YES", "YES", "YES"...

clean_names()
is a convenience version of
make_clean_names()
that can be used for piped data.frame workflows. The equivalent steps with
clean_names()
would be:
roster_raw %
  row_to_names(row_number = 1) %>%
  clean_names()

The data.frame now has clean names. Let’s tidy it up further:

roster %
  remove_empty(c("rows", "cols")) %>%
  remove_constant(na.rm = TRUE, quiet = FALSE) %>% # remove the column of all "Yes" values 
  mutate(hire_date = convert_to_date(hire_date, # handle the mixed-format dates
                                     character_fun = lubridate::mdy),
         cert = dplyr::coalesce(certification, certification_2)) %>%
  select(-certification, -certification_2) # drop unwanted columns
#> Removing 1 constant columns of 10 columns total (Removed: active).

roster #> # A tibble: 12 x 8 #> first_name last_name employee_status subject hire_date percent_allocated full_time cert
#>
#> 1 Jason Bourne Teacher PE 2008-08-30 0.75 Yes Physical ed
#> 2 Jason Bourne Teacher Drafting 2019-01-14 0.25 Yes Physical ed
#> 3 Alicia Keys Teacher Music 2001-08-15 1 Yes Instr. music
#> 4 Ada Lovelace Teacher 2005-08-08 1 Yes PENDING
#> 5 Desus Nice Administration Dean 2017-02-25 1 Yes PENDING
#> 6 Chien-Shiung Wu Teacher Physics 1930-03-20 0.5 Yes Science 6-12
#> 7 Chien-Shiung Wu Teacher Chemistry 1930-03-20 0.5 Yes Science 6-12
#> 8 James Joyce Teacher English 1999-09-20 0.5 No English 6-12
#> 9 Hedy Lamarr Teacher Science 1976-06-08 0.5 No PENDING
#> 10 Carlos Boozer Coach Basketball 2015-08-05 No Physical ed
#> 11 Young Boozer Coach 1995-01-01 No Political sci. #> 12 Micheal Larsen Teacher English 2009-09-15 0.8 No Vocal music

Examining dirty data

Finding duplicates

Use

get_dupes()
to identify and examine duplicate records during data cleaning. Let’s see if any teachers are listed more than once:
roster %>% get_dupes(contains("name"))
#> # A tibble: 4 x 9
#>   first_name   last_name dupe_count employee_status subject   hire_date  percent_allocat~ full_time cert      
#>                                                                 
#> 1 Chien-Shiung Wu                 2 Teacher         Physics   1930-03-20 0.5              Yes       Science 6~
#> 2 Chien-Shiung Wu                 2 Teacher         Chemistry 1930-03-20 0.5              Yes       Science 6~
#> 3 Jason        Bourne             2 Teacher         PE        2008-08-30 0.75             Yes       Physical ~
#> 4 Jason        Bourne             2 Teacher         Drafting  2019-01-14 0.25             Yes       Physical ~

Yes, some teachers appear twice. We ought to address this before counting employees.

Tabulating tools

A variable (or combinations of two or three variables) can be tabulated with

tabyl()
. The resulting data.frame can be tweaked and formatted with the suite of
adorn_
functions for quick analysis and printing of pretty results in a report.
adorn_
functions can be helpful with non-tabyls, too.

tabyl()

Like

table()
, but pipe-able, data.frame-based, and fully featured.

tabyl
can be called two ways:
  • On a vector, when tabulating a single variable:
    tabyl(roster$subject)
  • On a data.frame, specifying 1, 2, or 3 variable names to tabulate:
    roster %>% tabyl(subject, employee_status)
    .
    • Here the data.frame is passed in with the
      %>%
      pipe; this allows
      tabyl
      to be used in an analysis pipeline

One variable:

roster %>%
  tabyl(subject)
#>     subject n    percent valid_percent
#>  Basketball 1 0.08333333           0.1
#>   Chemistry 1 0.08333333           0.1
#>        Dean 1 0.08333333           0.1
#>    Drafting 1 0.08333333           0.1
#>     English 2 0.16666667           0.2
#>       Music 1 0.08333333           0.1
#>          PE 1 0.08333333           0.1
#>     Physics 1 0.08333333           0.1
#>     Science 1 0.08333333           0.1
#>         2 0.16666667            NA

Two variables:

roster %>%
  filter(hire_date > as.Date("1950-01-01")) %>%
  tabyl(employee_status, full_time)
#>  employee_status No Yes
#>   Administration  0   1
#>            Coach  2   0
#>          Teacher  3   4

Three variables:

roster %>%
  tabyl(full_time, subject, employee_status, show_missing_levels = FALSE)
#> $Administration
#>  full_time Dean
#>        Yes    1
#> 
#> $Coach
#>  full_time Basketball NA_
#>         No          1   1
#> 
#> $Teacher
#>  full_time Chemistry Drafting English Music PE Physics Science NA_
#>         No         0        0       2     0  0       0       1   0
#>        Yes         1        1       0     1  1       1       0   1
Adorning tabyls

The

adorn_
functions dress up the results of these tabulation calls for fast, basic reporting. Here are some of the functions that augment a summary table for reporting:
roster %>%
  tabyl(employee_status, full_time) %>%
  adorn_totals("row") %>%
  adorn_percentages("row") %>%
  adorn_pct_formatting() %>%
  adorn_ns() %>%
  adorn_title("combined")
#>  employee_status/full_time         No        Yes
#>             Administration   0.0% (0) 100.0% (1)
#>                      Coach 100.0% (2)   0.0% (0)
#>                    Teacher  33.3% (3)  66.7% (6)
#>                      Total  41.7% (5)  58.3% (7)

Pipe that right into

knitr::kable()
in your RMarkdown report.

These modular adornments can be layered to reduce R’s deficit against Excel and SPSS when it comes to quick, informative counts. Learn more about

tabyl()
and the
adorn_
functions from the tabyls vignette.

Contact me

You are welcome to:

We use cookies. If you continue to browse the site, you agree to the use of cookies. For more information on our use of cookies please see our Privacy Policy.