MIMIC-Extract:A Data Extraction, Preprocessing, and Representation Pipeline for MIMIC-III

About

This repo contains code for MIMIC-Extract. It has been divided into the following folders: * Data: Locally contains the data to be extracted. * Notebooks: Jupyter Notebooks demonstrating test cases and usage of output data in risk and intervention prediction tasks. * Resources: Consist of Rohititemid.txt which describes the correlation of MIMIC-III item ids with those of MIMIC II as used by Rohit; itemidtovariablemap.csv which is the main file used in data extraction - consists of groupings of item ids as well as which item ids are ready to extract; variableranges.csv which describes the normal variable ranges for the levels assisting in extraction of proper data. It also contains expected schema of output tables. * Utils: scripts and detailed instructions for running MIMIC-Extract data pipeline. * `mimicdirect_extract.py`: extraction script.

Paper

If you use this code in your research, please cite the following publication:

Shirly Wang, Matthew B. A. McDermott, Geeticka Chauhan, Michael C. Hughes, Tristan Naumann, 
and Marzyeh Ghassemi. MIMIC-Extract: A Data Extraction, Preprocessing, and Representation 
Pipeline for MIMIC-III. arXiv:1907.08322. 

Pre-processed Output

If you simply wish to use the output of this pipeline in your own research, a preprocessed version with default parameters is available via gcp, here.

To access this, you will need to be credentialed for MIMIC-III GCP access through physionet. Instructions for that are available on physionet.

This output is released on an as-is basis, with no guarantees, but if you find any issues with it please let us know via Github issues.

Step-by-step instructions

The first several steps are the same here as above. These instructions are tested with mimic-code at version 762943eab64deb30bdb2abcf7db43602ccb25908

Step 0: Required software and prereqs

Your local system should have the following executables on the PATH:

• conda
• psql (PostgreSQL 9.4 or higher)
• git
• MIMIC-iii psql relational database (Refer to MIT-LCP Repo)

All instructions below should be executed from a terminal, with current directory set to utils/

Step 1: Create conda environment

Next, make a new conda environment from mimicextractenv_py36.yml and activate that environment.

conda env create --force -f ../mimic_extract_env_py36.yml

This step will report failure on the pip installation stage. This is not the end of the world. Instead, simply activate the environment (which should work despite the former "failure"):

conda activate mimic_data_extraction

And then install any failed packages with pip (e.g.,

pip install [package]

datapackage

spacy

scispacy

python -m spacy download en_core_web_sm

Expected Outcome

The desired enviroment will be created and activated.

Expected Resources

Will typically take less than 5 minutes. Requires a good internet connection.

Step 3: Build Views for Feature Extraction

Materialized views in the MIMIC PostgreSQL database will be generated. This includes all concept tables in MIT-LCP Repo and tables for extracting non-mechanical ventilation, and injections of crystalloid bolus and colloid bolus.

Note that you need to have schema edit permission on your postgres user to make concepts in this way. First, you must clone this github repository to a directory, which here we assume is stored in the environment variable

$MIMIC_CODE_DIR

cd $MIMIC_CODE_DIR/concepts
psql -d mimic -f postgres-functions.sql
bash postgres_make_concepts.sh

Next, you'll need to build 3 additional materialized views necessary for this pipeline. To do this (again with schema edit permission), navigate to

utils

bash postgres_make_extended_concepts.sh

psql -d mimic -f niv-durations.sql

Step 4: Set Cohort Selection and Extraction Criteria

Next, navigate to the root directory of this repository, activate your conda environment and run

python mimic_direct_extract.py ...

Expected Outcome

The default setting will create an hdf5 file inside MIMICEXTRACTOUTPUTDIR with four tables: * patients: static demographics, static outcomes * One row per (subjid,hadmid,icustayid)

• vitals_labs: time-varying vitals and labs (hourly mean, count and standard deviation)

  • One row per (subjid,hadmid,icustayid,hoursin)

• vitalslabsmean: time-varying vitals and labs (hourly mean only)

  • One row per (subjid,hadmid,icustayid,hoursin)

• interventions: hourly binary indicators for administered interventions

  • One row per (subjid,hadmid,icustayid,hoursin)

Expected Resources

Will probably take 5-10 hours. Will require a good machine with at least 50GB RAM.

Setting the population size

By default, this step builds a dataset with all eligible patients. Sometimes, we wish to run with only a small subset of patients (debugging, etc.).

To do this, just set the POP_SIZE environmental variable. For example, to build a curated dataset with only the first 1000 patients, we could do:

Common Errors / FAQ:

1. When running

   mimic_direct_extract.py

   , I encounter an error of the form:

       psycopg2.OperationalError: could not connect to server: No such file or directory
       Is the server running locally and accepting
       connections on Unix domain socket "/tmp/.s.PGSQL.5432"?
   

   or

       psycopg2.OperationalError: could not connect to server: No such file or directory
       Is the server running locally and accepting
       connections on Unix domain socket "/var/run/postgresql/..."?
   

   For this issue, see this stackoverflow post and use our

   --psql_host

   argument, which you can pass either directly when calling

   mimic_direct_extract.py

   or use via the Makefile instructions by setting the

   HOST

   environment variable.

2. relation "code_status" does not exist

   In this error, the table

   code_status

   hasn't been built successfully, and you'll need to rebuild your MIMIC-III concepts. Instructions for this can be found in Step 3 of either instruction set. Also see below for our issues specific to building concepts.

Common Errors with Building Concepts

1. When I built concepts, the system complained it didn't have permissions to edit schema mimiciii. This error indicates that your default psql user doesn't have authority to build concepts. You need to login as a higher authority postgres user to and have it run the commands. This is common in setups where multiple users have read-only access to postgres at once. If you do this, you may need to take extra steps to expose the resulting concepts tables to other users.
2. I built concepts, but now the code can't see them. This can be for a few reasons - firslty, you may not have permissions to read the new tables, and secondly, they may be in the wrong namespace. Our code expects them to be fully visible and within the mimiciii namespace. To adjust these properties, login as the owning postgres user and adjust the permissions and namespaces of those views manually. A few commands that are relevant are:

   • ALTER TABLE code_status SET SCHEMA mimiciii;

   • GRANT SELECT ON mimiciii.code_status TO [USER];

     Note that you'll need to run these on every concepts table accessed by the script.