Additional Materials and Links

Week 1

Recap of main ML algorithms

Overview of methods

• Scikit-Learn (or sklearn) library
• Overview of k-NN (sklearn's documentation)
• Overview of Linear Models (sklearn's documentation)
• Overview of Decision Trees (sklearn's documentation)
• Overview of algorithms and parameters in H2O documentation

Additional Tools

• Vowpal Wabbit repository
• XGBoost repository
• LightGBM repository
• Interactive demo of simple feed-forward Neural Net
• Frameworks for Neural Nets: Keras,PyTorch,TensorFlow,MXNet, Lasagne
• Example from sklearn with different decision surfaces
• Arbitrary order factorization machines

Software/Hardware requirements

StandCloud Computing:

• AWS, Google Cloud, Microsoft Azure

AWS spot option:

• Overview of Spot mechanism
• Spot Setup Guide

Stack and packages:

• Basic SciPy stack (ipython, numpy, pandas, matplotlib)
• Jupyter Notebook
• Stand-alone python tSNE package
• Libraries to work with sparse CTR-like data: LibFM, LibFFM
• Another tree-based method: RGF (implemetation, paper)
• Python distribution with all-included packages: Anaconda
• Blog "datas-frame" (contains posts about effective Pandas usage)

Feature preprocessing and generation with respect to models

Feature preprocessing

• Preprocessing in Sklearn
• Andrew NG about gradient descent and feature scaling
• Feature Scaling and the effect of standardization for machine learning algorithms

Feature generation

• Discover Feature Engineering, How to Engineer Features and How to Get Good at It
• Discussion of feature engineering on Quora

Feature extraction from text and images

Feature extraction from text

Bag of words * Feature extraction from text with Sklearn * More examples of using Sklearn

Word2vec * Tutorial to Word2vec * Tutorial to word2vec usage * Text Classification With Word2Vec * Introduction to Word Embedding Models with Word2Vec

NLP Libraries * NLTK * TextBlob

Feature extraction from images

Pretrained models * Using pretrained models in Keras * Image classification with a pre-trained deep neural network

Finetuning * How to Retrain Inception's Final Layer for New Categories in Tensorflow * Fine-tuning Deep Learning Models in Keras

Week 2

Exploratory data analysis

Visualization tools

• Seaborn
• Plotly
• Bokeh
• ggplot
• Graph visualization with NetworkX

Others

• Biclustering algorithms for sorting corrplots

Validation

• Validation in Sklearn
• Advices on validation in a competition

Data leakages

• Perfect score script by Oleg Trott -- used to probe leaderboard
• Page about data leakages on Kaggle

Week 3

Metrics optimization

Classification

• Evaluation Metrics for Classification Problems: Quick Examples + References
• Decision Trees: “Gini” vs. “Entropy” criteria
• Understanding ROC curves

Ranking

• Learning to Rank using Gradient Descent -- original paper about pairwise method for AUC optimization
• Overview of further developments of RankNet
• RankLib (implemtations for the 2 papers from above)
• Learning to Rank Overview

Clustering

• Evaluation metrics for clustering

Week 4

Hyperparameter tuning

• Tuning the hyper-parameters of an estimator (sklearn)
• Optimizing hyperparameters with hyperopt
• Complete Guide to Parameter Tuning in Gradient Boosting (GBM) in Python

Tips and tricks

• Far0n's framework for Kaggle competitions "kaggletils"
• 28 Jupyter Notebook tips, tricks and shortcuts

Advanced features II

Matrix Factorization:

• Overview of Matrix Decomposition methods (sklearn)

t-SNE:

• Multicore t-SNE implementation
• Comparison of Manifold Learning methods (sklearn)
• How to Use t-SNE Effectively (distill.pub blog)
• tSNE homepage (Laurens van der Maaten)
• Example: tSNE with different perplexities (sklearn)

Interactions:

• Facebook Research's paper about extracting categorical features from trees
• Example: Feature transformations with ensembles of trees (sklearn)

Ensembling

• Kaggle ensembling guide at MLWave.com (overview of approaches)
• StackNet — a computational, scalable and analytical meta modelling framework (by KazAnova)
• Heamy — a set of useful tools for competitive data science (including ensembling)

Week 5

Competitions go through

You can often find a solution of the competition you're interested on its forum. Here we put links to collections of such solutions that will prove useful to you.

Past solutions

• http://ndres.me/kaggle-past-solutions/
• https://www.kaggle.com/wiki/PastSolutions
• http://www.chioka.in/kaggle-competition-solutions/
• https://github.com/ShuaiW/kaggle-classification/