https://github.com/ddbourgin/numpy-ml

numpy-ml

Ever wish you had an inefficient but somewhat legible collection of machine learning algorithms implemented exclusively in NumPy? No?

Installation

For rapid experimentation

To use this code as a starting point for ML prototyping / experimentation, just clone the repository, create a new virtualenv, and start hacking:

$ git clone <https://github.com/ddbourgin/numpy-ml.git>
$ cd numpy-ml && virtualenv npml && source npml/bin/activate
$ pip3 install -r requirements-dev.txt

As a package

If you don't plan to modify the source, you can also install numpy-ml as a Python package: pip3 install -u numpy_ml.

The reinforcement learning agents train on environments defined in the OpenAI gym. To install these alongside numpy-ml, you can use pip3 install -u 'numpy_ml[rl]'.

Documentation

For more details on the available models, see the project documentation.

Available models

Click to expand!

**Gaussian mixture model**

- EM training

**Hidden Markov model**

- Viterbi decoding
- Likelihood computation
- MLE parameter estimation via Baum-Welch/forward-backward algorithm

**Latent Dirichlet allocation** (topic model)

- Standard model with MLE parameter estimation via variational EM
- Smoothed model with MAP parameter estimation via MCMC

**Neural networks**

- Layers / Layer-wise ops
    - Add
    - Flatten
    - Multiply
    - Softmax
    - Fully-connected/Dense
    - Sparse evolutionary connections
    - LSTM
    - Elman-style RNN
    - Max + average pooling
    - Dot-product attention
    - Embedding layer
    - Restricted Boltzmann machine (w. CD-n training)
    - 2D deconvolution (w. padding and stride)
    - 2D convolution (w. padding, dilation, and stride)
    - 1D convolution (w. padding, dilation, stride, and causality)
- Modules
    - Bidirectional LSTM
    - ResNet-style residual blocks (identity and convolution)
    - WaveNet-style residual blocks with dilated causal convolutions
    - Transformer-style multi-headed scaled dot product attention
- Regularizers
    - Dropout
- Normalization
    - Batch normalization (spatial and temporal)
    - Layer normalization (spatial and temporal)
- Optimizers
    - SGD w/ momentum
    - AdaGrad
    - RMSProp
    - Adam
- Learning Rate Schedulers
    - Constant
    - Exponential
    - Noam/Transformer
    - Dlib scheduler
- Weight Initializers
    - Glorot/Xavier uniform and normal
    - He/Kaiming uniform and normal
    - Standard and truncated normal
- Losses
    - Cross entropy
    - Squared error
    - Bernoulli VAE loss
    - Wasserstein loss with gradient penalty
    - Noise contrastive estimation loss
- Activations
    - ReLU
    - Tanh
    - Affine
    - Sigmoid
    - Leaky ReLU
    - ELU
    - SELU
    - GELU
    - Exponential
    - Hard Sigmoid
    - Softplus
- Models
    - Bernoulli variational autoencoder
    - Wasserstein GAN with gradient penalty
    - word2vec encoder with skip-gram and CBOW architectures
- Utilities
    - `col2im` (MATLAB port)
    - `im2col` (MATLAB port)
    - `conv1D`
    - `conv2D`
    - `deconv2D`
    - `minibatch`

**Tree-based models**

- Decision trees (CART)
- [Bagging] Random forests
- [Boosting] Gradient-boosted decision trees