Preprocessing

The preprocessing module implements common data preprocessing routines.

• nlp.py: Routines and objects for handling text data.

  • n-gram generators
  • Word and character tokenization
  • Punctuation and stop-word removal
  • Vocabulary / unigram count objects
  • Byte-pair encoding (Gage, 1994; Sennrich, Haddow, & Birch, 2015)
  • Huffman tree encoding / decoding
  • Term frequency-inverse document frequency (tf-idf) encoding

• dsp.py: Routines for handling audio and image data.

  • Signal windowing
  • Signal autocorrelation
  • Discrete Fourier transform
  • Discrete cosine transform (type II)
  • Signal resampling via (bi-)linear interpolation and nearest neighbor
  • Mel-frequency cepstral coefficients (MFCCs) (Mermelstein, 1976; Davis & Mermelstein, 1980)

• general.py: General data preprocessing objects and functions.

  • Feature hashing (Moody, 1989)
  • Mini-batch generators
  • One-hot encoding / decoding
  • Feature standardization