Sequence Projection Models

This repository contains implementation of the following papers.

• PRADO: Projection Attention Networks for Document Classification On-Device
• Self-Governing Neural Networks for On-Device Short Text Classification
• [Tiny Neural Models for Seq2Seq] (https://arxiv.org/abs/2108.03340)

Description

We provide a family of models that projects sequence to fixed sized features. The idea behind is to build embedding-free models that minimize the model size. Instead of using embedding table to lookup embeddings, sequence projection models computes them on the fly.

Authors or Maintainers

• Prabhu Kaliamoorthi
• Arun Kandoor
• Yicheng Fan (@thunderfyc)

Requirements

Training

Train a PRADO model on civil comments dataset

bazel run -c opt :trainer -- \
--config_path=$(pwd)/configs/civil_comments_prado.txt \
--runner_mode=train --logtostderr --output_dir=/tmp/prado

Train a SGNN model to detect languages:

bazel run -c opt sgnn:train -- --logtostderr --output_dir=/tmp/sgnn

Evaluation

Evaluate PRADO model:

bazel run -c opt :trainer -- \
--config_path=$(pwd)/configs/civil_comments_prado.txt \
--runner_mode=eval --logtostderr --output_dir=/tmp/prado

Evaluate SGNN model:

bazel run -c opt sgnn:run_tflite -- --model=/tmp/sgnn/model.tflite "Hello world"

References

1. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

   Sergey Ioffe, Christian Szegedy

   [link]. In ICML, 2015.

2. Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference

   Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, Dmitry Kalenichenko

   [link]. In CVPR, 2018.

3. PRADO: Projection Attention Networks for Document Classification On-Device

   Prabhu Kaliamoorthi, Sujith Ravi, Zornitsa Kozareva

   [link]. In EMNLP-IJCNLP, 2019

4. Self-Governing Neural Networks for On-Device Short Text Classification

   Sujith Ravi, Zornitsa Kozareva

   [link]. In EMNLP, 2018

License

This project is licensed under the terms of the Apache License 2.0.