EdgeFormer: A Parameter-Efficient Transformer for On-Device Seq2seq Generation
release_svctxmmyvrfj3bd2hituoku2jy
by
Tao Ge, Furu Wei
2022
Abstract
We propose EdgeFormer -- a parameter-efficient Transformer of the
encoder-decoder architecture for on-device seq2seq generation, which is
customized under the strict computation and memory constraints. EdgeFormer
proposes two novel principles for cost-effective parameterization and further
enhance the model with efficient layer adaptation. We conduct extensive
experiments on two practical on-device seq2seq tasks: Machine Translation and
Grammatical Error Correction, and show that EdgeFormer can effectively
outperform previous parameter-efficient Transformer baselines and achieve very
competitive results with knowledge distillation under both the computation and
memory constraints.
In text/plain
format
Archived Content
There are no accessible files associated with this release. You could check other releases for this work for an accessible version.
Know of a fulltext copy of on the public web? Submit a URL and we will archive it
2202.07959v1
access all versions, variants, and formats of this works (eg, pre-prints)