Compressed Nonparametric Language Modelling release_f3hnat6scjfqjijofkrixra6ty

by Ehsan Shareghi, Gholamreza Haffari, Trevor Cohn

Published in International Joint Conference on Artificial Intelligence by International Joint Conferences on Artificial Intelligence Organization.

2017   p2701-2707

Abstract

Hierarchical Pitman-Yor Process priors are compelling for learning language models, outperforming point-estimate based methods. However, these models remain unpopular due to computational and statistical inference issues, such as memory and time usage, as well as poor mixing of sampler. In this work we propose a novel framework which represents the HPYP model compactly using compressed suffix trees. Then, we develop an efficient approximate inference scheme in this framework that has a much lower memory footprint compared to full HPYP and is fast in the inference time. The experimental results illustrate that our model can be built on significantly larger datasets compared to previous HPYP models, while being several orders of magnitudes smaller, fast for training and inference, and outperforming the perplexity of the state-of-the-art Modified Kneser-Ney count-based LM smoothing by up to 15%.
In application/xml+jats format

Archived Files and Locations

application/pdf  295.8 kB
file_fl7cvhyouvhg5m42txrwas4j3y
www.ijcai.org (web)
web.archive.org (webarchive)
Read Archived PDF
Preserved and Accessible
Type  paper-conference
Stage   published
Year   2017
Work Entity
access all versions, variants, and formats of this works (eg, pre-prints)
Catalog Record
Revision: d4be2b73-c0ec-4b92-9920-afbb0708cf31
API URL: JSON