Effective Neural Solution for Multi-Criteria Word Segmentation release_abvenw5swjau7dey4ev7h7jdxu

by Han He, Lei Wu, Hua Yan, Zhimin Gao, Yi Feng, George Townsend

Released as a article .

2018  

Abstract

We present a simple yet elegant solution to train a single joint model on multi-criteria corpora for Chinese Word Segmentation (CWS). Our novel design requires no private layers in model architecture, instead, introduces two artificial tokens at the beginning and ending of input sentence to specify the required target criteria. The rest of the model including Long Short-Term Memory (LSTM) layer and Conditional Random Fields (CRFs) layer remains unchanged and is shared across all datasets, keeping the size of parameter collection minimal and constant. On Bakeoff 2005 and Bakeoff 2008 datasets, our innovative design has surpassed both single-criterion and multi-criteria state-of-the-art learning results. To the best knowledge, our design is the first one that has achieved the latest high performance on such large scale datasets. Source codes and corpora of this paper are available on GitHub.
In text/plain format

Archived Files and Locations

application/pdf  165.8 kB
file_ldik4rlyujenvkqc2llceqymvq
arxiv.org (repository)
web.archive.org (webarchive)
Read Archived PDF
Preserved and Accessible
Type  article
Stage   submitted
Date   2018-01-04
Version   v2
Language   en ?
arXiv  1712.02856v2
Work Entity
access all versions, variants, and formats of this works (eg, pre-prints)
Catalog Record
Revision: 828c656e-1391-4abc-a802-ec6ebae2be9c
API URL: JSON