SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing

This paper presents SentencePiece, a simple, language-independent subword tokenizer and detokenizer designed for neural-network-based text processing such as neural machine translation. It provides open-source C++ and Python implementations of subword units. Its key distinction from existing subword segmentation tools is that, rather than assuming the input has already been pre-tokenized into word sequences, SentencePiece can train subword models directly from raw sentences, allowing a purely end-to-end and language-independent system.

To validate the approach, the authors run neural machine translation experiments on English-Japanese translation and find that training subwords directly from raw sentences can achieve accuracy comparable to conventional direct subword training. They also compare subword training and segmentation across various configurations. Released under the Apache 2 license, SentencePiece enables a purely end-to-end tokenization pipeline that does not depend on language-specific pre-tokenization.