Upgrading sentencepiece 0.0 to 0.1
Breaking0.0.9 → 0.1.99Breaking2Platform2Deprecated1New22Changed20Fixed20Other5
Fetching 23 releases from GitHub…16 / 23
BreakingUpgrade verdict
2 breaking changes and 2 platform requirement changes.
In plain terms
- Uses the efficient encoding algorithm by default and removes the ability to switch the Viterbi tokenization algorithm.
- The input_sentence_size setting now accepts larger non-negative values, but code expecting a 32-bit signed integer may need updating.
- You can now install this package on Python 3.11 using prebuilt wheels.
- Users on ppc64le and s390x can no longer install the prebuilt pip package.
Must act 5 items need code or config changes
What you must change
Breaking2
Uses the efficient encoding algorithm by default and removes the ability to switch the Viterbi tokenization algorithm.
Changes the type of input_sentence_size from int32 to uint64.
input_sentence_sizePlatform2
Adds Python 3.11 support with wheel packages available for Python 3.11.
Removes the pip package for ppc64le and s390x architectures because cibuildtool does not support them.
Deprecations to plan for
Deprecated1
Deprecates --mining_sentence_size and --training_sentence_size; all sentences are loaded by default, and --input_sentence_size can limit loading.
--mining_sentence_size--training_sentence_size--input_sentence_size