Skip to content

Upgrading sentencepiece 0.1 to 0.2

Breaking0.1.99 → 0.2.2Security8Platform4Deprecated1New12Changed11Fixed5Other6

Full comparisonBack to sentencepiece
BreakingUpgrade verdict

4 platform requirement changes and 8 security fixes.

In plain terms
  • The package now installs and runs on the newest Python versions, 3.13 and the 3.14 release candidate.
  • Older Python versions are no longer supported; you need Python 3.9 or newer.
  • Windows on Arm devices now get a prebuilt package.
  • The package can now be used with Python 3.12.
Must act Top 5 of 13 that need code or config changes

Requires Python >=3.9 from 0.2.1 (none stated before).

What you must change

Security8
Adds bounds checking to prevent out-of-bounds reads in offset mapping.
0.2.2GitHub ReleaseSummarizedView source
Verifies darts-clone trie offsets when loading models to prevent segfaults on corrupted or malicious model files.
darts-clone
0.2.2GitHub ReleaseSummarizedView source
Adds bounds checks to piece accessors and validates value offsets in DecompileCharsMap.
DecompileCharsMap
0.2.2GitHub ReleaseSummarizedView source
Replaces internal CHECK macros with DCHECK or returned errors to prevent host application crashes on invalid inputs.
CHECKDCHECK
0.2.2GitHub ReleaseSummarizedView source
Explicitly disallows vocabulary pieces containing null characters.
0.2.2GitHub ReleaseSummarizedView source
Implements a global timeout for NBest A* search to prevent hangs on pathological inputs.
0.2.2GitHub ReleaseSummarizedView source
Fixes a heap overflow that could occur with a model containing an invalid precompiled normalization model.
0.2.1GitHub ReleaseSummarizedView source
Sentencepiece has a a heap overflow issue
0.2.1GitHub AdvisoryOriginalView source
Platform4
Adds wheel builds and support for Python 3.13 and 3.14 (rc1).
0.2.1GitHub ReleaseSummarizedView source
Raises the minimum supported Python version to 3.9.
0.2.1GitHub ReleaseSummarizedView source
Adds a wheel for Windows Arm64.
0.2.1GitHub ReleaseSummarizedView source
Supports Python 3.12.
0.2.0GitHub ReleaseSummarizedView source

Deprecations to plan for

Deprecated1
Deprecates the Linux i686 wheel package.
0.2.1GitHub ReleaseSummarizedView source

API changes

Generated by DepDelta from published Python source. Not mixed with prose notes.

  • removed: class ImmutableNBestSentencePieceText(object)
  • removed: class ImmutableNBestSentencePieceText.ImmutableSentencePieceTextIterator
  • removed: class ImmutableSentencePieceText(object)
  • removed: class ImmutableSentencePieceText.ImmutableSentencePieceIterator
  • removed: class ImmutableSentencePieceText_ImmutableSentencePiece(object)
  • removed: constructor ImmutableNBestSentencePieceText()
  • removed: constructor ImmutableNBestSentencePieceText.ImmutableSentencePieceTextIterator(proto)
  • removed: constructor ImmutableSentencePieceText()
  • removed: constructor ImmutableSentencePieceText.ImmutableSentencePieceIterator(proto)
  • removed: constructor ImmutableSentencePieceText_ImmutableSentencePiece()
  • removed: field ImmutableNBestSentencePieceText.thisown
  • removed: field ImmutableSentencePieceText.score
  • removed: field ImmutableSentencePieceText.text
  • removed: field ImmutableSentencePieceText.thisown
  • removed: field ImmutableSentencePieceText_ImmutableSentencePiece.begin
  • removed: field ImmutableSentencePieceText_ImmutableSentencePiece.end
  • removed: field ImmutableSentencePieceText_ImmutableSentencePiece.id
  • removed: field ImmutableSentencePieceText_ImmutableSentencePiece.piece
  • removed: field ImmutableSentencePieceText_ImmutableSentencePiece.surface
  • removed: field ImmutableSentencePieceText_ImmutableSentencePiece.thisown
  • removed: field SentencePieceProcessor.thisown
  • removed: field SentencePieceTrainer.thisown
  • removed: function SentencePieceTrainer__TrainFromMap(args)
  • removed: function SentencePieceTrainer__TrainFromMap2(args, iter)
  • removed: function SentencePieceTrainer__TrainFromMap3(args)
  • removed: function SentencePieceTrainer__TrainFromMap4(args, iter)
  • removed: function SentencePieceTrainer__TrainFromString(arg)
  • removed: method ImmutableNBestSentencePieceText.SerializeAsString()
  • removed: method ImmutableSentencePieceText.SerializeAsString()
  • removed: method SentencePieceProcessor.Init(model_file=…, model_proto=…, out_type=…, add_bos=…, add_eos=…, reverse=…, emit_unk_piece=…, enable_sampling=…, nbest_size=…, alpha=…, num_threads=…)
  • removed: method SentencePieceProcessor.LoadVocabulary(filename, threshold)
  • removed: method SentencePieceProcessor.ResetVocabulary()
  • removed: method SentencePieceProcessor.SetDecodeExtraOptions(extra_option)
  • removed: method SentencePieceProcessor.SetEncodeExtraOptions(extra_option)
  • removed: method SentencePieceProcessor.SetVocabulary(valid_vocab)
  • removed: property ImmutableNBestSentencePieceText.nbests
  • removed: property ImmutableSentencePieceText.pieces
  • removed: variable set_random_generator_seed
  • changed: class SentencePieceProcessor(object) → class SentencePieceProcessor
  • changed: class SentencePieceTrainer(object) → class SentencePieceTrainer
  • changed: constructor SentencePieceProcessor() → constructor SentencePieceProcessor(model_file: str | None=…, model_proto: bytes | None=…, return_type: _OutType | None=…, add_bos: bool=…, add_eos: bool=…, reverse: bool=…, emit_unk_piece: bool=…, enable_sampling: bool=…, nbest_size: int=…, alpha: float=…, num_threads: int=…, out_type: _OutType | None=…) -> None
  • changed: constructor SentencePieceTrainer(*args, **kwargs) → constructor SentencePieceTrainer() -> None
  • changed: function SetRandomGeneratorSeed(seed) → function SetRandomGeneratorSeed(seed: int) -> None
  • changed: method SentencePieceProcessor.CalculateEntropy(input, alpha, num_threads=…) → method SentencePieceProcessor.CalculateEntropy(input: str | Sequence[str], alpha: float, num_threads: int | None=…) -> float | list[float]
  • changed: method SentencePieceProcessor.Decode(input, out_type=…, num_threads=…) → method SentencePieceProcessor.Decode(input: int | str | bytes | Sequence[int] | Sequence[str] | Sequence[bytes] | Sequence[Sequence[int]] | Sequence[Sequence[str]] | Sequence[Sequence[bytes]], return_type: _DecodeOutType | None=…, num_threads: int | None=…, thread_pool: ThreadPool | None=…, out_type: _DecodeOutType | None=…, return_bytes: bool | None=…) -> Any
  • changed: method SentencePieceProcessor.DecodeIds(input, out_type=…, **kwargs) → method SentencePieceProcessor.DecodeIds(input: int | Sequence[int] | Sequence[Sequence[int]], return_type: _DecodeOutType=…, out_type: _DecodeOutType=…, **kwargs: Any) -> Any
  • changed: method SentencePieceProcessor.DecodeIdsAsImmutableProto(input, out_type=…, **kwargs) → method SentencePieceProcessor.DecodeIdsAsImmutableProto(input: int | Sequence[int] | Sequence[Sequence[int]], **kwargs: Any) -> Any
  • changed: method SentencePieceProcessor.DecodeIdsAsSerializedProto(input, out_type=…, **kwargs) → method SentencePieceProcessor.DecodeIdsAsSerializedProto(input: int | Sequence[int] | Sequence[Sequence[int]], return_type: Literal['serialized_proto']=…, out_type: Literal['serialized_proto']=…, **kwargs: Any) -> Any
  • changed: method SentencePieceProcessor.DecodePieces(input, out_type=…, **kwargs) → method SentencePieceProcessor.DecodePieces(input: str | bytes | Sequence[str] | Sequence[bytes] | Sequence[Sequence[str]] | Sequence[Sequence[bytes]], return_type: _DecodeOutType=…, out_type: _DecodeOutType=…, **kwargs: Any) -> Any
  • changed: method SentencePieceProcessor.DecodePiecesAsImmutableProto(input, out_type=…, **kwargs) → method SentencePieceProcessor.DecodePiecesAsImmutableProto(input: str | bytes | Sequence[str] | Sequence[bytes] | Sequence[Sequence[str]] | Sequence[Sequence[bytes]], **kwargs: Any) -> Any
  • changed: method SentencePieceProcessor.DecodePiecesAsSerializedProto(input, out_type=…, **kwargs) → method SentencePieceProcessor.DecodePiecesAsSerializedProto(input: str | bytes | Sequence[str] | Sequence[bytes] | Sequence[Sequence[str]] | Sequence[Sequence[bytes]], return_type: Literal['serialized_proto']=…, out_type: Literal['serialized_proto']=…, **kwargs: Any) -> Any
  • changed: method SentencePieceProcessor.Encode(input, out_type=…, add_bos=…, add_eos=…, reverse=…, emit_unk_piece=…, enable_sampling=…, nbest_size=…, alpha=…, num_threads=…) → method SentencePieceProcessor.Encode(input: str | bytes | Sequence[str] | Sequence[bytes], return_type: _OutType | None=…, add_bos: bool | None=…, add_eos: bool | None=…, reverse: bool | None=…, emit_unk_piece: bool | None=…, enable_sampling: bool | None=…, nbest_size: int | None=…, alpha: float | None=…, num_threads: int | None=…, thread_pool: ThreadPool | None=…, out_type: _OutType | None=…) ->…
  • changed: method SentencePieceProcessor.EncodeAsIds(input, **kwargs) → method SentencePieceProcessor.EncodeAsIds(input: str | bytes | Sequence[str] | Sequence[bytes], **kwargs: Any) -> Any
  • changed: method SentencePieceProcessor.EncodeAsImmutableProto(input, **kwargs) → method SentencePieceProcessor.EncodeAsImmutableProto(input: str | bytes | Sequence[str] | Sequence[bytes], **kwargs: Any) -> Any
  • changed: method SentencePieceProcessor.EncodeAsPieces(input, **kwargs) → method SentencePieceProcessor.EncodeAsPieces(input: str | bytes | Sequence[str] | Sequence[bytes], **kwargs: Any) -> Any
  • changed: method SentencePieceProcessor.EncodeAsSerializedProto(input, **kwargs) → method SentencePieceProcessor.EncodeAsSerializedProto(input: str | bytes | Sequence[str] | Sequence[bytes], **kwargs: Any) -> Any
  • changed: method SentencePieceProcessor.GetPieceSize() → method SentencePieceProcessor.GetPieceSize() -> int
  • changed: method SentencePieceProcessor.GetScore(id) → method SentencePieceProcessor.GetScore(id: int) -> float | (id: list[int]) -> list[float]
  • changed: method SentencePieceProcessor.IdToPiece(id) → method SentencePieceProcessor.IdToPiece(id: int) -> str | (id: list[int]) -> list[str]
  • changed: method SentencePieceProcessor.IsByte(id) → method SentencePieceProcessor.IsByte(id: int) -> bool | (id: list[int]) -> list[bool]
  • changed: method SentencePieceProcessor.IsControl(id) → method SentencePieceProcessor.IsControl(id: int) -> bool | (id: list[int]) -> list[bool]
  • changed: method SentencePieceProcessor.IsUnknown(id) → method SentencePieceProcessor.IsUnknown(id: int) -> bool | (id: list[int]) -> list[bool]
  • changed: method SentencePieceProcessor.IsUnused(id) → method SentencePieceProcessor.IsUnused(id: int) -> bool | (id: list[int]) -> list[bool]
  • changed: method SentencePieceProcessor.Load(model_file=…, model_proto=…) → method SentencePieceProcessor.Load(model_file: str | None=…, model_proto: bytes | None=…) -> bool
  • changed: method SentencePieceProcessor.LoadFromFile(arg) → method SentencePieceProcessor.LoadFromFile(arg: str) -> bool
  • changed: method SentencePieceProcessor.LoadFromSerializedProto(serialized) → method SentencePieceProcessor.LoadFromSerializedProto(serialized: bytes) -> bool
  • changed: method SentencePieceProcessor.NBestEncode(input, out_type=…, add_bos=…, add_eos=…, reverse=…, emit_unk_piece=…, nbest_size=…) → method SentencePieceProcessor.NBestEncode(input: str | bytes | Sequence[str] | Sequence[bytes], return_type: _OutType | None=…, add_bos: bool | None=…, add_eos: bool | None=…, reverse: bool | None=…, emit_unk_piece: bool | None=…, nbest_size: int | None=…, out_type: _OutType | None=…) -> Any
  • changed: method SentencePieceProcessor.NBestEncodeAsIds(input, nbest_size=…, **kwargs) → method SentencePieceProcessor.NBestEncodeAsIds(input: str | bytes | Sequence[str] | Sequence[bytes], nbest_size: int | None=…, **kwargs: Any) -> Any
  • changed: method SentencePieceProcessor.NBestEncodeAsImmutableProto(input, nbest_size=…, **kwargs) → method SentencePieceProcessor.NBestEncodeAsImmutableProto(input: str | bytes | Sequence[str] | Sequence[bytes], nbest_size: int | None=…, **kwargs: Any) -> Any
  • changed: method SentencePieceProcessor.NBestEncodeAsPieces(input, nbest_size=…, **kwargs) → method SentencePieceProcessor.NBestEncodeAsPieces(input: str | bytes | Sequence[str] | Sequence[bytes], nbest_size: int | None=…, **kwargs: Any) -> Any
  • changed: method SentencePieceProcessor.NBestEncodeAsSerializedProto(input, nbest_size=…, **kwargs) → method SentencePieceProcessor.NBestEncodeAsSerializedProto(input: str | bytes | Sequence[str] | Sequence[bytes], nbest_size: int | None=…, **kwargs: Any) -> Any
  • changed: method SentencePieceProcessor.PieceToId(piece) → method SentencePieceProcessor.PieceToId(piece: str) -> int | (piece: list[str]) -> list[int]
  • changed: method SentencePieceProcessor.SampleEncodeAndScore(input, out_type=…, add_bos=…, add_eos=…, reverse=…, emit_unk_piece=…, num_samples=…, alpha=…, wor=…, include_best=…) → method SentencePieceProcessor.SampleEncodeAndScore(input: str | bytes | Sequence[str] | Sequence[bytes], return_type: _OutType | None=…, add_bos: bool | None=…, add_eos: bool | None=…, reverse: bool | None=…, emit_unk_piece: bool | None=…, num_samples: int | None=…, alpha: float | None=…, wor: bool | None=…, include_best: bool | None=…, out_type: _OutType | None=…) -> Any
  • changed: method SentencePieceProcessor.SampleEncodeAndScoreAsIds(input, num_samples=…, alpha=…, **kwargs) → method SentencePieceProcessor.SampleEncodeAndScoreAsIds(input: str | bytes | Sequence[str] | Sequence[bytes], num_samples: int | None=…, alpha: float | None=…, **kwargs: Any) -> Any
  • changed: method SentencePieceProcessor.SampleEncodeAndScoreAsImmutableProto(input, num_samples=…, alpha=…, **kwargs) → method SentencePieceProcessor.SampleEncodeAndScoreAsImmutableProto(input: str | bytes | Sequence[str] | Sequence[bytes], num_samples: int | None=…, alpha: float | None=…, **kwargs: Any) -> Any
  • changed: method SentencePieceProcessor.SampleEncodeAndScoreAsPieces(input, num_samples=…, alpha=…, **kwargs) → method SentencePieceProcessor.SampleEncodeAndScoreAsPieces(input: str | bytes | Sequence[str] | Sequence[bytes], num_samples: int | None=…, alpha: float | None=…, **kwargs: Any) -> Any
  • changed: method SentencePieceProcessor.SampleEncodeAndScoreAsSerializedProto(input, num_samples=…, alpha=…, **kwargs) → method SentencePieceProcessor.SampleEncodeAndScoreAsSerializedProto(input: str | bytes | Sequence[str] | Sequence[bytes], num_samples: int | None=…, alpha: float | None=…, **kwargs: Any) -> Any
  • changed: method SentencePieceProcessor.SampleEncodeAsIds(input, nbest_size=…, alpha=…, **kwargs) → method SentencePieceProcessor.SampleEncodeAsIds(input: str | bytes | Sequence[str] | Sequence[bytes], nbest_size: int | None=…, alpha: float | None=…, **kwargs: Any) -> Any
  • changed: method SentencePieceProcessor.SampleEncodeAsImmutableProto(input, nbest_size=…, alpha=…, **kwargs) → method SentencePieceProcessor.SampleEncodeAsImmutableProto(input: str | bytes | Sequence[str] | Sequence[bytes], nbest_size: int | None=…, alpha: float | None=…, **kwargs: Any) -> Any
  • changed: method SentencePieceProcessor.SampleEncodeAsPieces(input, nbest_size=…, alpha=…, **kwargs) → method SentencePieceProcessor.SampleEncodeAsPieces(input: str | bytes | Sequence[str] | Sequence[bytes], nbest_size: int | None=…, alpha: float | None=…, **kwargs: Any) -> Any
  • changed: method SentencePieceProcessor.SampleEncodeAsSerializedProto(input, nbest_size=…, alpha=…, **kwargs) → method SentencePieceProcessor.SampleEncodeAsSerializedProto(input: str | bytes | Sequence[str] | Sequence[bytes], nbest_size: int | None=…, alpha: float | None=…, **kwargs: Any) -> Any
  • changed: method SentencePieceProcessor.bos_id() → method SentencePieceProcessor.bos_id() -> int
  • changed: method SentencePieceProcessor.eos_id() → method SentencePieceProcessor.eos_id() -> int
  • changed: method SentencePieceProcessor.pad_id() → method SentencePieceProcessor.pad_id() -> int
  • changed: method SentencePieceProcessor.piece_size() → method SentencePieceProcessor.piece_size() -> int
  • changed: method SentencePieceProcessor.serialized_model_proto() → method SentencePieceProcessor.serialized_model_proto() -> bytes
  • changed: method SentencePieceProcessor.unk_id() → method SentencePieceProcessor.unk_id() -> int
  • changed: method SentencePieceProcessor.vocab_size() → method SentencePieceProcessor.vocab_size() -> int
  • changed: static method SentencePieceTrainer.Train(arg=…, logstream=…, **kwargs) → static method SentencePieceTrainer.Train(arg: str | None=…, logstream: IO[Any] | None=…, normalizer: SentencePieceNormalizer | None=…, **kwargs: Any) -> None
  • added: class SentencePieceNormalizer
  • added: class ThreadPool
  • added: constructor SentencePieceNormalizer(model_file: str | None=…, model_proto: bytes | None=…, normalizer_spec: bytes | None=…, rule_tsv: str | None=…, rule_name: str | None=…, norm_map: Sequence[tuple[str, str]] | None=…, add_dummy_prefix: bool=…, escape_whitespaces: bool=…, remove_extra_whitespaces: bool=…) -> None
  • added: constructor ThreadPool(num_threads: int) -> None
  • added: function SetDataDir(data_dir: str) -> None
  • added: function SetMinLogLevel(v: int) -> None
  • added: function SetNBestTimeout(timeout_ms: int) -> None
  • added: function set_min_log_level(v: int) -> None
  • added: function set_nbest_timeout(timeout_ms: int) -> None
  • added: function set_random_generator_seed(seed: int) -> None
  • added: method SentencePieceNormalizer.Decompile() -> list[tuple[str, str]]
  • added: method SentencePieceNormalizer.Normalize(input: str | bytes | Sequence[str] | Sequence[bytes], with_offsets: bool | None=…) -> Any
  • added: method SentencePieceNormalizer.decompile() -> list[tuple[str, str]]
  • added: method SentencePieceNormalizer.normalize(input: str | bytes | Sequence[str] | Sequence[bytes], with_offsets: bool | None=…) -> Any
  • added: method SentencePieceNormalizer.serialized_model_proto() -> bytes
  • added: method SentencePieceNormalizer.serialized_normalizer_spec() -> bytes
  • added: method SentencePieceProcessor.DecodeIdsAsProto(input: int | Sequence[int] | Sequence[Sequence[int]], **kwargs: Any) -> Any
  • added: method SentencePieceProcessor.DecodePiecesAsProto(input: str | bytes | Sequence[str] | Sequence[bytes] | Sequence[Sequence[str]] | Sequence[Sequence[bytes]], **kwargs: Any) -> Any
  • added: method SentencePieceProcessor.Detokenize(input: int | str | bytes | Sequence[int] | Sequence[str] | Sequence[bytes] | Sequence[Sequence[int]] | Sequence[Sequence[str]] | Sequence[Sequence[bytes]], return_type: _DecodeOutType | None=…, num_threads: int | None=…, thread_pool: ThreadPool | None=…, out_type: _DecodeOutType | None=…, return_bytes: bool | None=…) -> Any
  • added: method SentencePieceProcessor.EncodeAsNumpy(input: str | bytes | Sequence[str] | Sequence[bytes], **kwargs: Any) -> Any
  • added: method SentencePieceProcessor.EncodeAsOffsetMapping(input: str | bytes | Sequence[str] | Sequence[bytes], **kwargs: Any) -> Any
  • added: method SentencePieceProcessor.EncodeAsProto(input: str | bytes | Sequence[str] | Sequence[bytes], **kwargs: Any) -> Any
  • added: method SentencePieceProcessor.NBestEncodeAsNumpy(input: str | bytes | Sequence[str] | Sequence[bytes], nbest_size: int | None=…, **kwargs: Any) -> Any
  • added: method SentencePieceProcessor.NBestEncodeAsProto(input: str | bytes | Sequence[str] | Sequence[bytes], nbest_size: int | None=…, **kwargs: Any) -> Any
  • added: method SentencePieceProcessor.Normalize(input: str | bytes | Sequence[str] | Sequence[bytes], with_offsets: bool | None=…) -> Any
  • added: method SentencePieceProcessor.OverrideNormalizerSpec(**kwargs: object) -> bool
  • added: method SentencePieceProcessor.ParallelEncode(input: str | bytes | Sequence[str] | Sequence[bytes], return_type: _OutType | None=…, add_bos: bool | None=…, add_eos: bool | None=…, reverse: bool | None=…, emit_unk_piece: bool | None=…, chunk_len: int | None=…, num_threads: int | None=…, thread_pool: ThreadPool | None=…, out_type: _OutType | None=…) -> Any
  • added: method SentencePieceProcessor.ParallelEncodeAsIds(input: str | bytes | Sequence[str] | Sequence[bytes], **kwargs: Any) -> Any
  • added: method SentencePieceProcessor.ParallelEncodeAsImmutableProto(input: str | bytes | Sequence[str] | Sequence[bytes], **kwargs: Any) -> Any
  • added: method SentencePieceProcessor.ParallelEncodeAsNumpy(input: str | bytes | Sequence[str] | Sequence[bytes], **kwargs: Any) -> Any
  • added: method SentencePieceProcessor.ParallelEncodeAsPieces(input: str | bytes | Sequence[str] | Sequence[bytes], **kwargs: Any) -> Any
  • added: method SentencePieceProcessor.ParallelEncodeAsProto(input: str | bytes | Sequence[str] | Sequence[bytes], **kwargs: Any) -> Any
  • added: method SentencePieceProcessor.ParallelEncodeAsSerializedProto(input: str | bytes | Sequence[str] | Sequence[bytes], **kwargs: Any) -> Any
  • added: method SentencePieceProcessor.SampleEncodeAndScoreAsProto(input: str | bytes | Sequence[str] | Sequence[bytes], num_samples: int | None=…, alpha: float | None=…, **kwargs: Any) -> Any
  • added: method SentencePieceProcessor.SampleEncodeAsNumpy(input: str | bytes | Sequence[str] | Sequence[bytes], nbest_size: int | None=…, alpha: float | None=…, **kwargs: Any) -> Any
  • added: method SentencePieceProcessor.SampleEncodeAsProto(input: str | bytes | Sequence[str] | Sequence[bytes], nbest_size: int | None=…, alpha: float | None=…, **kwargs: Any) -> Any
  • added: method SentencePieceProcessor.Tokenize(input: str | bytes | Sequence[str] | Sequence[bytes], return_type: _OutType | None=…, add_bos: bool | None=…, add_eos: bool | None=…, reverse: bool | None=…, emit_unk_piece: bool | None=…, enable_sampling: bool | None=…, nbest_size: int | None=…, alpha: float | None=…, num_threads: int | None=…, thread_pool: ThreadPool | None=…, out_type: _OutType | None=…)…
  • added: method SentencePieceProcessor.calculate_entropy(input: str | Sequence[str], alpha: float, num_threads: int | None=…) -> float | list[float]
  • added: method SentencePieceProcessor.decode(input: int | str | bytes | Sequence[int] | Sequence[str] | Sequence[bytes] | Sequence[Sequence[int]] | Sequence[Sequence[str]] | Sequence[Sequence[bytes]], return_type: _DecodeOutType | None=…, num_threads: int | None=…, thread_pool: ThreadPool | None=…, out_type: _DecodeOutType | None=…, return_bytes: bool | None=…) -> Any
  • added: method SentencePieceProcessor.decode_ids(input: int | Sequence[int] | Sequence[Sequence[int]], return_type: _DecodeOutType=…, out_type: _DecodeOutType=…, **kwargs: Any) -> Any
  • added: method SentencePieceProcessor.decode_ids_as_immutable_proto(input: int | Sequence[int] | Sequence[Sequence[int]], **kwargs: Any) -> Any
  • added: method SentencePieceProcessor.decode_ids_as_proto(input: int | Sequence[int] | Sequence[Sequence[int]], **kwargs: Any) -> Any
  • added: method SentencePieceProcessor.decode_ids_as_serialized_proto(input: int | Sequence[int] | Sequence[Sequence[int]], return_type: Literal['serialized_proto']=…, out_type: Literal['serialized_proto']=…, **kwargs: Any) -> Any
  • added: method SentencePieceProcessor.decode_pieces(input: str | bytes | Sequence[str] | Sequence[bytes] | Sequence[Sequence[str]] | Sequence[Sequence[bytes]], return_type: _DecodeOutType=…, out_type: _DecodeOutType=…, **kwargs: Any) -> Any
  • added: method SentencePieceProcessor.decode_pieces_as_immutable_proto(input: str | bytes | Sequence[str] | Sequence[bytes] | Sequence[Sequence[str]] | Sequence[Sequence[bytes]], **kwargs: Any) -> Any
  • added: method SentencePieceProcessor.decode_pieces_as_proto(input: str | bytes | Sequence[str] | Sequence[bytes] | Sequence[Sequence[str]] | Sequence[Sequence[bytes]], **kwargs: Any) -> Any
  • added: method SentencePieceProcessor.decode_pieces_as_serialized_proto(input: str | bytes | Sequence[str] | Sequence[bytes] | Sequence[Sequence[str]] | Sequence[Sequence[bytes]], return_type: Literal['serialized_proto']=…, out_type: Literal['serialized_proto']=…, **kwargs: Any) -> Any
  • added: method SentencePieceProcessor.detokenize(input: int | str | bytes | Sequence[int] | Sequence[str] | Sequence[bytes] | Sequence[Sequence[int]] | Sequence[Sequence[str]] | Sequence[Sequence[bytes]], return_type: _DecodeOutType | None=…, num_threads: int | None=…, thread_pool: ThreadPool | None=…, out_type: _DecodeOutType | None=…, return_bytes: bool | None=…) -> Any
  • added: method SentencePieceProcessor.encode(input: str | bytes | Sequence[str] | Sequence[bytes], return_type: _OutType | None=…, add_bos: bool | None=…, add_eos: bool | None=…, reverse: bool | None=…, emit_unk_piece: bool | None=…, enable_sampling: bool | None=…, nbest_size: int | None=…, alpha: float | None=…, num_threads: int | None=…, thread_pool: ThreadPool | None=…, out_type: _OutType | None=…) ->…
  • added: method SentencePieceProcessor.encode_as_ids(input: str | bytes | Sequence[str] | Sequence[bytes], **kwargs: Any) -> Any
  • added: method SentencePieceProcessor.encode_as_immutable_proto(input: str | bytes | Sequence[str] | Sequence[bytes], **kwargs: Any) -> Any
  • added: method SentencePieceProcessor.encode_as_numpy(input: str | bytes | Sequence[str] | Sequence[bytes], **kwargs: Any) -> Any
  • added: method SentencePieceProcessor.encode_as_offset_mapping(input: str | bytes | Sequence[str] | Sequence[bytes], **kwargs: Any) -> Any
  • added: method SentencePieceProcessor.encode_as_pieces(input: str | bytes | Sequence[str] | Sequence[bytes], **kwargs: Any) -> Any
  • added: method SentencePieceProcessor.encode_as_proto(input: str | bytes | Sequence[str] | Sequence[bytes], **kwargs: Any) -> Any
  • added: method SentencePieceProcessor.encode_as_serialized_proto(input: str | bytes | Sequence[str] | Sequence[bytes], **kwargs: Any) -> Any
  • added: method SentencePieceProcessor.get_piece_size() -> int
  • added: method SentencePieceProcessor.get_score(id: int) -> float | (id: list[int]) -> list[float]
  • added: method SentencePieceProcessor.id_to_piece(id: int) -> str | (id: list[int]) -> list[str]
  • added: method SentencePieceProcessor.is_byte(id: int) -> bool | (id: list[int]) -> list[bool]
  • added: method SentencePieceProcessor.is_control(id: int) -> bool | (id: list[int]) -> list[bool]
  • added: method SentencePieceProcessor.is_unknown(id: int) -> bool | (id: list[int]) -> list[bool]
  • added: method SentencePieceProcessor.is_unused(id: int) -> bool | (id: list[int]) -> list[bool]
  • added: method SentencePieceProcessor.load(model_file: str | None=…, model_proto: bytes | None=…) -> bool
  • added: method SentencePieceProcessor.load_from_file(arg: str) -> bool
  • added: method SentencePieceProcessor.load_from_serialized_proto(serialized: bytes) -> bool
  • added: method SentencePieceProcessor.nbest_encode(input: str | bytes | Sequence[str] | Sequence[bytes], return_type: _OutType | None=…, add_bos: bool | None=…, add_eos: bool | None=…, reverse: bool | None=…, emit_unk_piece: bool | None=…, nbest_size: int | None=…, out_type: _OutType | None=…) -> Any
  • added: method SentencePieceProcessor.nbest_encode_as_ids(input: str | bytes | Sequence[str] | Sequence[bytes], nbest_size: int | None=…, **kwargs: Any) -> Any
  • added: method SentencePieceProcessor.nbest_encode_as_immutable_proto(input: str | bytes | Sequence[str] | Sequence[bytes], nbest_size: int | None=…, **kwargs: Any) -> Any
  • added: method SentencePieceProcessor.nbest_encode_as_numpy(input: str | bytes | Sequence[str] | Sequence[bytes], nbest_size: int | None=…, **kwargs: Any) -> Any
  • added: method SentencePieceProcessor.nbest_encode_as_pieces(input: str | bytes | Sequence[str] | Sequence[bytes], nbest_size: int | None=…, **kwargs: Any) -> Any
  • added: method SentencePieceProcessor.nbest_encode_as_proto(input: str | bytes | Sequence[str] | Sequence[bytes], nbest_size: int | None=…, **kwargs: Any) -> Any
  • added: method SentencePieceProcessor.nbest_encode_as_serialized_proto(input: str | bytes | Sequence[str] | Sequence[bytes], nbest_size: int | None=…, **kwargs: Any) -> Any
  • added: method SentencePieceProcessor.normalize(input: str | bytes | Sequence[str] | Sequence[bytes], with_offsets: bool | None=…) -> Any
  • added: method SentencePieceProcessor.override_normalizer_spec(**kwargs: object) -> bool
  • added: method SentencePieceProcessor.parallel_encode(input: str | bytes | Sequence[str] | Sequence[bytes], return_type: _OutType | None=…, add_bos: bool | None=…, add_eos: bool | None=…, reverse: bool | None=…, emit_unk_piece: bool | None=…, chunk_len: int | None=…, num_threads: int | None=…, thread_pool: ThreadPool | None=…, out_type: _OutType | None=…) -> Any
  • added: method SentencePieceProcessor.parallel_encode_as_ids(input: str | bytes | Sequence[str] | Sequence[bytes], **kwargs: Any) -> Any
  • added: method SentencePieceProcessor.parallel_encode_as_immutable_proto(input: str | bytes | Sequence[str] | Sequence[bytes], **kwargs: Any) -> Any
  • added: method SentencePieceProcessor.parallel_encode_as_numpy(input: str | bytes | Sequence[str] | Sequence[bytes], **kwargs: Any) -> Any
  • added: method SentencePieceProcessor.parallel_encode_as_pieces(input: str | bytes | Sequence[str] | Sequence[bytes], **kwargs: Any) -> Any
  • added: method SentencePieceProcessor.parallel_encode_as_proto(input: str | bytes | Sequence[str] | Sequence[bytes], **kwargs: Any) -> Any
  • added: method SentencePieceProcessor.parallel_encode_as_serialized_proto(input: str | bytes | Sequence[str] | Sequence[bytes], **kwargs: Any) -> Any
  • added: method SentencePieceProcessor.piece_to_id(piece: str) -> int | (piece: list[str]) -> list[int]
  • added: method SentencePieceProcessor.sample_encode_and_score(input: str | bytes | Sequence[str] | Sequence[bytes], return_type: _OutType | None=…, add_bos: bool | None=…, add_eos: bool | None=…, reverse: bool | None=…, emit_unk_piece: bool | None=…, num_samples: int | None=…, alpha: float | None=…, wor: bool | None=…, include_best: bool | None=…, out_type: _OutType | None=…) -> Any
  • added: method SentencePieceProcessor.sample_encode_and_score_as_ids(input: str | bytes | Sequence[str] | Sequence[bytes], num_samples: int | None=…, alpha: float | None=…, **kwargs: Any) -> Any
  • added: method SentencePieceProcessor.sample_encode_and_score_as_immutable_proto(input: str | bytes | Sequence[str] | Sequence[bytes], num_samples: int | None=…, alpha: float | None=…, **kwargs: Any) -> Any
  • added: method SentencePieceProcessor.sample_encode_and_score_as_pieces(input: str | bytes | Sequence[str] | Sequence[bytes], num_samples: int | None=…, alpha: float | None=…, **kwargs: Any) -> Any
  • added: method SentencePieceProcessor.sample_encode_and_score_as_proto(input: str | bytes | Sequence[str] | Sequence[bytes], num_samples: int | None=…, alpha: float | None=…, **kwargs: Any) -> Any
  • added: method SentencePieceProcessor.sample_encode_and_score_as_serialized_proto(input: str | bytes | Sequence[str] | Sequence[bytes], num_samples: int | None=…, alpha: float | None=…, **kwargs: Any) -> Any
  • added: method SentencePieceProcessor.sample_encode_as_ids(input: str | bytes | Sequence[str] | Sequence[bytes], nbest_size: int | None=…, alpha: float | None=…, **kwargs: Any) -> Any
  • added: method SentencePieceProcessor.sample_encode_as_immutable_proto(input: str | bytes | Sequence[str] | Sequence[bytes], nbest_size: int | None=…, alpha: float | None=…, **kwargs: Any) -> Any
  • added: method SentencePieceProcessor.sample_encode_as_numpy(input: str | bytes | Sequence[str] | Sequence[bytes], nbest_size: int | None=…, alpha: float | None=…, **kwargs: Any) -> Any
  • added: method SentencePieceProcessor.sample_encode_as_pieces(input: str | bytes | Sequence[str] | Sequence[bytes], nbest_size: int | None=…, alpha: float | None=…, **kwargs: Any) -> Any
  • added: method SentencePieceProcessor.sample_encode_as_proto(input: str | bytes | Sequence[str] | Sequence[bytes], nbest_size: int | None=…, alpha: float | None=…, **kwargs: Any) -> Any
  • added: method SentencePieceProcessor.sample_encode_as_serialized_proto(input: str | bytes | Sequence[str] | Sequence[bytes], nbest_size: int | None=…, alpha: float | None=…, **kwargs: Any) -> Any
  • added: method SentencePieceProcessor.tokenize(input: str | bytes | Sequence[str] | Sequence[bytes], return_type: _OutType | None=…, add_bos: bool | None=…, add_eos: bool | None=…, reverse: bool | None=…, emit_unk_piece: bool | None=…, enable_sampling: bool | None=…, nbest_size: int | None=…, alpha: float | None=…, num_threads: int | None=…, thread_pool: ThreadPool | None=…, out_type: _OutType | None=…)…
  • added: method ThreadPool.num_threads() -> int
  • added: static method SentencePieceProcessor.from_file(model_file: str, return_type: _OutType | None=…, **kwargs: Any) -> SentencePieceProcessor
  • added: static method SentencePieceProcessor.from_proto(model_proto: bytes, return_type: _OutType | None=…, **kwargs: Any) -> SentencePieceProcessor
  • added: static method SentencePieceTrainer.train(arg: str | None=…, logstream: IO[Any] | None=…, normalizer: SentencePieceNormalizer | None=…, **kwargs: Any) -> None
  • removed: variable ModelProto
  • removed: variable NormalizerSpec
  • removed: variable SelfTestData
  • removed: variable TrainerSpec
  • removed: variable NBestSentencePieceText
  • removed: variable SentencePieceText