Upgrading datasets 3 to 4
Breaking3.6.0 → 4.8.5Breaking5Platform1New45Changed5Fixed108Other22
BreakingUpgrade verdict
5 breaking changes and a platform requirement change.
In plain terms
- Remove unused data files optims.
- Remove pre-release workaround in CI for transformers v5 and huggingface_hub v1.
- Remove scripts altogether.
- Torchcodec decoding.
Must act Top 4 of 6 that need code or config changes
Python requirement changed in 4.6.0: >=3.9.0 → >=3.10.0.
Adds httpx <1.0.0.
Installs 35 packages in all (was 36).
What you must change
Breaking5
Remove unused data files optims.
Remove pre-release workaround in CI for transformers v5 and huggingface_hub v1.
Remove scripts altogether.
trust_remote_codeTorchcodec decoding.
Replace Sequence by List.
ListSequencedictPlatform1
Drop python 3.9.
API changes
Generated by DepDelta from published Python source. Not mixed with prose notes.
- added:
export Column - added:
export IterableColumn
- changed:
method Dataset.add_column(name: str, column: Union[list, np.array], new_fingerprint: str, feature: Optional[FeatureType]=…) → method Dataset.add_column(name: str, column: Union[list, np.ndarray], new_fingerprint: Optional[str]=…, feature: Optional[FeatureType]=…) - changed:
method Dataset.add_item(item: dict, new_fingerprint: str) → method Dataset.add_item(item: dict, new_fingerprint: Optional[str]=…) - changed:
method Dataset.to_dict(batch_size: Optional[int]=…) -> Union[dict, Iterator[dict]] → method Dataset.to_dict(batch_size: Optional[int]=…, batched: bool=…) -> Union[dict, Iterator[dict]] - changed:
static method Dataset.from_csv(path_or_paths: Union[PathLike, list[PathLike]], split: Optional[NamedSplit]=…, features: Optional[Features]=…, cache_dir: str=…, keep_in_memory: bool=…, num_proc: Optional[int]=…, **kwargs) → static method Dataset.from_csv(path_or_paths: Union[PathLike, list[PathLike]], split: Optional[NamedSplit]=…, features: Optional[Features]=…, cache_dir: str=…, keep_in_memory: bool=…, num_proc: Optional[int]=…, **kwargs) -> Dataset - changed:
static method Dataset.from_dict(mapping: dict, features: Optional[Features]=…, info: Optional[DatasetInfo]=…, split: Optional[NamedSplit]=…) -> Dataset → static method Dataset.from_dict(mapping: dict, features: Optional[Features]=…, info: Optional[DatasetInfo]=…, split: Optional[NamedSplit]=…, on_mixed_types: Optional[Literal['use_json']]=…) -> Dataset - changed:
static method Dataset.from_generator(generator: Callable, features: Optional[Features]=…, cache_dir: str=…, keep_in_memory: bool=…, gen_kwargs: Optional[dict]=…, num_proc: Optional[int]=…, split: NamedSplit=…, **kwargs) → static method Dataset.from_generator(generator: Callable, features: Optional[Features]=…, cache_dir: str=…, keep_in_memory: bool=…, gen_kwargs: Optional[dict]=…, num_proc: Optional[int]=…, split: NamedSplit=…, fingerprint: Optional[str]=…, **kwargs) -> Dataset - changed:
static method Dataset.from_json(path_or_paths: Union[PathLike, list[PathLike]], split: Optional[NamedSplit]=…, features: Optional[Features]=…, cache_dir: str=…, keep_in_memory: bool=…, field: Optional[str]=…, num_proc: Optional[int]=…, **kwargs) → static method Dataset.from_json(path_or_paths: Union[PathLike, list[PathLike]], split: Optional[NamedSplit]=…, features: Optional[Features]=…, cache_dir: str=…, keep_in_memory: bool=…, field: Optional[str]=…, num_proc: Optional[int]=…, **kwargs) -> Dataset - changed:
static method Dataset.from_list(mapping: list[dict], features: Optional[Features]=…, info: Optional[DatasetInfo]=…, split: Optional[NamedSplit]=…) -> Dataset → static method Dataset.from_list(mapping: list[dict], features: Optional[Features]=…, info: Optional[DatasetInfo]=…, split: Optional[NamedSplit]=…, on_mixed_types: Optional[Literal['use_json']]=…) -> Dataset - changed:
static method Dataset.from_parquet(path_or_paths: Union[PathLike, list[PathLike]], split: Optional[NamedSplit]=…, features: Optional[Features]=…, cache_dir: str=…, keep_in_memory: bool=…, columns: Optional[list[str]]=…, num_proc: Optional[int]=…, **kwargs) → static method Dataset.from_parquet(path_or_paths: Union[PathLike, list[PathLike]], split: Optional[NamedSplit]=…, features: Optional[Features]=…, cache_dir: str=…, keep_in_memory: bool=…, columns: Optional[list[str]]=…, num_proc: Optional[int]=…, filters: Optional[Union[pds.Expression, list[tuple], list[list[tuple]]]]=…, fragment_scan_options: Optional[pds.ParquetFragmentScanOptions]=…, on_bad_fi… - changed:
static method Dataset.from_polars(df: pl.DataFrame, features: Optional[Features]=…, info: Optional[DatasetInfo]=…, split: Optional[NamedSplit]=…) -> Dataset → static method Dataset.from_polars(df: Union[pl.DataFrame, pl.LazyFrame], features: Optional[Features]=…, info: Optional[DatasetInfo]=…, split: Optional[NamedSplit]=…) -> Dataset - changed:
static method Dataset.from_spark(df: pyspark.sql.DataFrame, split: Optional[NamedSplit]=…, features: Optional[Features]=…, keep_in_memory: bool=…, cache_dir: str=…, working_dir: str=…, load_from_cache_file: bool=…, **kwargs) → static method Dataset.from_spark(df: pyspark.sql.DataFrame, split: Optional[NamedSplit]=…, features: Optional[Features]=…, keep_in_memory: bool=…, cache_dir: str=…, working_dir: str=…, load_from_cache_file: bool=…, **kwargs) -> Dataset - changed:
static method Dataset.from_sql(sql: Union[str, sqlalchemy.sql.Selectable], con: Union[str, sqlalchemy.engine.Connection, sqlalchemy.engine.Engine, sqlite3.Connection], features: Optional[Features]=…, cache_dir: str=…, keep_in_memory: bool=…, **kwargs) → static method Dataset.from_sql(sql: Union[str, sqlalchemy.sql.Selectable], con: Union[str, sqlalchemy.engine.Connection, sqlalchemy.engine.Engine, sqlite3.Connection], features: Optional[Features]=…, cache_dir: str=…, keep_in_memory: bool=…, **kwargs) -> Dataset - changed:
static method Dataset.from_text(path_or_paths: Union[PathLike, list[PathLike]], split: Optional[NamedSplit]=…, features: Optional[Features]=…, cache_dir: str=…, keep_in_memory: bool=…, num_proc: Optional[int]=…, **kwargs) → static method Dataset.from_text(path_or_paths: Union[PathLike, list[PathLike]], split: Optional[NamedSplit]=…, features: Optional[Features]=…, cache_dir: str=…, keep_in_memory: bool=…, num_proc: Optional[int]=…, keep_linebreaks: bool=…, sample_by: Literal['line', 'paragraph', 'document']=…, **kwargs) -> Dataset - added:
class Column(Sequence_) - added:
constructor Column(source: Union[Dataset, Column], column_name: str) - added:
variable BucketNotFoundError - added:
variable HfFileSystemResolvedBucketPath - added:
variable HfFileSystemResolvedRepositoryPath
- removed:
function get_writer_batch_size(features: Optional[Features]) -> Optional[int] - removed:
method ArrowWriter.check_duplicate_keys() - changed:
constructor ArrowWriter(schema: Optional[pa.Schema]=…, features: Optional[Features]=…, path: Optional[str]=…, stream: Optional[pa.NativeFile]=…, fingerprint: Optional[str]=…, writer_batch_size: Optional[int]=…, hash_salt: Optional[str]=…, check_duplicates: Optional[bool]=…, disable_nullable: bool=…, update_features: bool=…, with_metadata: bool=…, unit: str=…, embed_local_files: bool=…, storage_op… → constructor ArrowWriter(schema: Optional[pa.Schema]=…, features: Optional[Features]=…, path: Optional[str]=…, stream: Optional[pa.NativeFile]=…, fingerprint: Optional[str]=…, writer_batch_size: Optional[int]=…, disable_nullable: bool=…, update_features: bool=…, on_mixed_types: Optional[Literal['use_json']]=…, with_metadata: bool=…, unit: str=…, embed_local_files: bool=…, storage_options: Optional… - changed:
constructor OptimizedTypedSequence(data, type: Optional[FeatureType]=…, try_type: Optional[FeatureType]=…, col: Optional[str]=…, optimized_int_type: Optional[FeatureType]=…) → constructor OptimizedTypedSequence(data, type: Optional[FeatureType]=…, try_type: Optional[FeatureType]=…, col: Optional[str]=…, optimized_int_type: Optional[FeatureType]=…, on_mixed_types: Optional[Literal['use_json']]=…) - changed:
constructor TypedSequence(data: Iterable, type: Optional[FeatureType]=…, try_type: Optional[FeatureType]=…, optimized_int_type: Optional[FeatureType]=…) → constructor TypedSequence(data: Iterable, type: Optional[FeatureType]=…, try_type: Optional[FeatureType]=…, optimized_int_type: Optional[FeatureType]=…, on_mixed_types: Optional[Literal['use_json']]=…) - changed:
method ArrowWriter.write(example: dict[str, Any], key: Optional[Union[str, int, bytes]]=…, writer_batch_size: Optional[int]=…) → method ArrowWriter.write(example: dict[str, Any], writer_batch_size: Optional[int]=…) - added:
constructor ParquetWriter(*args, use_content_defined_chunking=…, write_page_index=…, **kwargs) - added:
function get_arrow_writer_batch_size_from_features(features: Optional[Features]) -> Optional[int] - added:
function get_writer_batch_size_from_data_size(num_rows: int, num_bytes: int) -> int - added:
function get_writer_batch_size_from_features(features: Optional[Features]) -> Optional[int]
- removed:
method DatasetBuilder.download_post_processing_resources(dl_manager) - removed:
method DatasetBuilder.get_exported_dataset_info() -> DatasetInfo - removed:
property DatasetBuilder.manual_download_instructions -> Optional[str] - removed:
static method DatasetBuilder.get_all_exported_dataset_infos() -> DatasetInfosDict - changed:
method DatasetBuilder.as_dataset(split: Optional[Split]=…, run_post_process=…, verification_mode: Optional[Union[VerificationMode, str]]=…, in_memory=…) -> Union[Dataset, DatasetDict] → method DatasetBuilder.as_dataset(split: Optional[Union[str, Split, list[str], list[Split]]]=…, in_memory=…) -> Union[Dataset, DatasetDict] - added:
class Key - added:
field Key.item_or_batch_id: int - added:
field Key.original_shard_id: int
- changed:
function interleave_datasets(datasets: list[DatasetType], probabilities: Optional[list[float]]=…, seed: Optional[int]=…, info: Optional[DatasetInfo]=…, split: Optional[NamedSplit]=…, stopping_strategy: Literal['first_exhausted', 'all_exhausted']=…) -> DatasetType → function interleave_datasets(datasets: list[DatasetType], probabilities: Optional[list[float]]=…, seed: Optional[int]=…, info: Optional[DatasetInfo]=…, split: Optional[NamedSplit]=…, stopping_strategy: Literal['first_exhausted', 'all_exhausted', 'all_exhausted_without_replacement']=…) -> DatasetType
- removed:
class ConvertCommand(BaseDatasetsCLICommand) - removed:
constructor ConvertCommand(tfds_path: str, datasets_directory: str, *args) - removed:
function convert_command_factory(args: Namespace) - removed:
method ConvertCommand.run() - removed:
static method ConvertCommand.register_subcommand(parser: ArgumentParser) - removed:
variable HIGHLIGHT_MESSAGE_POST - removed:
variable HIGHLIGHT_MESSAGE_PRE - removed:
variable TO_CONVERT - removed:
variable TO_HIGHLIGHT
- removed:
class ConvertToParquetCommand(BaseDatasetsCLICommand) - removed:
constructor ConvertToParquetCommand(dataset_id: str, token: Optional[str], revision: Optional[str], trust_remote_code: bool) - removed:
method ConvertToParquetCommand.run() -> None - removed:
static method ConvertToParquetCommand.register_subcommand(parser)
- removed:
variable ARCHIVED_DATA_FILES_MAX_NUMBER_FOR_MODULE_INFERENCE - removed:
variable DATA_FILES_MAX_NUMBER_FOR_MODULE_INFERENCE - removed:
variable GLOBBED_DATA_FILES_MAX_NUMBER_FOR_MODULE_INFERENCE - removed:
variable HF_DATASETS_TRUST_REMOTE_CODE: Optional[bool] - removed:
variable TIME_OUT_REMOTE_CODE - added:
variable ARCHIVES_MAX_NUMBER_FOR_MODULE_INFERENCE - added:
variable ARROW_RECORD_BATCH_SIZE_FOR_AUDIO_DATASETS - added:
variable ARROW_RECORD_BATCH_SIZE_FOR_BINARY_DATASETS - added:
variable ARROW_RECORD_BATCH_SIZE_FOR_IMAGE_DATASETS - added:
variable ARROW_RECORD_BATCH_SIZE_FOR_VIDEO_DATASETS - added:
variable DEFAULT_CDC_OPTIONS - added:
variable MAX_ROW_GROUP_SIZE - added:
variable NIBABEL_AVAILABLE - added:
variable SAVE_ORIGINAL_SHARD_LENGTHS - added:
variable STREAMING_OPEN_MAX_RETRIES - added:
variable STREAMING_OPEN_RETRY_INTERVAL - added:
variable STREAMING_READ_RATE_LIMIT_RETRY_INTERVAL - added:
variable STREAMING_READ_SERVER_UNAVAILABLE_RETRY_INTERVAL - added:
variable TORCHCODEC_AVAILABLE
- added:
variable DEFAULT_PATTERNS_LOGS
- changed:
class DatasetDict(dict) → class DatasetDict(dict[Union[str, NamedSplit], 'Dataset']) - changed:
class IterableDatasetDict(dict) → class IterableDatasetDict(dict[Union[str, NamedSplit], IterableDataset]) - changed:
method DatasetDict.push_to_hub(repo_id, config_name: str=…, set_default: Optional[bool]=…, data_dir: Optional[str]=…, commit_message: Optional[str]=…, commit_description: Optional[str]=…, private: Optional[bool]=…, token: Optional[str]=…, revision: Optional[str]=…, create_pr: Optional[bool]=…, max_shard_size: Optional[Union[int, str]]=…, num_shards: Optional[dict[str, int]]=…, embed_external_file… → method DatasetDict.push_to_hub(repo_id: str, config_name: str=…, set_default: Optional[bool]=…, data_dir: Optional[str]=…, commit_message: Optional[str]=…, commit_description: Optional[str]=…, private: Optional[bool]=…, token: Optional[str]=…, revision: Optional[str]=…, create_pr: Optional[bool]=…, max_shard_size: Optional[Union[int, str]]=…, num_shards: Optional[dict[str, int]]=…, embed_external… - added:
method IterableDatasetDict.push_to_hub(repo_id: str, config_name: str=…, set_default: Optional[bool]=…, data_dir: Optional[str]=…, commit_message: Optional[str]=…, commit_description: Optional[str]=…, private: Optional[bool]=…, token: Optional[str]=…, revision: Optional[str]=…, create_pr: Optional[bool]=…, max_shard_size: Optional[Union[int, str]]=…, num_shards: Optional[dict[str, int]]=…, embed_… - added:
property IterableDatasetDict.column_names -> dict[str, Optional[list[str]]] - added:
property IterableDatasetDict.num_columns -> dict[str, Optional[int]] - added:
variable BucketNotFoundError - added:
variable HfFileSystemResolvedBucketPath - added:
variable HfFileSystemResolvedRepositoryPath
- added:
export Json - added:
export List - added:
export Nifti
- removed:
field Audio.mono: bool - changed:
method Audio.decode_example(value: dict, token_per_repo_id: Optional[dict[str, Union[str, bool, None]]]=…) -> dict → method Audio.decode_example(value: dict, token_per_repo_id: Optional[dict[str, Union[str, bool, None]]]=…) -> AudioDecoder - changed:
method Audio.embed_storage(storage: pa.StructArray) -> pa.StructArray → method Audio.embed_storage(storage: pa.StructArray, token_per_repo_id=…, local_files: bool=…, remote_files: bool=…) -> pa.StructArray - changed:
method Audio.encode_example(value: Union[str, bytes, bytearray, dict]) -> dict → method Audio.encode_example(value: Union[str, bytes, bytearray, dict, AudioDecoder]) -> dict - added:
field Audio.num_channels: Optional[int] - added:
field Audio.stream_index: Optional[int] - added:
function encode_torchcodec_audio(audio: AudioDecoder) -> dict
- removed:
field Sequence.dtype: ClassVar[str] - removed:
field Sequence.feature: Any - removed:
field Sequence.id: Optional[str] - removed:
field Sequence.length: int - removed:
field Sequence.pa_type: ClassVar[Any] - changed:
method Features.decode_column(column: list, column_name: str) → method Features.decode_column(column: list, column_name: str, token_per_repo_id: Optional[dict[str, Union[str, bool, None]]]=…) - added:
class Json - added:
class List(Sequence) - added:
field Json.decode: bool - added:
field Json.id: Optional[str] - added:
field Json.pa_type: ClassVar[Any] - added:
field List.feature: Any - added:
field List.id: Optional[str] - added:
field List.length: int - added:
field List.pa_type: ClassVar[Any] - added:
method Json.cast_storage(storage: Union[pa.Array]) -> pa.JsonArray - added:
method Json.decode_example(example_data, token_per_repo_id: Optional[dict[str, Union[str, bool, None]]]=…) - added:
method Json.encode_example(example_data)
- changed:
method Image.embed_storage(storage: pa.StructArray) -> pa.StructArray → method Image.embed_storage(storage: pa.StructArray, token_per_repo_id=…, local_files: bool=…, remote_files: bool=…) -> pa.StructArray
- added:
class Nifti - added:
class Nifti1ImageWrapper(nib.nifti1.Nifti1Image) - added:
constructor Nifti1ImageWrapper(nifti_image: nib.nifti1.Nifti1Image) - added:
field Nifti.decode: bool - added:
field Nifti.dtype: ClassVar[str] - added:
field Nifti.id: Optional[str] - added:
field Nifti.pa_type: ClassVar[Any] - added:
function encode_nibabel_image(img: nib.Nifti1Image, force_bytes: bool=…) -> dict[str, Optional[Union[str, bytes]]] - added:
method Nifti.cast_storage(storage: Union[pa.StringArray, pa.StructArray, pa.BinaryArray]) -> pa.StructArray - added:
method Nifti.decode_example(value: dict, token_per_repo_id=…) -> Nifti1ImageWrapper - added:
method Nifti.embed_storage(storage: pa.StructArray, token_per_repo_id=…, local_files: bool=…, remote_files: bool=…) -> pa.StructArray - added:
method Nifti.encode_example(value: Union[str, bytes, bytearray, dict, nib.Nifti1Image]) -> dict - added:
method Nifti.flatten() -> Union[FeatureType, Dict[str, FeatureType]]
- added:
method Pdf.embed_storage(storage: pa.StructArray, token_per_repo_id=…, local_files: bool=…, remote_files: bool=…) -> pa.StructArray
- removed:
function encode_torchvision_video(video: VideoReader) -> Example - changed:
function hf_video_reader(path: str, token_per_repo_id: Optional[dict[str, Union[bool, str]]]=…, stream: str=…) -> VideoReader → function hf_video_reader(path: str, token_per_repo_id: Optional[dict[str, Union[bool, str]]]=…, stream: str=…, dimension_order: Literal['NCHW', 'NHWC']=…, num_ffmpeg_threads: int=…, device: Optional[Union[str, torch.device]]=…, seek_mode: Literal['exact', 'approximate']=…) -> VideoDecoder - changed:
function video_to_bytes(video: VideoReader) -> bytes → function video_to_bytes(video: VideoDecoder) -> bytes - changed:
method Video.decode_example(value: Union[str, Example], token_per_repo_id: Optional[dict[str, Union[bool, str]]]=…) -> VideoReader → method Video.decode_example(value: Union[str, Example], token_per_repo_id: Optional[dict[str, Union[bool, str]]]=…) -> VideoDecoder - changed:
method Video.encode_example(value: Union[str, bytes, bytearray, Example, np.ndarray, VideoReader]) -> Example → method Video.encode_example(value: Union[str, bytes, bytearray, Example, np.ndarray, VideoDecoder]) -> Example - added:
field Video.device: Optional[Union[str, torch.device]] - added:
field Video.dimension_order: Literal['NCHW', 'NHWC'] - added:
field Video.num_ffmpeg_threads: int - added:
field Video.seek_mode: Literal['exact', 'approximate'] - added:
field Video.stream_index: Optional[int] - added:
function encode_torchcodec_video(video: VideoDecoder) -> Example - added:
method Video.embed_storage(storage: pa.StructArray, token_per_repo_id=…, local_files: bool=…, remote_files: bool=…) -> pa.StructArray
- removed:
field BaseCompressedFileFileSystem.extension: str - removed:
field Bz2FileSystem.extension - removed:
field GzipFileSystem.extension - removed:
field Lz4FileSystem.extension - removed:
field XzFileSystem.extension - removed:
field ZstdFileSystem.extension - added:
field BaseCompressedFileFileSystem.extensions: list[str] - added:
field Bz2FileSystem.extensions - added:
field GzipFileSystem.extensions - added:
field Lz4FileSystem.extensions - added:
field XzFileSystem.extensions - added:
field ZstdFileSystem.extensions
- removed:
function convert_to_parquet(repo_id: str, revision: Optional[str]=…, token: Optional[Union[bool, str]]=…, trust_remote_code: Optional[bool]=…) -> CommitInfo
- changed:
field DatasetInfo.splits: Optional[dict] → field DatasetInfo.splits: Optional[SplitDict]
- changed:
function get_dataset_config_names(path: str, revision: Optional[Union[str, Version]]=…, download_config: Optional[DownloadConfig]=…, download_mode: Optional[Union[DownloadMode, str]]=…, dynamic_modules_path: Optional[str]=…, data_files: Optional[Union[dict, list, str]]=…, **download_kwargs) → function get_dataset_config_names(path: str, revision: Optional[Union[str, Version]]=…, download_config: Optional[DownloadConfig]=…, download_mode: Optional[Union[DownloadMode, str]]=…, data_files: Optional[Union[dict, list, str]]=…, **download_kwargs) - changed:
function get_dataset_default_config_name(path: str, revision: Optional[Union[str, Version]]=…, download_config: Optional[DownloadConfig]=…, download_mode: Optional[Union[DownloadMode, str]]=…, dynamic_modules_path: Optional[str]=…, data_files: Optional[Union[dict, list, str]]=…, **download_kwargs) -> Optional[str] → function get_dataset_default_config_name(path: str, revision: Optional[Union[str, Version]]=…, download_config: Optional[DownloadConfig]=…, download_mode: Optional[Union[DownloadMode, str]]=…, data_files: Optional[Union[dict, list, str]]=…, **download_kwargs) -> Optional[str]
- changed:
constructor GeneratorDatasetInputStream(generator: Callable, features: Optional[Features]=…, cache_dir: str=…, keep_in_memory: bool=…, streaming: bool=…, gen_kwargs: Optional[dict]=…, num_proc: Optional[int]=…, split: NamedSplit=…, **kwargs) → constructor GeneratorDatasetInputStream(generator: Callable, features: Optional[Features]=…, cache_dir: str=…, keep_in_memory: bool=…, streaming: bool=…, gen_kwargs: Optional[dict]=…, num_proc: Optional[int]=…, split: NamedSplit=…, fingerprint: Optional[str]=…, **kwargs)
- changed:
constructor ParquetDatasetWriter(dataset: Dataset, path_or_buf: Union[PathLike, BinaryIO], batch_size: Optional[int]=…, storage_options: Optional[dict]=…, **parquet_writer_kwargs) → constructor ParquetDatasetWriter(dataset: Dataset, path_or_buf: Union[PathLike, BinaryIO], batch_size: Optional[int]=…, storage_options: Optional[dict]=…, use_content_defined_chunking: Union[bool, dict]=…, write_page_index: bool=…, **parquet_writer_kwargs)
- removed:
class ShuffledDataSourcesArrowExamplesIterable(ArrowExamplesIterable) - removed:
class ShuffledDataSourcesExamplesIterable(ExamplesIterable) - removed:
class ShufflingConfig - removed:
constructor ShuffledDataSourcesArrowExamplesIterable(generate_tables_fn: Callable[..., tuple[Key, pa.Table]], kwargs: dict, generator: np.random.Generator) - removed:
constructor ShuffledDataSourcesExamplesIterable(generate_examples_fn: Callable[..., tuple[Key, dict]], kwargs: dict, generator: np.random.Generator) - removed:
field ShufflingConfig.generator: np.random.Generator - removed:
method ShuffledDataSourcesArrowExamplesIterable.shard_data_sources(num_shards: int, index: int, contiguous=…) -> ArrowExamplesIterable - removed:
method ShuffledDataSourcesExamplesIterable.shard_data_sources(num_shards: int, index: int, contiguous=…) -> ExamplesIterable - removed:
property RepeatExamplesIterable.n_shards -> int - changed:
constructor ArrowExamplesIterable(generate_tables_fn: Callable[..., tuple[Key, pa.Table]], kwargs: dict) → constructor ArrowExamplesIterable(generate_tables_fn: Callable[..., Iterator[tuple[Key, pa.Table]]], kwargs: dict, generate_more_kwargs_fn: Optional[Callable[..., Iterator[dict]]]=…) - changed:
constructor CyclingMultiSourcesExamplesIterable(ex_iterables: list[_BaseExamplesIterable], stopping_strategy: Literal['first_exhausted', 'all_exhausted']=…) → constructor CyclingMultiSourcesExamplesIterable(ex_iterables: list[_BaseExamplesIterable], stopping_strategy: Literal['first_exhausted', 'all_exhausted', 'all_exhausted_without_replacement']=…) - changed:
constructor ExamplesIterable(generate_examples_fn: Callable[..., tuple[Key, dict]], kwargs: dict) → constructor ExamplesIterable(generate_examples_fn: Callable[..., Iterator[tuple[Key, dict]]], kwargs: dict, generate_more_kwargs_fn: Optional[Callable[..., Iterator[dict]]]=…) - changed:
constructor FormattedExamplesIterable(ex_iterable: _BaseExamplesIterable, formatting: Optional[FormattingConfig], features: Optional[Features], token_per_repo_id: dict[str, Union[str, bool, None]]) → constructor FormattedExamplesIterable(ex_iterable: _BaseExamplesIterable, formatting: Optional[FormattingConfig], features: Optional[Features], token_per_repo_id: dict[str, Union[str, bool, None]], force_convert_to_python: bool=…) - changed:
constructor IterableDataset(ex_iterable: _BaseExamplesIterable, info: Optional[DatasetInfo]=…, split: Optional[NamedSplit]=…, formatting: Optional[FormattingConfig]=…, shuffling: Optional[ShufflingConfig]=…, distributed: Optional[DistributedConfig]=…, token_per_repo_id: Optional[dict[str, Union[str, bool, None]]]=…) → constructor IterableDataset(ex_iterable: _BaseExamplesIterable, info: Optional[DatasetInfo]=…, split: Optional[NamedSplit]=…, formatting: Optional[FormattingConfig]=…, distributed: Optional[DistributedConfig]=…, token_per_repo_id: Optional[dict[str, Union[str, bool, None]]]=…) - changed:
constructor RandomlyCyclingMultiSourcesExamplesIterable(ex_iterables: list[_BaseExamplesIterable], generator: np.random.Generator, probabilities: Optional[list[float]]=…, stopping_strategy: Literal['first_exhausted', 'all_exhausted']=…) → constructor RandomlyCyclingMultiSourcesExamplesIterable(ex_iterables: list[_BaseExamplesIterable], generator: np.random.Generator, probabilities: Optional[list[float]]=…, stopping_strategy: Literal['first_exhausted', 'all_exhausted', 'all_exhausted_without_replacement']=…) - changed:
constructor RebatchedArrowExamplesIterable(ex_iterable: _BaseExamplesIterable, batch_size: Optional[int], drop_last_batch: bool=…) → constructor RebatchedArrowExamplesIterable(ex_iterable: _BaseExamplesIterable, batch_size: Optional[int], drop_last_batch: bool=…, force_convert_to_arrow: bool=…) - changed:
method RepeatExamplesIterable.shard_data_sources(worker_id: int, num_workers: int) -> RepeatExamplesIterable → method RepeatExamplesIterable.shard_data_sources(num_shards: int, index: int, contiguous=…) -> RepeatExamplesIterable - added:
class IterableColumn - added:
constructor IterableColumn(source: Union[IterableDataset, IterableColumn], column_name: str) - added:
function shift_ex_examples_rngs(ex_iterable: _BaseExamplesIterable, value: int) -> _BaseExamplesIterable - added:
method ArrowExamplesIterable.reshard_data_sources() -> ArrowExamplesIterable - added:
method BufferShuffledExamplesIterable.reshard_data_sources() -> BufferShuffledExamplesIterable - added:
method BufferShuffledExamplesIterable.shift_rngs(value: int) -> _BaseExamplesIterable - added:
method CyclingMultiSourcesExamplesIterable.reshard_data_sources() -> CyclingMultiSourcesExamplesIterable - added:
method ExamplesIterable.reshard_data_sources() -> ExamplesIterable - added:
method FilteredExamplesIterable.reshard_data_sources() -> FilteredExamplesIterable - added:
method FormattedExamplesIterable.reshard_data_sources() -> FormattedExamplesIterable - added:
method HorizontallyConcatenatedMultiSourcesExamplesIterable.reshard_data_sources() -> HorizontallyConcatenatedMultiSourcesExamplesIterable - added:
method IterableDataset.push_to_hub(repo_id: str, config_name: str=…, set_default: Optional[bool]=…, split: Optional[str]=…, data_dir: Optional[str]=…, commit_message: Optional[str]=…, commit_description: Optional[str]=…, private: Optional[bool]=…, token: Optional[str]=…, revision: Optional[str]=…, create_pr: Optional[bool]=…, max_shard_size: Optional[Union[int, str]]=…, num_shards: Optional[int]=… - added:
method IterableDataset.reshard() -> IterableDataset - added:
method IterableDataset.to_csv(path_or_buf: Union[PathLike, BinaryIO], batch_size: Optional[int]=…, storage_options: Optional[dict]=…, **to_csv_kwargs) -> int - added:
method IterableDataset.to_dict(batch_size: Optional[int]=…, batched: bool=…) -> Union[dict, Iterator[dict]] - added:
method IterableDataset.to_json(path_or_buf: Union[PathLike, BinaryIO], batch_size: Optional[int]=…, storage_options: Optional[dict]=…, **to_json_kwargs) -> int - added:
method IterableDataset.to_list() -> list - added:
method IterableDataset.to_pandas(batch_size: Optional[int]=…, batched: bool=…) -> Union[pd.DataFrame, Iterator[pd.DataFrame]] - added:
method IterableDataset.to_parquet(path_or_buf: Union[PathLike, BinaryIO], batch_size: Optional[int]=…, storage_options: Optional[dict]=…, **parquet_writer_kwargs) -> int - added:
method IterableDataset.to_polars(batch_size: Optional[int]=…, batched: bool=…, schema_overrides: Optional[dict]=…, rechunk: bool=…) -> Union[pl.DataFrame, Iterator[pl.DataFrame]] - added:
method IterableDataset.to_sql(name: str, con: Union[str, sqlalchemy.engine.Connection, sqlalchemy.engine.Engine, sqlite3.Connection], batch_size: Optional[int]=…, **sql_writer_kwargs) -> int - added:
method MappedExamplesIterable.reshard_data_sources() -> MappedExamplesIterable - added:
method RandomlyCyclingMultiSourcesExamplesIterable.reshard_data_sources() -> RandomlyCyclingMultiSourcesExamplesIterable - added:
method RandomlyCyclingMultiSourcesExamplesIterable.shift_rngs(value: int) -> _BaseExamplesIterable - added:
method RebatchedArrowExamplesIterable.reshard_data_sources() -> RebatchedArrowExamplesIterable - added:
method RepeatExamplesIterable.reshard_data_sources() -> RepeatExamplesIterable - added:
method SelectColumnsIterable.reshard_data_sources() -> SelectColumnsIterable - added:
method SkipExamplesIterable.reshard_data_sources() -> SkipExamplesIterable - added:
method StepExamplesIterable.reshard_data_sources() -> StepExamplesIterable - added:
method TakeExamplesIterable.reshard_data_sources() -> TakeExamplesIterable - added:
method VerticallyConcatenatedMultiSourcesExamplesIterable.reshard_data_sources() -> VerticallyConcatenatedMultiSourcesExamplesIterable - added:
property BufferShuffledExamplesIterable.iter_arrow - added:
property CyclingMultiSourcesExamplesIterable.iter_arrow - added:
property HorizontallyConcatenatedMultiSourcesExamplesIterable.iter_arrow - added:
property IterableDataset.num_columns -> Optional[int] - added:
property RepeatExamplesIterable.num_shards -> int - added:
property SkipExamplesIterable.iter_arrow - added:
property StepExamplesIterable.iter_arrow - added:
property TakeExamplesIterable.iter_arrow - added:
static method IterableDataset.from_csv(path_or_paths: Union[PathLike, list[PathLike]], split: Optional[NamedSplit]=…, features: Optional[Features]=…, keep_in_memory: bool=…, **kwargs) -> IterableDataset - added:
static method IterableDataset.from_dict(mapping: dict, features: Optional[Features]=…, info: Optional[DatasetInfo]=…, split: Optional[NamedSplit]=…, num_shards: Optional[int]=…) -> IterableDataset - added:
static method IterableDataset.from_json(path_or_paths: Union[PathLike, list[PathLike]], split: Optional[NamedSplit]=…, features: Optional[Features]=…, keep_in_memory: bool=…, field: Optional[str]=…, **kwargs) -> IterableDataset - added:
static method IterableDataset.from_list(mapping: list[dict], features: Optional[Features]=…, info: Optional[DatasetInfo]=…, split: Optional[NamedSplit]=…, num_shards: Optional[int]=…) -> IterableDataset - added:
static method IterableDataset.from_pandas(df: pd.DataFrame, features: Optional[Features]=…, info: Optional[DatasetInfo]=…, split: Optional[NamedSplit]=…, preserve_index: Optional[bool]=…, num_shards: Optional[int]=…) -> IterableDataset - added:
static method IterableDataset.from_parquet(path_or_paths: Union[PathLike, list[PathLike]], split: Optional[NamedSplit]=…, features: Optional[Features]=…, keep_in_memory: bool=…, columns: Optional[list[str]]=…, filters: Optional[Union[pds.Expression, list[tuple], list[list[tuple]]]]=…, fragment_scan_options: Optional[pds.ParquetFragmentScanOptions]=…, on_bad_files: Literal['error', 'warn', 'skip']… - added:
static method IterableDataset.from_polars(df: Union[pl.DataFrame, pl.LazyFrame], features: Optional[Features]=…, info: Optional[DatasetInfo]=…, split: Optional[NamedSplit]=…) -> IterableDataset - added:
static method IterableDataset.from_text(path_or_paths: Union[PathLike, list[PathLike]], split: Optional[NamedSplit]=…, features: Optional[Features]=…, keep_in_memory: bool=…, keep_linebreaks: bool=…, sample_by: Literal['line', 'paragraph', 'document']=…, **kwargs) -> IterableDataset - added:
variable BucketNotFoundError - added:
variable HfFileSystemResolvedBucketPath - added:
variable HfFileSystemResolvedRepositoryPath
- removed:
class DuplicatedKeysError(Exception) - removed:
class InvalidKeyError(Exception) - removed:
class KeyHasher - removed:
constructor DuplicatedKeysError(key, duplicate_key_indices, fix_msg=…) - removed:
constructor InvalidKeyError(hash_data) - removed:
constructor KeyHasher(hash_salt: str) - removed:
method KeyHasher.hash(key: Union[str, int, bytes]) -> int
- removed:
class HubDatasetModuleFactoryWithScript(_DatasetModuleFactory) - removed:
class HubDatasetModuleFactoryWithoutScript(_DatasetModuleFactory) - removed:
class LocalDatasetModuleFactoryWithScript(_DatasetModuleFactory) - removed:
class LocalDatasetModuleFactoryWithoutScript(_DatasetModuleFactory) - removed:
constructor HubDatasetModuleFactoryWithScript(name: str, commit_hash: str, download_config: Optional[DownloadConfig]=…, download_mode: Optional[Union[DownloadMode, str]]=…, dynamic_modules_path: Optional[str]=…, trust_remote_code: Optional[bool]=…) - removed:
constructor HubDatasetModuleFactoryWithoutScript(name: str, commit_hash: str, data_dir: Optional[str]=…, data_files: Optional[Union[str, list, dict]]=…, download_config: Optional[DownloadConfig]=…, download_mode: Optional[Union[DownloadMode, str]]=…, use_exported_dataset_infos: bool=…) - removed:
constructor LocalDatasetModuleFactoryWithScript(path: str, download_config: Optional[DownloadConfig]=…, download_mode: Optional[Union[DownloadMode, str]]=…, dynamic_modules_path: Optional[str]=…, trust_remote_code: Optional[bool]=…) - removed:
constructor LocalDatasetModuleFactoryWithoutScript(path: str, data_dir: Optional[str]=…, data_files: Optional[Union[str, list, dict]]=…, download_mode: Optional[Union[DownloadMode, str]]=…) - removed:
field DatasetModule.importable_file_path: Optional[str] - removed:
function files_to_hash(file_paths: list[str]) -> str - removed:
function init_dynamic_modules(name: str=…, hf_modules_cache: Optional[Union[Path, str]]=…) - removed:
function resolve_trust_remote_code(trust_remote_code: Optional[bool], repo_id: str) -> bool - removed:
method HubDatasetModuleFactoryWithScript.download_dataset_infos_file() -> str - removed:
method HubDatasetModuleFactoryWithScript.download_dataset_readme_file() -> str - removed:
method HubDatasetModuleFactoryWithScript.download_loading_script() -> str - removed:
method HubDatasetModuleFactoryWithScript.get_module() -> DatasetModule - removed:
method HubDatasetModuleFactoryWithoutScript.get_module() -> DatasetModule - removed:
method LocalDatasetModuleFactoryWithScript.get_module() -> DatasetModule - removed:
method LocalDatasetModuleFactoryWithoutScript.get_module() -> DatasetModule - removed:
variable ALL_ALLOWED_EXTENSIONS - changed:
constructor CachedDatasetModuleFactory(name: str, cache_dir: Optional[str]=…, dynamic_modules_path: Optional[str]=…) → constructor CachedDatasetModuleFactory(name: str, cache_dir: Optional[str]=…) - changed:
function dataset_module_factory(path: str, revision: Optional[Union[str, Version]]=…, download_config: Optional[DownloadConfig]=…, download_mode: Optional[Union[DownloadMode, str]]=…, dynamic_modules_path: Optional[str]=…, data_dir: Optional[str]=…, data_files: Optional[Union[dict, list, str, DataFilesDict]]=…, cache_dir: Optional[str]=…, trust_remote_code: Optional[bool]=…, _require_default_conf… → function dataset_module_factory(path: str, revision: Optional[Union[str, Version]]=…, download_config: Optional[DownloadConfig]=…, download_mode: Optional[Union[DownloadMode, str]]=…, data_dir: Optional[str]=…, data_files: Optional[Union[dict, list, str, DataFilesDict]]=…, cache_dir: Optional[str]=…, **download_kwargs) -> DatasetModule - changed:
function load_dataset(path: str, name: Optional[str]=…, data_dir: Optional[str]=…, data_files: Optional[Union[str, Sequence[str], Mapping[str, Union[str, Sequence[str]]]]]=…, split: Optional[Union[str, Split]]=…, cache_dir: Optional[str]=…, features: Optional[Features]=…, download_config: Optional[DownloadConfig]=…, download_mode: Optional[Union[DownloadMode, str]]=…, verification_mode: Optional[… → function load_dataset(path: str, name: Optional[str]=…, data_dir: Optional[str]=…, data_files: Optional[Union[str, Sequence[str], Mapping[str, Union[str, Sequence[str]]]]]=…, split: None=…, cache_dir: Optional[str]=…, features: Optional[Features]=…, download_config: Optional[DownloadConfig]=…, download_mode: Optional[Union[DownloadMode, str]]=…, verification_mode: Optional[Union[VerificationMode,… - added:
class HubBucketDatasetModuleFactory(_DatasetModuleFactory) - added:
class HubDatasetModuleFactory(_DatasetModuleFactory) - added:
class LocalDatasetModuleFactory(_DatasetModuleFactory) - added:
constructor HubBucketDatasetModuleFactory(path: str, data_dir: Optional[str]=…, data_files: Optional[Union[str, list, dict]]=…, download_config: Optional[DownloadConfig]=…, download_mode: Optional[Union[DownloadMode, str]]=…) - added:
constructor HubDatasetModuleFactory(name: str, commit_hash: str, data_dir: Optional[str]=…, data_files: Optional[Union[str, list, dict]]=…, download_config: Optional[DownloadConfig]=…, download_mode: Optional[Union[DownloadMode, str]]=…, use_exported_dataset_infos: bool=…) - added:
constructor LocalDatasetModuleFactory(path: str, data_dir: Optional[str]=…, data_files: Optional[Union[str, list, dict]]=…, download_mode: Optional[Union[DownloadMode, str]]=…) - added:
method HubBucketDatasetModuleFactory.get_module() -> DatasetModule - added:
method HubDatasetModuleFactory.get_module() -> DatasetModule - added:
method LocalDatasetModuleFactory.get_module() -> DatasetModule - added:
variable BucketNotFoundError
- added:
export eval - added:
export hdf5 - added:
export lance - added:
export niftifolder
- added:
class Eval(datasets.GeneratorBasedBuilder) - added:
field Eval.NUM_EXAMPLES_FOR_FEATURES_INFERENCE - added:
variable logger
- added:
field FolderBasedBuilderConfig.metadata_filenames: list[str]
- added:
class HDF5(datasets.ArrowBasedBuilder) - added:
class HDF5Config(datasets.BuilderConfig) - added:
field HDF5.BUILDER_CONFIG_CLASS - added:
field HDF5Config.batch_size: Optional[int] - added:
field HDF5Config.features: Optional[datasets.Features] - added:
variable EXTENSIONS - added:
variable logger
- removed:
function ujson_dumps(*args, **kwargs) - removed:
function ujson_loads(*args, **kwargs) - added:
class FullReadDisallowed(Exception) - added:
field JsonConfig.on_mixed_types: Optional[Literal['use_json']] - added:
field JsonConfig.parse_agent_traces: bool - added:
function has_agent_traces_markers(features: datasets.Features) -> bool - added:
function parse_traces_info(traces: list[str]) -> tuple[Optional[str], Optional[str]] - added:
variable AGENT_TRACES_FEATURES - added:
variable AGENT_TRACES_FEATURES_MARKERS - added:
variable AGENT_TRACES_TYPES_VALUES - added:
variable AGENT_TRACES_TYPE_TO_HARNESS
- added:
class Lance(datasets.ArrowBasedBuilder, datasets.builder._CountableBuilderMixin) - added:
class LanceConfig(datasets.BuilderConfig) - added:
field Lance.BUILDER_CONFIG_CLASS - added:
field Lance.METADATA_EXTENSIONS - added:
field LanceConfig.batch_size: Optional[int] - added:
field LanceConfig.columns: Optional[List[str]] - added:
field LanceConfig.features: Optional[datasets.Features] - added:
field LanceConfig.token: Optional[str] - added:
function resolve_dataset_uris(files: List[str]) -> Dict[str, List[str]] - added:
variable MAGIC_BYTES_EXTENSION_AND_FEATURE_TYPES - added:
variable logger
- added:
class NiftiFolder(folder_based_builder.FolderBasedBuilder) - added:
class NiftiFolderConfig(folder_based_builder.FolderBasedBuilderConfig) - added:
field NiftiFolder.BASE_COLUMN_NAME - added:
field NiftiFolder.BASE_FEATURE - added:
field NiftiFolder.BUILDER_CONFIG_CLASS - added:
field NiftiFolder.EXTENSIONS: list[str] - added:
field NiftiFolderConfig.drop_labels: bool - added:
field NiftiFolderConfig.drop_metadata: bool - added:
variable logger
- added:
field ParquetConfig.fragment_scan_options: Optional[ds.ParquetFragmentScanOptions] - added:
field ParquetConfig.on_bad_files: Literal['error', 'warn', 'skip']
- changed:
field TextConfig.sample_by: str → field TextConfig.sample_by: Literal['line', 'paragraph', 'document']
- changed:
class SplitDict(dict) → class SplitDict(dict[str, SplitInfo]) - changed:
constructor NamedSplit(name) → constructor NamedSplit(name: str) - added:
field SplitInfo.original_shard_lengths: Optional[list[int]]
- changed:
function embed_array_storage(array: pa.Array, feature: FeatureType) → function embed_array_storage(array: pa.Array, feature: FeatureType, token_per_repo_id=…, local_files: bool=…, remote_files: bool=…) - changed:
function embed_table_storage(table: pa.Table) → function embed_table_storage(table: pa.Table, token_per_repo_id=…, local_files: bool=…, remote_files: bool=…)
- changed:
static method TarExtractor.safemembers(members, output_path) → static method TarExtractor.safemembers(members: tarfile.TarFile, output_path: Union[Path, str]) - added:
static method RarExtractor.safemembers(members: list[rarfile.RarInfo], output_path: Union[Path, str]) - added:
static method SevenZipExtractor.safemembers(members: list[py7zr.FileInfo], output_path: Union[Path, str]) - added:
static method ZipExtractor.safemembers(members: list[zipfile.ZipInfo], output_path: Union[Path, str])
- removed:
function init_hf_modules(hf_modules_cache: Optional[Union[Path, str]]=…) -> str - added:
variable CONNECTION_ERRORS_TO_RETRY - added:
variable RATE_LIMIT_CODE - added:
variable SERVER_UNAVAILABLE_CODE
- removed:
function check_auth(hf_api, repo_id, token=…)
- added:
function find_mixed_struct_types_field_paths(examples: list, allow_root=…) -> list[list[str]] - added:
function get_json_field_path_from_pyarrow_json_error(err_str: str) -> list[str] - added:
function get_json_field_paths_from_feature(feature: FeatureType) -> list[list[str]] - added:
function insert_json_field_path(json_field_paths: list[list[str]], json_field_path: list[str]) -> None - added:
function json_decode_field(example: Any, json_field_path: str) -> Any - added:
function json_encode_field(example: Any, json_field_path: list[str]) -> Any - added:
function json_encode_fields_in_json_lines(original_batch: bytes, json_field_paths: list[list[str]]) -> bytes - added:
function set_json_types_in_feature(feature: FeatureType, json_field_paths: list[list[str]]) -> None - added:
function ujson_dumps(*args, **kwargs) - added:
function ujson_loads(*args, **kwargs)
- removed:
function get_imports(file_path: str) -> tuple[str, str, str, str] - removed:
function lock_importable_file(importable_local_file: str) -> FileLock