Upgrading dask 2024 to 2025
Breaking2024.12.1 → 2025.12.0Breaking8New28Changed51Fixed91Other108
BreakingUpgrade verdict
8 breaking changes.
In plain terms
- Removes bytes as keys.
- Removes an old way of handling time series data, so code relying on it will stop working.
- Cleans out leftover code related to the fastparquet engine.
- Code that referred to the old remote store name must use FsspecStore instead.
Must act Top 3 of 8 that need code or config changes
What you must change
Breaking8
Removes bytes as keys.
Drops the legacy timeseries implementation.
Removes remaining artifacts of fastparquet.
fastparquetRenames remote store to FsspecStore.
FsspecStoreRemoves IO wrapper functions.
Renames data_producer and adds a flag to dataframe io stuff.
data_producerRemoves subgraph callable.
Removes the legacy DataFrame implementation.
DataFrameAPI changes
Generated by DepDelta from published Python source. Not mixed with prose notes.
- removed:
export get_versions - removed:
variable versions - added:
export Expr - added:
export HLGExpr - added:
export LLGExpr - added:
export SingletonExpr
- removed:
export da - removed:
function raise_not_implemented_error(attr_name) - removed:
variable ARRAY_EXPR_ENABLED: bool | None - removed:
variable abs - removed:
variable absolute - removed:
variable add - removed:
variable all - removed:
variable angle - removed:
variable any - removed:
variable apply_gufunc - removed:
variable arccos - removed:
variable arccosh - removed:
variable arcsin - removed:
variable arcsinh - removed:
variable arctan - removed:
variable arctan2 - removed:
variable arctanh - removed:
variable array - removed:
variable as_gufunc - removed:
variable asanyarray - removed:
variable bitwise_and - removed:
variable bitwise_not - removed:
variable bitwise_or - removed:
variable bitwise_xor - removed:
variable blockwise - removed:
variable cbrt - removed:
variable ceil - removed:
variable clip - removed:
variable concatenate - removed:
variable conj - removed:
variable copysign - removed:
variable cos - removed:
variable cosh - removed:
variable da - removed:
variable deg2rad - removed:
variable degrees - removed:
variable divide - removed:
variable divmod - removed:
variable empty_like - removed:
variable equal - removed:
variable exp - removed:
variable exp2 - removed:
variable expm1 - removed:
variable fabs - removed:
variable fix - removed:
variable float_power - removed:
variable floor - removed:
variable floor_divide - removed:
variable fmax - removed:
variable fmin - removed:
variable fmod - removed:
variable frexp - removed:
variable frompyfunc - removed:
variable full - removed:
variable full_like - removed:
variable greater - removed:
variable greater_equal - removed:
variable gufunc - removed:
variable hypot - removed:
variable i0 - removed:
variable imag - removed:
variable invert - removed:
variable iscomplex - removed:
variable isfinite - removed:
variable isinf - removed:
variable isnan - removed:
variable isneginf - removed:
variable isposinf - removed:
variable isreal - removed:
variable ldexp - removed:
variable left_shift - removed:
variable less - removed:
variable less_equal - removed:
variable log - removed:
variable log10 - removed:
variable log1p - removed:
variable log2 - removed:
variable logaddexp - removed:
variable logaddexp2 - removed:
variable logical_and - removed:
variable logical_not - removed:
variable logical_or - removed:
variable logical_xor - removed:
variable map_blocks - removed:
variable map_overlap - removed:
variable max - removed:
variable maximum - removed:
variable min - removed:
variable minimum - removed:
variable mod - removed:
variable modf - removed:
variable msg - removed:
variable multiply - removed:
variable nan_to_num - removed:
variable nanmax - removed:
variable nanmin - removed:
variable nanprod - removed:
variable negative - removed:
variable nextafter - removed:
variable not_equal - removed:
variable ones_like - removed:
variable overlap - removed:
variable positive - removed:
variable power - removed:
variable rad2deg - removed:
variable radians - removed:
variable real - removed:
variable rechunk - removed:
variable reciprocal - removed:
variable reduction - removed:
variable remainder - removed:
variable repeat - removed:
variable right_shift - removed:
variable rint - removed:
variable sign - removed:
variable signbit - removed:
variable sin - removed:
variable sinc - removed:
variable sinh - removed:
variable spacing - removed:
variable sqrt - removed:
variable square - removed:
variable stack - removed:
variable subtract - removed:
variable tan - removed:
variable tanh - removed:
variable true_divide - removed:
variable trunc - removed:
variable zeros_like - added:
export bool - added:
export complex128 - added:
export complex64 - added:
export e - added:
export euler_gamma - added:
export float32 - added:
export float64 - added:
export inf - added:
export int16 - added:
export int32 - added:
export int64 - added:
export int8 - added:
export nan - added:
export newaxis - added:
export pi - added:
export uint16 - added:
export uint32 - added:
export uint64 - added:
export uint8 - added:
function array_expr_enabled()
- removed:
function insert_to_ooc(keys: list, chunks: tuple[tuple[int, ...], ...], out: ArrayLike, name: str, *, lock: Lock | bool=…, region: tuple[slice, ...] | slice | None=…, return_stored: bool=…, load_stored: bool=…) -> dict - removed:
function retrieve_from_ooc(keys: Collection[Key], dsk_pre: Graph, dsk_post: Graph) -> dict[tuple, Any] - removed:
function store_chunk(x: ArrayLike, out: ArrayLike, index: slice, lock: Any, return_stored: bool) - changed:
function load_chunk(out: A, index: slice, lock: Any) -> A → function load_chunk(out: A, index: slice, lock: Any, region: slice | None) -> A - changed:
function load_store_chunk(x: Any, out: Any, index: slice, lock: Any, return_stored: bool, load_stored: bool) → function load_store_chunk(x: Any, out: Any, index: slice | None, region: slice | None, lock: Any, return_stored: bool, load_stored: bool) -> Any - changed:
function to_zarr(arr, url, component=…, storage_options=…, overwrite=…, region=…, compute=…, return_stored=…, **kwargs) → function to_zarr(arr, url, component=…, storage_options=…, region=…, compute=…, return_stored=…, zarr_array_kwargs=…, zarr_read_kwargs=…, **kwargs) - added:
function concatenate_shaped(arrays, shape) - added:
function normalize_chunks_cached(chunks, shape=…, limit=…, dtype=…, previous_chunks=…) - added:
variable xr
- changed:
function arange(*args, chunks=…, like=…, dtype=…, **kwargs) → function arange(start=…, /, stop=…, step=…, *, chunks=…, like=…, dtype=…) - changed:
function get_pad_shapes_chunks(array, pad_width, axes) → function get_pad_shapes_chunks(array, pad_width, axes, mode)
- added:
variable NUMPY_GE_220 - added:
variable NUMPY_GE_240
- added:
variable GETNOREMOVE - added:
variable GETTERS
- removed:
function partial_reduce(func, x, split_every, keepdims=…, dtype=…, name=…, reduced_meta=…) - removed:
function reduction(x, chunk, aggregate, axis=…, keepdims=…, dtype=…, split_every=…, combine=…, name=…, out=…, concatenate=…, output_size=…, meta=…, weights=…) - added:
variable numbagg
- changed:
function cov(m, y=…, rowvar=…, bias=…, ddof=…) → function cov(m, y=…, rowvar=…, bias=…, ddof=…, fweights=…, aweights=…, *, dtype=…)
- changed:
function slice_array(out_name, in_name, blockdims, index, itemsize) → function slice_array(out_name, in_name, blockdims, index) - added:
class SlicingNoop(Exception)
- removed:
variable msg
- changed:
function blockwise(func, output, output_indices, *arrind_pairs, numblocks=…, concatenate=…, new_axes=…, dependencies=…, **kwargs) → function blockwise(func, output, output_indices, *arrind_pairs, numblocks, concatenate=…, new_axes=…, dependencies=…, _data_producer=…, **kwargs) - added:
property Blockwise.has_legacy_tasks
- added:
function reshapelist(shape, seq)
- removed:
export dask - removed:
export dd - removed:
export methods - removed:
export optimize - removed:
export rolling - removed:
export test_dataframe - removed:
function raise_not_implemented_error(attr_name) - removed:
variable DASK_EXPR_ENABLED: bool | None - removed:
variable dd - removed:
variable msg - added:
export Scalar - added:
export from_graph - added:
export get_collection_type
- removed:
class Accessor - removed:
class DatetimeAccessor(Accessor) - removed:
class StringAccessor(Accessor) - removed:
constructor Accessor(series) - removed:
function str_cat(self, *others, **kwargs) - removed:
function str_extractall(series, pat, flags) - removed:
function str_get(series, index) - removed:
method DatetimeAccessor.isocalendar() - removed:
method StringAccessor.cat(others=…, sep=…, na_rep=…) - removed:
method StringAccessor.endswith(*args, **kwargs) - removed:
method StringAccessor.extract(*args, **kwargs) - removed:
method StringAccessor.extractall(pat, flags=…) - removed:
method StringAccessor.rsplit(pat=…, n=…, expand=…) - removed:
method StringAccessor.split(pat=…, n=…, expand=…) - removed:
method StringAccessor.startswith(*args, **kwargs)
- removed:
function get_parallel_type_frame(o) - changed:
static method PandasBackendEntrypoint.to_backend(data: _Frame, **kwargs) → static method PandasBackendEntrypoint.to_backend(data, **kwargs) - added:
variable HAS_PYARROW
- removed:
class CategoricalAccessor(Accessor) - removed:
function categorize(df, columns=…, index=…, split_every=…, **kwargs) - removed:
method CategoricalAccessor.as_known(**kwargs) - removed:
method CategoricalAccessor.as_unknown() - removed:
method CategoricalAccessor.remove_unused_categories() - removed:
property CategoricalAccessor.categories - removed:
property CategoricalAccessor.codes - removed:
property CategoricalAccessor.known - removed:
property CategoricalAccessor.ordered
- removed:
class DataFrame(_Frame) - removed:
class Index(Series) - removed:
class Scalar(DaskMethodsMixin, OperatorMethodMixin) - removed:
class Series(_Frame) - removed:
constructor DataFrame(dsk, name, meta, divisions) - removed:
constructor Scalar(dsk, name, meta, divisions=…) - removed:
field Series.cat - removed:
field Series.dt - removed:
field Series.str - removed:
function apply_concat_apply(args, chunk=…, aggregate=…, combine=…, meta=…, token=…, chunk_kwargs=…, aggregate_kwargs=…, combine_kwargs=…, split_every=…, split_out=…, split_out_setup=…, split_out_setup_kwargs=…, sort=…, ignore_index=…, **kwargs) - removed:
function elemwise(op, *args, meta=…, out=…, transform_divisions=…, **kwargs) - removed:
function finalize(results) - removed:
function handle_out(out, result) - removed:
function hash_shard(df, nparts, split_out_setup=…, split_out_setup_kwargs=…, ignore_index=…) - removed:
function is_broadcastable(dfs, s) - removed:
function isna(arg) - removed:
function map_partitions(func, *args, meta=…, enforce_metadata=…, transform_divisions=…, align_dataframes=…, **kwargs) - removed:
function mapseries(base_chunk, concat_map) - removed:
function mapseries_combine(index, concat_result) - removed:
function maybe_shift_divisions(df, periods, freq) - removed:
function new_dd_object(dsk, name, meta, divisions, parent_meta=…) - removed:
function partitionwise_graph(func, layer_name, *args, **kwargs) - removed:
function pd_split(df, p, random_state=…, shuffle=…) - removed:
function prefix_reduction(f, ddf, identity, **kwargs) - removed:
function quantile(df, q, method=…) - removed:
function repartition(df, divisions=…, force=…) - removed:
function repartition_divisions(a, b, name, out1, out2, force=…) - removed:
function repartition_freq(df, freq=…) - removed:
function repartition_npartitions(df, npartitions) - removed:
function repartition_size(df, size) - removed:
function series_map(base_series, map_series) - removed:
function split_out_on_cols(df, cols=…) - removed:
function split_out_on_index(df) - removed:
function suffix_reduction(f, ddf, identity, **kwargs) - removed:
function to_datetime(arg, meta=…, **kwargs) - removed:
function to_timedelta(arg, unit=…, errors=…) - removed:
method DataFrame.apply(func, axis=…, broadcast=…, raw=…, reduce=…, args=…, meta=…, result_type=…, **kwds) - removed:
method DataFrame.applymap(func, meta=…) - removed:
method DataFrame.assign(**kwargs) - removed:
method DataFrame.categorize(columns=…, index=…, split_every=…, **kwargs) - removed:
method DataFrame.clip(lower=…, upper=…, axis=…) - removed:
method DataFrame.corr(method=…, min_periods=…, numeric_only=…, split_every=…) - removed:
method DataFrame.cov(min_periods=…, numeric_only=…, split_every=…) - removed:
method DataFrame.drop(labels=…, axis=…, columns=…, errors=…) - removed:
method DataFrame.dropna(how=…, subset=…, thresh=…) - removed:
method DataFrame.eval(expr, inplace=…, **kwargs) - removed:
method DataFrame.explode(column) - removed:
method DataFrame.groupby(by=…, group_keys=…, sort=…, observed=…, dropna=…, **kwargs) - removed:
method DataFrame.info(buf=…, verbose=…, memory_usage=…) - removed:
method DataFrame.items() - removed:
method DataFrame.iterrows() - removed:
method DataFrame.itertuples(index=…, name=…) - removed:
method DataFrame.join(other, on=…, how=…, lsuffix=…, rsuffix=…, npartitions=…, shuffle_method=…) - removed:
method DataFrame.map(func, meta=…, na_action=…) - removed:
method DataFrame.melt(id_vars=…, value_vars=…, var_name=…, value_name=…, col_level=…) - removed:
method DataFrame.memory_usage(index=…, deep=…) - removed:
method DataFrame.merge(right, how=…, on=…, left_on=…, right_on=…, left_index=…, right_index=…, suffixes=…, indicator=…, npartitions=…, shuffle_method=…, broadcast=…) - removed:
method DataFrame.mode(dropna=…, split_every=…, numeric_only=…) - removed:
method DataFrame.nlargest(n=…, columns=…, split_every=…) - removed:
method DataFrame.nsmallest(n=…, columns=…, split_every=…) - removed:
method DataFrame.nunique(split_every=…, dropna=…, axis=…) - removed:
method DataFrame.pivot_table(index=…, columns=…, values=…, aggfunc=…) - removed:
method DataFrame.pop(item) - removed:
method DataFrame.query(expr, **kwargs) - removed:
method DataFrame.rename(index=…, columns=…) - removed:
method DataFrame.round(decimals=…) - removed:
method DataFrame.select_dtypes(include=…, exclude=…) - removed:
method DataFrame.set_index(other: str | Series, drop: bool=…, sorted: bool=…, npartitions: int | Literal['auto'] | None=…, divisions: Sequence | None=…, inplace: bool=…, sort: bool=…, **kwargs) - removed:
method DataFrame.sort_values(by: str | list[str], npartitions: int | Literal['auto'] | None=…, ascending: bool=…, na_position: Literal['first'] | Literal['last']=…, sort_function: Callable[[pd.DataFrame], pd.DataFrame] | None=…, sort_function_kwargs: Mapping[str, Any] | None=…, **kwargs) -> DataFrame - removed:
method DataFrame.squeeze(axis=…) - removed:
method DataFrame.to_bag(index=…, format=…) - removed:
method DataFrame.to_html(max_rows=…) - removed:
method DataFrame.to_orc(path, *args, **kwargs) - removed:
method DataFrame.to_parquet(path, *args, **kwargs) - removed:
method DataFrame.to_records(index=…, lengths=…) - removed:
method DataFrame.to_string(max_rows=…) - removed:
method DataFrame.to_timestamp(freq=…, how=…, axis=…) - removed:
method Index.count(split_every=…) - removed:
method Index.drop_duplicates(split_every=…, split_out=…, shuffle_method=…, **kwargs) - removed:
method Index.head(n=…, compute=…) - removed:
method Index.map(arg, na_action=…, meta=…, is_monotonic=…) - removed:
method Index.max(split_every=…) - removed:
method Index.memory_usage(deep=…) - removed:
method Index.min(split_every=…) - removed:
method Index.shift(periods=…, freq=…) - removed:
method Index.to_frame(index=…, name=…) - removed:
method Index.to_series() - removed:
method Scalar.to_delayed(optimize_graph=…) - removed:
method Series.align(other, join=…, axis=…, fill_value=…) - removed:
method Series.apply(func, convert_dtype=…, meta=…, args=…, **kwds) - removed:
method Series.autocorr(lag=…, split_every=…) - removed:
method Series.between(left, right, inclusive=…) - removed:
method Series.clip(lower=…, upper=…, axis=…) - removed:
method Series.combine(other, func, fill_value=…) - removed:
method Series.combine_first(other) - removed:
method Series.corr(other, method=…, min_periods=…, split_every=…) - removed:
method Series.count(split_every=…) - removed:
method Series.cov(other, min_periods=…, split_every=…) - removed:
method Series.dropna() - removed:
method Series.explode() - removed:
method Series.groupby(by=…, group_keys=…, sort=…, observed=…, dropna=…, **kwargs) - removed:
method Series.isin(values) - removed:
method Series.map(arg, na_action=…, meta=…) - removed:
method Series.median(method=…) - removed:
method Series.median_approximate(method=…) - removed:
method Series.memory_usage(index=…, deep=…) - removed:
method Series.mode(dropna=…, split_every=…) - removed:
method Series.nlargest(n=…, split_every=…) - removed:
method Series.nsmallest(n=…, split_every=…) - removed:
method Series.nunique(split_every=…, dropna=…) - removed:
method Series.quantile(q=…, method=…) - removed:
method Series.rename(index=…, inplace=…, sorted_index=…) - removed:
method Series.round(decimals=…) - removed:
method Series.squeeze() - removed:
method Series.to_bag(index=…, format=…) - removed:
method Series.to_frame(name=…) - removed:
method Series.to_string(max_rows=…) - removed:
method Series.to_timestamp(freq=…, how=…, axis=…) - removed:
method Series.unique(split_every=…, split_out=…) - removed:
method Series.value_counts(sort=…, ascending=…, dropna=…, normalize=…, split_every=…, split_out=…) - removed:
method Series.view(dtype) - removed:
property DataFrame.axes - removed:
property DataFrame.columns - removed:
property DataFrame.dtypes - removed:
property DataFrame.empty - removed:
property DataFrame.iloc - removed:
property DataFrame.ndim - removed:
property DataFrame.shape - removed:
property Index.index - removed:
property Index.is_monotonic_decreasing - removed:
property Index.is_monotonic_increasing - removed:
property Scalar.divisions - removed:
property Scalar.dtype - removed:
property Scalar.key - removed:
property Series.axes - removed:
property Series.dtype - removed:
property Series.is_monotonic_decreasing - removed:
property Series.is_monotonic_increasing - removed:
property Series.name - removed:
property Series.nbytes - removed:
property Series.ndim - removed:
property Series.shape - removed:
static method DataFrame.from_dict(data, *, npartitions, orient=…, dtype=…, columns=…) - removed:
variable GROUP_KEYS_DEFAULT: bool | None - removed:
variable aca - changed:
function meta_warning(df) → function meta_warning(df, method=…)
- added:
export Aggregation - added:
export datasets - added:
export from_delayed - added:
export get_dummies - added:
export to_bag - added:
export to_parquet - added:
export to_records
- added:
function timeseries(start=…, end=…, freq=…, partition_freq=…, dtypes=…, seed=…, **kwargs)
- added:
export analyze - added:
export explain
- added:
function to_bag(df, index=…, format=…)
- added:
class BlockwiseIO(Blockwise, IO) - added:
class FromArray(PartitionsFiltered, BlockwiseIO) - added:
class FromGraph(IO) - added:
class FromMap(PartitionsFiltered, BlockwiseIO) - added:
class FromMapProjectable(FromMap) - added:
class FromPandas(PartitionsFiltered, BlockwiseIO) - added:
class FromPandasDivisions(FromPandas) - added:
class FromScalars(IO) - added:
class FusedIO(BlockwiseIO) - added:
class FusedParquetIO(FusedIO) - added:
class IO(Expr) - added:
field FromPandasDivisions.sort - added:
method FusedIO.dependencies() - added:
property FromArray.original_columns - added:
property FromArray.unfiltered_divisions - added:
property FromMap.apply_func - added:
property FromMap.apply_kwargs - added:
property FromMapProjectable.apply_kwargs - added:
property FromMapProjectable.columns - added:
property FromMapProjectable.columns_operand - added:
property FromMapProjectable.frame_meta - added:
property FromMapProjectable.kwargs - added:
property FromPandas.columns - added:
property FromPandas.frame - added:
property FromPandas.npartitions - added:
property FusedIO.npartitions
- added:
class FragmentWrapper - added:
class ReadParquet(PartitionsFiltered, BlockwiseIO) - added:
class ReadParquetFSSpec(ReadParquet) - added:
class ReadParquetPyarrowFS(ReadParquet) - added:
class ToParquet(Expr) - added:
class ToParquetBarrier(Expr) - added:
class ToParquetData(Blockwise) - added:
constructor FragmentWrapper(fragment, filesystem, file_size=…, fragment_packed=…) -> None - added:
function normalize_pa_ds(ds) - added:
function normalize_pa_file_format(file_format) - added:
function normalize_pa_schema(schema) - added:
function normalize_pq_filemetadata(meta) - added:
function normalize_pq_schema(schema) - added:
function to_parquet(df, path, compression=…, write_index=…, append=…, overwrite=…, ignore_divisions=…, partition_on=…, storage_options=…, custom_metadata=…, write_metadata_file=…, compute=…, compute_kwargs=…, schema=…, name_function=…, filesystem=…, engine=…, **kwargs) - added:
method FragmentWrapper.pack() - added:
method FragmentWrapper.unpack() - added:
method ReadParquetPyarrowFS.all_statistics_known() -> bool - added:
method ReadParquetPyarrowFS.approx_statistics() -> dict - added:
method ReadParquetPyarrowFS.load_statistics(files=…, fragments=…) - added:
method ReadParquetPyarrowFS.sample_statistics(n=…) - added:
property FragmentWrapper.fragment - added:
property ReadParquet.checksum - added:
property ReadParquet.columns - added:
property ReadParquetFSSpec.engine - added:
property ReadParquetPyarrowFS.aggregated_statistics - added:
property ReadParquetPyarrowFS.fragments - added:
property ReadParquetPyarrowFS.fragments_unsorted - added:
property ReadParquetPyarrowFS.fs - added:
property ReadParquetPyarrowFS.normalized_path - added:
property ReadParquetPyarrowFS.raw_statistics - added:
property ToParquetData.io_func - added:
variable NONE_LABEL - added:
variable PYARROW_NULLABLE_DTYPE_MAPPING
- added:
function to_records(df)
- removed:
class DataFrameGroupBy(_GroupBy) - removed:
class SeriesGroupBy(_GroupBy) - removed:
constructor SeriesGroupBy(df, by=…, slice=…, observed=…, **kwargs) - removed:
function numeric_only_not_implemented(func) - removed:
method DataFrameGroupBy.agg(arg=…, split_every=…, split_out=…, shuffle_method=…, **kwargs) - removed:
method DataFrameGroupBy.aggregate(arg=…, split_every=…, split_out=…, shuffle_method=…, **kwargs) - removed:
method SeriesGroupBy.agg(arg=…, split_every=…, split_out=…, shuffle_method=…, **kwargs) - removed:
method SeriesGroupBy.aggregate(arg=…, split_every=…, split_out=…, shuffle_method=…, **kwargs) - removed:
method SeriesGroupBy.head(n=…, split_every=…, split_out=…, shuffle_method=…) - removed:
method SeriesGroupBy.nunique(split_every=…, split_out=…) - removed:
method SeriesGroupBy.tail(n=…, split_every=…, split_out=…, shuffle_method=…) - removed:
method SeriesGroupBy.unique(split_every=…, split_out=…, shuffle_method=…) - removed:
method SeriesGroupBy.value_counts(split_every=…, split_out=…, shuffle_method=…) - added:
variable GROUP_KEYS_DEFAULT: bool | None
- removed:
export from_array - removed:
export from_delayed - removed:
export from_dict - removed:
export from_map - removed:
export from_pandas - removed:
export read_parquet - removed:
export to_backend - removed:
export to_bag - removed:
export to_parquet - removed:
export to_records
- removed:
class CSVFunctionWrapper(DataFrameIOFunction) - removed:
constructor CSVFunctionWrapper(full_columns, columns, colname, head, header, reader, dtypes, enforce, kwargs) - removed:
method CSVFunctionWrapper.project_columns(columns) - removed:
property CSVFunctionWrapper.columns
- removed:
class HDFFunctionWrapper(DataFrameIOFunction) - removed:
constructor HDFFunctionWrapper(columns, dim, lock, common_kwargs) - removed:
method HDFFunctionWrapper.project_columns(columns) - removed:
property HDFFunctionWrapper.columns
- removed:
function from_array(x, chunksize=…, columns=…, meta=…) - removed:
function from_delayed(dfs: Delayed | distributed.Future | Iterable[Delayed | distributed.Future], meta=…, divisions: tuple | Literal['sorted'] | None=…, prefix: str=…, verify_meta: bool=…) -> DataFrame | Series - removed:
function from_dict(data, npartitions, orient=…, dtype=…, columns=…, constructor=…) - removed:
function from_map(func, *iterables, args=…, meta=…, divisions=…, label=…, token=…, enforce_metadata=…, **kwargs) - removed:
function from_pandas(data: pd.DataFrame, npartitions: int | None=…, chunksize: int | None=…, sort: bool=…, name: str | None=…) -> DataFrame | (data: pd.Series, npartitions: int | None=…, chunksize: int | None=…, sort: bool=…, name: str | None=…) -> Series - removed:
function to_backend(ddf: _Frame, backend: str | None=…, **kwargs) - removed:
function to_bag(df, index=…, format=…) - removed:
function to_records(df)
- removed:
class ORCFunctionWrapper(DataFrameIOFunction) - removed:
constructor ORCFunctionWrapper(fs, columns, schema, engine, index) - removed:
method ORCFunctionWrapper.project_columns(columns) - removed:
property ORCFunctionWrapper.columns
- removed:
export read_parquet - removed:
export to_parquet
- removed:
static method ArrowDatasetEngine.read_metadata(fs, paths, categories=…, index=…, use_nullable_dtypes=…, dtype_backend=…, gather_statistics=…, filters=…, split_row_groups=…, blocksize=…, aggregate_files=…, ignore_metadata_file=…, metadata_task_size=…, parquet_file_extension=…, **kwargs)
- removed:
function read_parquet(path, columns=…, filters=…, categories=…, index=…, storage_options=…, engine=…, use_nullable_dtypes: bool | None=…, dtype_backend=…, calculate_divisions=…, ignore_metadata_file=…, metadata_task_size=…, split_row_groups=…, blocksize=…, aggregate_files=…, parquet_file_extension=…, filesystem=…, **kwargs) - removed:
function to_parquet(df, path, engine=…, compression=…, write_index=…, append=…, overwrite=…, ignore_divisions=…, partition_on=…, storage_options=…, custom_metadata=…, write_metadata_file=…, compute=…, compute_kwargs=…, schema=…, name_function=…, filesystem=…, **kwargs) - added:
class ParquetFunctionWrapper(DataFrameIOFunction) - added:
class ToParquetFunctionWrapper - added:
constructor ParquetFunctionWrapper(engine, fs, meta, columns, index, dtype_backend, kwargs, common_kwargs) - added:
constructor ToParquetFunctionWrapper(engine, path, fs, partition_on, write_metadata_file, i_offset, name_function, kwargs_pass) - added:
function aggregate_row_groups(parts, stats, blocksize, split_row_groups, fs, aggregation_depth) - added:
function apply_filters(parts, statistics, filters) - added:
function check_multi_support(engine) - added:
function create_metadata_file(paths, root_dir=…, out_dir=…, engine=…, storage_options=…, split_every=…, compute=…, compute_kwargs=…, fs=…) - added:
function get_engine(engine: Literal['auto', 'pyarrow'] | type[Engine]) -> type[Engine] - added:
function read_parquet_part(fs, engine, meta, part, columns, index, kwargs) - added:
function set_index_columns(meta, index, columns, auto_index_allowed) - added:
function sorted_columns(statistics, columns=…) - added:
method ParquetFunctionWrapper.project_columns(columns) - added:
property ParquetFunctionWrapper.columns - added:
variable NONE_LABEL
- removed:
class FastParquetEngine(Engine) - removed:
static method FastParquetEngine.initialize_write(df, fs, path, append=…, partition_on=…, ignore_divisions=…, division_info=…, schema=…, object_encoding=…, index_cols=…, custom_metadata=…, **kwargs) - removed:
static method FastParquetEngine.multi_support() - removed:
static method FastParquetEngine.pf_to_pandas(pf, fs=…, columns=…, categories=…, index=…, open_file_options=…, **kwargs) - removed:
static method FastParquetEngine.read_metadata(fs, paths, categories=…, index=…, use_nullable_dtypes=…, dtype_backend=…, gather_statistics=…, filters=…, split_row_groups=…, blocksize=…, aggregate_files=…, ignore_metadata_file=…, metadata_task_size=…, parquet_file_extension=…, **kwargs) - removed:
static method FastParquetEngine.read_partition(fs, pieces, columns, index, dtype_backend=…, categories=…, root_cats=…, root_file_scheme=…, base_path=…, **kwargs) - removed:
static method FastParquetEngine.write_metadata(parts, meta, fs, path, append=…, **kwargs) - removed:
static method FastParquetEngine.write_partition(df, path, fs, filename, partition_on, return_metadata, fmd=…, compression=…, custom_metadata=…, **kwargs) - removed:
variable paths_to_cats
- removed:
static method Engine.read_metadata(fs, paths, categories=…, index=…, use_nullable_dtypes=…, dtype_backend=…, gather_statistics=…, filters=…, **kwargs)
- removed:
function applymap(df, *args, **kwargs)
- removed:
function align_partitions(*dfs) - removed:
function broadcast_join(lhs, left_on, rhs, right_on, how=…, npartitions=…, suffixes=…, shuffle_method=…, indicator=…, parts_out=…) - removed:
function compute_heads(ddf, by=…) - removed:
function compute_tails(ddf, by=…) - removed:
function concat(dfs, axis=…, join=…, interleave_partitions=…, ignore_unknown_divisions=…, ignore_order=…, **kwargs) - removed:
function concat_indexed_dataframes(dfs, axis=…, join=…, ignore_order=…, **kwargs) - removed:
function concat_unindexed_dataframes(dfs, ignore_order=…, **kwargs) - removed:
function get_unsorted_columns(frames) - removed:
function hash_join(lhs, left_on, rhs, right_on, how=…, npartitions=…, suffixes=…, shuffle_method=…, indicator=…, max_branch=…) - removed:
function merge(left, right, how=…, on=…, left_on=…, right_on=…, left_index=…, right_index=…, suffixes=…, indicator=…, npartitions=…, shuffle_method=…, max_branch=…, broadcast=…) - removed:
function merge_asof(left, right, on=…, left_on=…, right_on=…, left_index=…, right_index=…, by=…, left_by=…, right_by=…, suffixes=…, tolerance=…, allow_exact_matches=…, direction=…) - removed:
function merge_asof_indexed(left, right, **kwargs) - removed:
function merge_indexed_dataframes(lhs, rhs, left_index=…, right_index=…, **kwargs) - removed:
function most_recent_head(left, right) - removed:
function most_recent_head_summary(left, right, by=…) - removed:
function most_recent_tail(left, right) - removed:
function most_recent_tail_summary(left, right, by=…) - removed:
function require(divisions, parts, required=…) - removed:
function single_partition_join(left, right, **kwargs) - removed:
function stack_partitions(dfs, divisions, join=…, ignore_order=…, **kwargs) - removed:
variable allowed_left - removed:
variable allowed_right - removed:
variable required - removed:
variable shuffle_func
- removed:
function to_numeric(arg, errors=…, meta=…)
- removed:
function optimize(dsk, keys, **kwargs) - removed:
function optimize_dataframe_getitem(dsk, keys)
- removed:
function partition_quantiles(df, npartitions, upsample=…, random_state=…)
- removed:
function get_dummies(data, prefix=…, prefix_sep=…, dummy_na=…, columns=…, sparse=…, drop_first=…, dtype=…, **kwargs) - removed:
function melt(frame, id_vars=…, value_vars=…, var_name=…, value_name=…, col_level=…) - removed:
function pivot_table(df, index=…, columns=…, values=…, aggfunc=…)
- removed:
class Rolling - removed:
class RollingGroupby(Rolling) - removed:
constructor Rolling(obj, window=…, min_periods=…, center=…, win_type=…, axis=…) - removed:
constructor RollingGroupby(groupby, window=…, min_periods=…, center=…, win_type=…, axis=…) - removed:
field Rolling.agg - removed:
function map_overlap(func, df, before, after, *args, meta=…, enforce_metadata=…, transform_divisions=…, align_dataframes=…, **kwargs) - removed:
method Rolling.aggregate(func, *args, **kwargs) - removed:
method Rolling.apply(func, raw=…, engine=…, engine_kwargs=…, args=…, kwargs=…) - removed:
method Rolling.count() - removed:
method Rolling.cov() - removed:
method Rolling.kurt() - removed:
method Rolling.max() - removed:
method Rolling.mean() - removed:
method Rolling.median() - removed:
method Rolling.min() - removed:
method Rolling.quantile(quantile) - removed:
method Rolling.skew() - removed:
method Rolling.std(ddof=…) - removed:
method Rolling.sum() - removed:
method Rolling.var(ddof=…) - removed:
static method Rolling.pandas_rolling_method(df, rolling_kwargs, name, *args, **kwargs) - removed:
static method RollingGroupby.pandas_rolling_method(df, rolling_kwargs, name, *args, groupby_kwargs=…, groupby_slice=…, **kwargs)
- removed:
function cleanup_partd_files(p, keys) - removed:
function compute_and_set_divisions(df: DataFrame, **kwargs) -> DataFrame - removed:
function compute_divisions(df: DataFrame, col: Any | None=…, **kwargs) -> tuple - removed:
function fix_overlap(ddf, mins, maxes, lens) - removed:
function rearrange_by_column(df, col, npartitions=…, max_branch=…, shuffle_method=…, compute=…, ignore_index=…) - removed:
function rearrange_by_column_disk(df, column, npartitions=…, compute=…) - removed:
function rearrange_by_column_tasks(df, column, max_branch=…, npartitions=…, ignore_index=…) - removed:
function rearrange_by_divisions(df, column, divisions, max_branch=…, shuffle_method=…, ascending=…, na_position=…, duplicates=…) - removed:
function set_index(df: DataFrame, index: str | Series, npartitions: int | Literal['auto'] | None=…, shuffle_method: str | None=…, compute: bool=…, drop: bool=…, upsample: float=…, divisions: Sequence | None=…, partition_size: float=…, sort: bool=…, **kwargs) -> DataFrame - removed:
function set_index_post_scalar(df, index_name, drop, column_dtype) - removed:
function set_index_post_series(df, index_name, drop, column_dtype) - removed:
function set_partition(df: DataFrame, index: str | Series, divisions: Sequence, max_branch: int=…, drop: bool=…, shuffle_method: str | None=…, compute: bool | None=…) -> DataFrame - removed:
function set_sorted_index(df: DataFrame, index: str | Series, drop: bool=…, divisions: Sequence | None=…, **kwargs) -> DataFrame - removed:
function shuffle(df, index, shuffle_method=…, npartitions=…, max_branch=…, ignore_index=…, compute=…) - removed:
function shuffle_group_3(df, col, npartitions, p) - removed:
function sort_values(df: DataFrame, by: str | list[str], npartitions: int | Literal['auto'] | None=…, shuffle_method: str | None=…, ascending: bool | list[bool]=…, na_position: Literal['first'] | Literal['last']=…, upsample: float=…, partition_size: float=…, sort_function: Callable[[pd.DataFrame], pd.DataFrame] | None=…, sort_function_kwargs: Mapping[str, Any] | None=…) -> DataFrame
- changed:
method Resampler.agg(agg_funcs, *args, **kwargs) → method Resampler.agg(func, *args, **kwargs) - added:
class ResampleAgg(ResampleReduction) - added:
class ResampleAggregation(Blockwise) - added:
class ResampleCount(ResampleReduction) - added:
class ResampleFirst(ResampleReduction) - added:
class ResampleLast(ResampleReduction) - added:
class ResampleMax(ResampleReduction) - added:
class ResampleMean(ResampleReduction) - added:
class ResampleMedian(ResampleReduction) - added:
class ResampleMin(ResampleReduction) - added:
class ResampleNUnique(ResampleReduction) - added:
class ResampleOhlc(ResampleReduction) - added:
class ResampleProd(ResampleReduction) - added:
class ResampleQuantile(ResampleReduction) - added:
class ResampleReduction(Expr) - added:
class ResampleSem(ResampleReduction) - added:
class ResampleSize(ResampleReduction) - added:
class ResampleStd(ResampleReduction) - added:
class ResampleSum(ResampleReduction) - added:
class ResampleVar(ResampleReduction) - added:
field ResampleAgg.how - added:
field ResampleAggregation.operation - added:
field ResampleCount.fill_value - added:
field ResampleCount.how - added:
field ResampleFirst.how - added:
field ResampleLast.how - added:
field ResampleMax.how - added:
field ResampleMean.how - added:
field ResampleMedian.how - added:
field ResampleMin.how - added:
field ResampleNUnique.fill_value - added:
field ResampleNUnique.how - added:
field ResampleOhlc.how - added:
field ResampleProd.how - added:
field ResampleQuantile.how - added:
field ResampleReduction.fill_value - added:
field ResampleReduction.how: str | None - added:
field ResampleSem.how - added:
field ResampleSize.fill_value - added:
field ResampleSize.how - added:
field ResampleStd.how - added:
field ResampleSum.fill_value - added:
field ResampleSum.how - added:
field ResampleVar.how - added:
property ResampleReduction.how_kwargs - added:
property ResampleReduction.kwargs - added:
property ResampleReduction.npartitions - added:
variable BlockwiseDep
- removed:
function check_numeric_only_valid(numeric_only: bool | NoDefault, name: str) -> dict - removed:
function shard_df_on_index(df, divisions) - added:
function is_scalar(x)
- added:
property Layer.has_legacy_tasks
- removed:
class BroadcastJoinLayer(Layer) - removed:
class DataFrameTreeReduction(Layer) - removed:
class ShuffleLayer(SimpleShuffleLayer) - removed:
class SimpleShuffleLayer(Layer) - removed:
constructor BroadcastJoinLayer(name, npartitions, lhs_name, lhs_npartitions, rhs_name, rhs_npartitions, parts_out=…, annotations=…, left_on=…, right_on=…, **merge_kwargs) - removed:
constructor DataFrameTreeReduction(name: str, name_input: str, npartitions_input: int, concat_func: Callable, tree_node_func: Callable, finalize_func: Callable | None=…, split_every: int=…, split_out: int | None=…, output_partitions: list[int] | None=…, tree_node_name: str | None=…, annotations: dict[str, Any] | None=…) - removed:
constructor ShuffleLayer(name, column, inputs, stage, npartitions, npartitions_input, nsplits, ignore_index, name_input, meta_input, parts_out=…, annotations=…) - removed:
constructor SimpleShuffleLayer(name, column, npartitions, npartitions_input, ignore_index, name_input, meta_input, parts_out=…, annotations=…) - removed:
field DataFrameTreeReduction.concat_func: Callable - removed:
field DataFrameTreeReduction.finalize_func: Callable | None - removed:
field DataFrameTreeReduction.height: int - removed:
field DataFrameTreeReduction.name: str - removed:
field DataFrameTreeReduction.name_input: str - removed:
field DataFrameTreeReduction.npartitions_input: int - removed:
field DataFrameTreeReduction.output_partitions: list[int] - removed:
field DataFrameTreeReduction.split_every: int - removed:
field DataFrameTreeReduction.split_out: int - removed:
field DataFrameTreeReduction.tree_node_func: Callable - removed:
field DataFrameTreeReduction.tree_node_name: str - removed:
field DataFrameTreeReduction.widths: list[int] - removed:
function reshapelist(shape, seq) - removed:
method BroadcastJoinLayer.cull(keys, all_keys) - removed:
method BroadcastJoinLayer.get_output_keys() - removed:
method BroadcastJoinLayer.is_materialized() - removed:
method DataFrameTreeReduction.cull(keys, all_keys) - removed:
method DataFrameTreeReduction.get_output_keys() - removed:
method DataFrameTreeReduction.is_materialized() - removed:
method SimpleShuffleLayer.cull(keys, all_keys) - removed:
method SimpleShuffleLayer.get_output_keys() - removed:
method SimpleShuffleLayer.is_materialized()
- changed:
function start_state_from_dask(dsk, cache=…, sortkey=…) → function start_state_from_dask(dsk, cache=…, sortkey=…, keys=…)
- removed:
class SubgraphCallable - removed:
constructor SubgraphCallable(dsk: Graph, outkey: Key, inkeys: Iterable[Key], name: str | None=…) - removed:
field SubgraphCallable.dsk: Graph - removed:
field SubgraphCallable.inkeys: tuple[Key, ...] - removed:
field SubgraphCallable.name: str - removed:
field SubgraphCallable.outkey: Key - changed:
function fuse(dsk, keys=…, dependencies=…, ave_width=…, max_width=…, max_height=…, max_depth_new_edges=…, rename_keys=…, fuse_subgraphs=…) → function fuse(dsk, keys=…, dependencies=…, ave_width=…, max_width=…, max_height=…, max_depth_new_edges=…, rename_keys=…)
- added:
export Alias - added:
export DataNode - added:
export GraphNode - added:
export List - added:
export Task - added:
export TaskRef
- added:
class disable_gc(ContextDecorator) - added:
constructor disable_gc(collect=…) - added:
function is_empty(obj)