Upgrading dask 2023 to 2024
Breaking2023.12.1 → 2024.12.1Breaking5Platform2Deprecated2New26Changed62Fixed62Other108
BreakingUpgrade verdict
5 breaking changes and 2 platform requirement changes.
In plain terms
- Removes the legacy DataFrame import.
- Removes optimize_slices.
- A low-level internal option for parsing task specifications is no longer available.
- Removes sequence dict classes from Tasks.
Must act Top 5 of 9 that need code or config changes
Python requirement changed in 2024.8.1: >=3.9 → >=3.10.
Drops importlib-metadata.
Installs 8 packages in all (was 10).
What you must change
Breaking5
Removes the legacy DataFrame import.
Removes optimize_slices.
optimize_slicesRemoves the only_refs parsing option for TaskSpec.
TaskSpeconly_refsRemoves sequence dict classes from Tasks.
Removes the deprecated dask.compatibility module.
dask.compatibilityPlatform2
Adds support for Python 3.13.
Drops support for Python 3.9.
Deprecations to plan for
Deprecated2
Deprecates the legacy dask dataframe implementation.
dask.dataframeRemoves persist usage and emits a warning.
persistAPI changes
Generated by DepDelta from published Python source. Not mixed with prose notes.
- added:
export da - added:
export nanpercentile - added:
export nanquantile - added:
export push - added:
export quantile - added:
export reshape_blockwise - added:
export shuffle - added:
function raise_not_implemented_error(attr_name) - added:
variable ARRAY_EXPR_ENABLED: bool | None - added:
variable abs - added:
variable absolute - added:
variable add - added:
variable all - added:
variable allclose - added:
variable angle - added:
variable any - added:
variable append - added:
variable apply_along_axis - added:
variable apply_gufunc - added:
variable apply_over_axes - added:
variable arccos - added:
variable arccosh - added:
variable arcsin - added:
variable arcsinh - added:
variable arctan - added:
variable arctan2 - added:
variable arctanh - added:
variable argmax - added:
variable argmin - added:
variable argtopk - added:
variable argwhere - added:
variable around - added:
variable array - added:
variable as_gufunc - added:
variable asanyarray - added:
variable atleast_1d - added:
variable atleast_2d - added:
variable atleast_3d - added:
variable atop - added:
variable average - added:
variable backends - added:
variable bincount - added:
variable bitwise_and - added:
variable bitwise_not - added:
variable bitwise_or - added:
variable bitwise_xor - added:
variable block - added:
variable blockwise - added:
variable broadcast_arrays - added:
variable broadcast_to - added:
variable cbrt - added:
variable ceil - added:
variable choose - added:
variable clip - added:
variable coarsen - added:
variable compress - added:
variable concatenate - added:
variable conj - added:
variable copysign - added:
variable corrcoef - added:
variable cos - added:
variable cosh - added:
variable count_nonzero - added:
variable cov - added:
variable cumprod - added:
variable cumsum - added:
variable da - added:
variable deg2rad - added:
variable degrees - added:
variable delete - added:
variable diag - added:
variable diagonal - added:
variable diff - added:
variable digitize - added:
variable divide - added:
variable divmod - added:
variable dot - added:
variable dstack - added:
variable ediff1d - added:
variable einsum - added:
variable empty_like - added:
variable equal - added:
variable exp - added:
variable exp2 - added:
variable expand_dims - added:
variable expm1 - added:
variable extract - added:
variable eye - added:
variable fabs - added:
variable fft - added:
variable fix - added:
variable flatnonzero - added:
variable flip - added:
variable fliplr - added:
variable flipud - added:
variable float_power - added:
variable floor - added:
variable floor_divide - added:
variable fmax - added:
variable fmin - added:
variable fmod - added:
variable frexp - added:
variable from_delayed - added:
variable from_npy_stack - added:
variable from_tiledb - added:
variable from_zarr - added:
variable fromfunction - added:
variable frompyfunc - added:
variable full - added:
variable full_like - added:
variable gradient - added:
variable greater - added:
variable greater_equal - added:
variable gufunc - added:
variable histogram - added:
variable histogram2d - added:
variable histogramdd - added:
variable hstack - added:
variable hypot - added:
variable i0 - added:
variable imag - added:
variable indices - added:
variable insert - added:
variable invert - added:
variable isclose - added:
variable iscomplex - added:
variable isfinite - added:
variable isin - added:
variable isinf - added:
variable isnan - added:
variable isneginf - added:
variable isnull - added:
variable isposinf - added:
variable isreal - added:
variable ldexp - added:
variable left_shift - added:
variable less - added:
variable less_equal - added:
variable lib - added:
variable linalg - added:
variable log - added:
variable log10 - added:
variable log1p - added:
variable log2 - added:
variable logaddexp - added:
variable logaddexp2 - added:
variable logical_and - added:
variable logical_not - added:
variable logical_or - added:
variable logical_xor - added:
variable ma - added:
variable map_blocks - added:
variable map_overlap - added:
variable matmul - added:
variable max - added:
variable maximum - added:
variable median - added:
variable meshgrid - added:
variable min - added:
variable minimum - added:
variable mod - added:
variable modf - added:
variable moveaxis - added:
variable multiply - added:
variable nan_to_num - added:
variable nanargmax - added:
variable nanargmin - added:
variable nancumprod - added:
variable nancumsum - added:
variable nanmax - added:
variable nanmedian - added:
variable nanmin - added:
variable nanprod - added:
variable ndim - added:
variable negative - added:
variable nextafter - added:
variable nonzero - added:
variable not_equal - added:
variable notnull - added:
variable ones_like - added:
variable optimize - added:
variable outer - added:
variable overlap - added:
variable pad - added:
variable percentile - added:
variable piecewise - added:
variable positive - added:
variable power - added:
variable ptp - added:
variable rad2deg - added:
variable radians - added:
variable ravel - added:
variable ravel_multi_index - added:
variable real - added:
variable rechunk - added:
variable reciprocal - added:
variable reduction - added:
variable register_chunk_type - added:
variable remainder - added:
variable repeat - added:
variable reshape - added:
variable result_type - added:
variable right_shift - added:
variable rint - added:
variable roll - added:
variable rollaxis - added:
variable rot90 - added:
variable round - added:
variable searchsorted - added:
variable select - added:
variable shape - added:
variable sign - added:
variable signbit - added:
variable sin - added:
variable sinc - added:
variable sinh - added:
variable spacing - added:
variable sqrt - added:
variable square - added:
variable squeeze - added:
variable stack - added:
variable store - added:
variable subtract - added:
variable swapaxes - added:
variable take - added:
variable tan - added:
variable tanh - added:
variable tensordot - added:
variable tile - added:
variable to_hdf5 - added:
variable to_npy_stack - added:
variable to_tiledb - added:
variable to_zarr - added:
variable topk - added:
variable trace - added:
variable transpose - added:
variable tri - added:
variable tril - added:
variable tril_indices - added:
variable tril_indices_from - added:
variable triu - added:
variable triu_indices - added:
variable triu_indices_from - added:
variable true_divide - added:
variable trunc - added:
variable unify_chunks - added:
variable union1d - added:
variable unique - added:
variable unravel_index - added:
variable vdot - added:
variable vstack - added:
variable where - added:
variable zeros_like
- added:
function sparse_take(array, idx, axis=…)
- changed:
function store(sources: Array | Collection[Array], targets: ArrayLike | Delayed | Collection[ArrayLike | Delayed], lock: bool | Lock=…, regions: tuple[slice, ...] | Collection[tuple[slice, ...]] | None=…, compute: bool=…, return_stored: bool=…, **kwargs) → function store(sources: Array | Collection[Array], targets: ArrayLike | Delayed | Collection[ArrayLike | Delayed], lock: bool | Lock=…, regions: tuple[slice, ...] | Collection[tuple[slice, ...]] | None=…, compute: bool=…, return_stored: bool=…, load_stored: bool | None=…, **kwargs) - added:
method Array.shuffle(indexer: list[list[int]], axis: int, chunks: Literal['auto']=…) - added:
variable ARRAY_TEMPLATE
- added:
variable take_lookup
- changed:
function fft_wrap(fft_func, kind=…, dtype=…) → function fft_wrap(fft_func, kind=…, dtype=…, allow_fftpack=…)
- removed:
function percentile(a, q, method=…) - removed:
variable AxisError - removed:
variable ComplexWarning - added:
variable NUMPY_GE_125 - added:
variable NUMPY_GE_200 - added:
variable NUMPY_GE_210
- removed:
function hold_keys(dsk, dependencies) - removed:
function optimize_slices(dsk) - removed:
variable GETNOREMOVE - removed:
variable GETTERS - changed:
function optimize(dsk, keys, fuse_keys=…, fast_functions=…, inline_functions_fast_functions=…, rename_fused_keys=…, **kwargs) → function optimize(dsk, keys, **kwargs)
- changed:
function sliding_window_view(x, window_shape, axis=…) → function sliding_window_view(x, window_shape, axis=…, automatic_rechunk=…) - added:
function push(array, n, axis)
- added:
function nanpercentile(a, q, **kwargs)
- added:
function nanquantile(a, q, axis=…, out=…, overwrite_input=…, method=…, keepdims=…, *, weights=…, interpolation=…) - added:
function quantile(a, q, axis=…, out=…, overwrite_input=…, method=…, keepdims=…, *, weights=…, interpolation=…)
- changed:
function reshape_rechunk(inshape, outshape, inchunks) → function reshape_rechunk(inshape, outshape, inchunks, disallow_dimension_expansion=…) - added:
function reshape_blockwise(x: Array, shape: int | tuple[int, ...], chunks: tuple[tuple[int, ...], ...] | None=…) -> Array
- removed:
function slicing_plan(chunks, index) - changed:
function slice_slices_and_integers(out_name, in_name, blockdims, index) → function slice_slices_and_integers(out_name, in_name, blockdims, index, allow_getitem_optimization=…) - changed:
function slice_with_newaxes(out_name, in_name, blockdims, index, itemsize) → function slice_with_newaxes(out_name, in_name, blockdims, index) - changed:
function slice_wrap_lists(out_name, in_name, blockdims, index, itemsize) → function slice_wrap_lists(out_name, in_name, blockdims, index, allow_getitem_optimization) - changed:
function take(outname, inname, chunks, index, itemsize, axis=…) → function take(outname, inname, chunks, index, axis=…)
- changed:
constructor CreationDispatch(module_name: str, default: str, entrypoint_class: type[BackendEntrypointType], name: str | None=…) → constructor CreationDispatch(module_name: str, default: str, entrypoint_class: type[BackendEntrypointType], name: str | None=…, entrypoint_root: str=…)
- removed:
function inline_singleton_lists(dsk, keys, dependencies=…) - changed:
function optimize(dsk, keys, fuse_keys=…, rename_fused_keys=…, **kwargs) → function optimize(dsk, keys, fuse_keys=…, **kwargs)
- removed:
function tokenize(*args, **kwargs) - removed:
variable normalize_token - added:
export normalize_token - added:
export tokenize
- removed:
field Blockwise.dsk: Graph - removed:
function lol_product(head, values) - removed:
function make_blockwise_graph(func, output, out_indices, *arrind_pairs, numblocks=…, concatenate=…, new_axes=…, output_blocks=…, dims=…, deserializing=…, func_future_args=…, return_key_deps=…, io_deps=…) - changed:
constructor Blockwise(output: str, output_indices: Iterable[str], dsk: Graph, indices: Iterable[tuple[str | BlockwiseDep, Iterable[str] | None]], numblocks: Mapping[str, Sequence[int]], concatenate: bool | None=…, new_axes: Mapping[str, int] | None=…, output_blocks: set[tuple[int, ...]] | None=…, annotations: Mapping[str, Any] | None=…, io_deps: Mapping[str, BlockwiseDep] | None=…) → constructor Blockwise(output: str, output_indices: Iterable[str], task: Task, indices: Iterable[tuple[str | TaskRef | BlockwiseDep, Iterable[str] | None]], numblocks: Mapping[str, Sequence[int]], concatenate: bool | None=…, new_axes: Mapping[str, int] | None=…, output_blocks: set[tuple[int, ...]] | None=…, annotations: Mapping[str, Any] | None=…, io_deps: Mapping[str, BlockwiseDep] | None=…) - changed:
field Blockwise.indices: tuple[tuple[str, tuple[str, ...] | None], ...] → field Blockwise.indices: tuple[tuple[str | TaskRef, tuple[str, ...] | None], ...] - added:
field Blockwise.task: Task
- added:
function config_find(key) - added:
function config_set(key, value, file) - added:
function save_config(key: str, value: Any, config_file: pathlib.Path | None=…) -> tuple[dict, pathlib.Path]
- changed:
constructor set(arg: Mapping | None=…, config: dict=…, lock: threading.Lock=…, **kwargs) → constructor set(arg: Mapping | None=…, config: dict | None=…, lock: threading.Lock=…, **kwargs) - changed:
function collect_yaml(paths: Sequence[str]=…) -> list[dict] → function collect_yaml(paths: Sequence[str], *, return_paths: Literal[False]=…) -> Iterator[dict] | (paths: Sequence[str], *, return_paths: Literal[True]) -> Iterator[tuple[pathlib.Path, dict]] - changed:
function get(key: str, default: Any=…, config: dict=…, override_with: Any=…) -> Any → function get(key: str, default: Any=…, config: dict | None=…, override_with: Any=…) -> Any - changed:
function refresh(config: dict=…, defaults: list[Mapping]=…, **kwargs) -> None → function refresh(config: dict | None=…, defaults: list[Mapping]=…, **kwargs) -> None - added:
function interpret_value(value: str) -> Any - added:
function paths_containing_key(key: str, paths: Sequence[str]=…) -> Iterator[pathlib.Path]
- added:
function pytest_collection_modifyitems(config, items) - added:
function shuffle_method(request) - added:
variable convert_string - added:
variable skip_with_pyarrow_strings - added:
variable xfail_with_pyarrow_strings
- removed:
function has_tasks(dsk, x) - changed:
function get(dsk, out, cache=…) → function get(dsk: Mapping, out: list | Key, cache: MutableMapping | None=…) -> Any
- removed:
variable Aggregation - removed:
variable from_array - removed:
variable from_dask_array - removed:
variable from_delayed - removed:
variable from_dict - removed:
variable get_dummies - removed:
variable isna - removed:
variable map_partitions - removed:
variable melt - removed:
variable merge_asof - removed:
variable read_fwf - removed:
variable read_hdf - removed:
variable read_json - removed:
variable read_orc - removed:
variable read_sql - removed:
variable read_sql_query - removed:
variable read_sql_table - removed:
variable read_table - removed:
variable to_bag - removed:
variable to_csv - removed:
variable to_hdf - removed:
variable to_json - removed:
variable to_orc - removed:
variable to_records - removed:
variable to_sql - added:
export dd - added:
export map_overlap - added:
export methods - added:
export test_dataframe - added:
variable DASK_EXPR_ENABLED: bool | None - added:
variable dd
- added:
method DatetimeAccessor.isocalendar() - added:
method StringAccessor.endswith(*args, **kwargs) - added:
method StringAccessor.extract(*args, **kwargs) - added:
method StringAccessor.startswith(*args, **kwargs)
- removed:
method DataFrame.append(other, interleave_partitions=…) - removed:
method Series.iteritems() - removed:
property Index.is_monotonic - removed:
property Series.is_monotonic - removed:
variable GROUP_KEYS_DEFAULT - changed:
method DataFrame.clip(lower=…, upper=…, out=…, axis=…) → method DataFrame.clip(lower=…, upper=…, axis=…) - changed:
method DataFrame.join(other, on=…, how=…, lsuffix=…, rsuffix=…, npartitions=…, shuffle=…) → method DataFrame.join(other, on=…, how=…, lsuffix=…, rsuffix=…, npartitions=…, shuffle_method=…) - changed:
method DataFrame.merge(right, how=…, on=…, left_on=…, right_on=…, left_index=…, right_index=…, suffixes=…, indicator=…, npartitions=…, shuffle=…, broadcast=…) → method DataFrame.merge(right, how=…, on=…, left_on=…, right_on=…, left_index=…, right_index=…, suffixes=…, indicator=…, npartitions=…, shuffle_method=…, broadcast=…) - changed:
method Index.drop_duplicates(split_every=…, split_out=…, shuffle=…, **kwargs) → method Index.drop_duplicates(split_every=…, split_out=…, shuffle_method=…, **kwargs) - changed:
method Series.clip(lower=…, upper=…, out=…, axis=…) → method Series.clip(lower=…, upper=…, axis=…)
- removed:
function numeric_only_deprecate_default(func) - removed:
variable SORT_SPLIT_OUT_WARNING - changed:
method DataFrameGroupBy.agg(arg=…, split_every=…, split_out=…, shuffle=…, **kwargs) → method DataFrameGroupBy.agg(arg=…, split_every=…, split_out=…, shuffle_method=…, **kwargs) - changed:
method DataFrameGroupBy.aggregate(arg=…, split_every=…, split_out=…, shuffle=…, **kwargs) → method DataFrameGroupBy.aggregate(arg=…, split_every=…, split_out=…, shuffle_method=…, **kwargs) - changed:
method SeriesGroupBy.agg(arg=…, split_every=…, split_out=…, shuffle=…, **kwargs) → method SeriesGroupBy.agg(arg=…, split_every=…, split_out=…, shuffle_method=…, **kwargs) - changed:
method SeriesGroupBy.aggregate(arg=…, split_every=…, split_out=…, shuffle=…, **kwargs) → method SeriesGroupBy.aggregate(arg=…, split_every=…, split_out=…, shuffle_method=…, **kwargs) - changed:
method SeriesGroupBy.head(n=…, split_every=…, split_out=…, shuffle=…) → method SeriesGroupBy.head(n=…, split_every=…, split_out=…, shuffle_method=…) - changed:
method SeriesGroupBy.tail(n=…, split_every=…, split_out=…, shuffle=…) → method SeriesGroupBy.tail(n=…, split_every=…, split_out=…, shuffle_method=…) - changed:
method SeriesGroupBy.unique(split_every=…, split_out=…, shuffle=…) → method SeriesGroupBy.unique(split_every=…, split_out=…, shuffle_method=…) - changed:
method SeriesGroupBy.value_counts(split_every=…, split_out=…, shuffle=…) → method SeriesGroupBy.value_counts(split_every=…, split_out=…, shuffle_method=…)
- added:
function tokenize_arrowfs(obj)
- changed:
function boundary_slice(df, start, stop, right_boundary=…, left_boundary=…, kind=…) → function boundary_slice(df, start, stop, right_boundary=…, left_boundary=…) - changed:
function describe_numeric_aggregate(stats, name=…, is_timedelta_col=…, is_datetime_col=…) → function describe_numeric_aggregate(stats, name=…, is_timedelta_col=…, is_datetime_col=…, unit=…) - changed:
function value_counts_aggregate(x, sort=…, ascending=…, normalize=…, total_length=…, **groupby_kwargs) → function value_counts_aggregate(x, total_length=…, sort=…, ascending=…, normalize=…, **groupby_kwargs)
- changed:
function broadcast_join(lhs, left_on, rhs, right_on, how=…, npartitions=…, suffixes=…, shuffle=…, indicator=…, parts_out=…) → function broadcast_join(lhs, left_on, rhs, right_on, how=…, npartitions=…, suffixes=…, shuffle_method=…, indicator=…, parts_out=…) - changed:
function hash_join(lhs, left_on, rhs, right_on, how=…, npartitions=…, suffixes=…, shuffle=…, indicator=…, max_branch=…) → function hash_join(lhs, left_on, rhs, right_on, how=…, npartitions=…, suffixes=…, shuffle_method=…, indicator=…, max_branch=…) - changed:
function merge(left, right, how=…, on=…, left_on=…, right_on=…, left_index=…, right_index=…, suffixes=…, indicator=…, npartitions=…, shuffle=…, max_branch=…, broadcast=…) → function merge(left, right, how=…, on=…, left_on=…, right_on=…, left_index=…, right_index=…, suffixes=…, indicator=…, npartitions=…, shuffle_method=…, max_branch=…, broadcast=…)
- changed:
function rearrange_by_column(df, col, npartitions=…, max_branch=…, shuffle=…, compute=…, ignore_index=…) → function rearrange_by_column(df, col, npartitions=…, max_branch=…, shuffle_method=…, compute=…, ignore_index=…) - changed:
function rearrange_by_divisions(df, column, divisions, max_branch=…, shuffle=…, ascending=…, na_position=…, duplicates=…) → function rearrange_by_divisions(df, column, divisions, max_branch=…, shuffle_method=…, ascending=…, na_position=…, duplicates=…) - changed:
function set_index(df: DataFrame, index: str | Series, npartitions: int | Literal['auto'] | None=…, shuffle: str | None=…, compute: bool=…, drop: bool=…, upsample: float=…, divisions: Sequence | None=…, partition_size: float=…, sort: bool=…, **kwargs) -> DataFrame → function set_index(df: DataFrame, index: str | Series, npartitions: int | Literal['auto'] | None=…, shuffle_method: str | None=…, compute: bool=…, drop: bool=…, upsample: float=…, divisions: Sequence | None=…, partition_size: float=…, sort: bool=…, **kwargs) -> DataFrame - changed:
function set_partition(df: DataFrame, index: str | Series, divisions: Sequence, max_branch: int=…, drop: bool=…, shuffle: str | None=…, compute: bool | None=…) -> DataFrame → function set_partition(df: DataFrame, index: str | Series, divisions: Sequence, max_branch: int=…, drop: bool=…, shuffle_method: str | None=…, compute: bool | None=…) -> DataFrame - changed:
function shuffle(df, index, shuffle=…, npartitions=…, max_branch=…, ignore_index=…, compute=…) → function shuffle(df, index, shuffle_method=…, npartitions=…, max_branch=…, ignore_index=…, compute=…) - changed:
function sort_values(df: DataFrame, by: str | list[str], npartitions: int | Literal['auto'] | None=…, shuffle: str | None=…, ascending: bool | list[bool]=…, na_position: Literal['first'] | Literal['last']=…, upsample: float=…, partition_size: float=…, sort_function: Callable[[pd.DataFrame], pd.DataFrame] | None=…, sort_function_kwargs: Mapping[str, Any] | None=…) -> DataFrame → function sort_values(df: DataFrame, by: str | list[str], npartitions: int | Literal['auto'] | None=…, shuffle_method: str | None=…, ascending: bool | list[bool]=…, na_position: Literal['first'] | Literal['last']=…, upsample: float=…, partition_size: float=…, sort_function: Callable[[pd.DataFrame], pd.DataFrame] | None=…, sort_function_kwargs: Mapping[str, Any] | None=…) -> DataFrame
- removed:
function pprint_task(task, keys, label_size=…)
- added:
function register_highlevelgraph(hlg)
- added:
class ArrayBlockIdDep(ArrayBlockwiseDep) - added:
class ArrayValuesDep(ArrayBlockwiseDep) - added:
constructor ArrayValuesDep(chunks: tuple[tuple[int, ...], ...], values: np.ndarray | dict) - added:
function normalize_array_block_id_dep(dep) - added:
function normalize_array_slice_dep(dep) - added:
function normalize_array_values_dep(dep)
- changed:
constructor SubgraphCallable(dsk, outkey, inkeys, name=…) → constructor SubgraphCallable(dsk: Graph, outkey: Key, inkeys: Iterable[Key], name: str | None=…) - added:
field SubgraphCallable.dsk: Graph - added:
field SubgraphCallable.inkeys: tuple[Key, ...] - added:
field SubgraphCallable.name: str - added:
field SubgraphCallable.outkey: Key
- added:
function register_xarray()
- added:
class TokenizationError(RuntimeError) - added:
function normalize_bound_method(meth) - added:
function normalize_builtin_function_or_method(func) - added:
function normalize_compose(func) - added:
function normalize_dict(d) - added:
function normalize_literal(lit) - added:
function normalize_object(o) - added:
function normalize_ordered_dict(d) - added:
function normalize_partial(func) - added:
function normalize_seq(seq) - added:
function normalize_set(s) - added:
function register_numba() - added:
function register_numpy() - added:
function register_pandas() - added:
function register_pyarrow() - added:
function tokenize(*args: object, ensure_deterministic: bool | None=…, **kwargs: object) -> str - added:
variable normalize_token - added:
variable tokenize_lock
- removed:
function dependency_depth(dsk) - removed:
function stringify_collection_keys(obj) - added:
function cached_max(seq) - added:
function unzip(ls, nout) - added:
variable F - added:
variable FuncType