Upgrading dask 2020 to 2021
Unknown2020.12.0 → 2021.12.0Security1
SecurityAn issue was discovered in Dask (aka python-dask) through 2021.09.1. Single machine Dask clusters started with dask.distributed.LocalCluster or dask.distributed.Client (which defaults to using LocalCluster) would mistakenly configure their respective Dask workers to listen on external interfaces (typically with a randomly selected high port) rather than only on localhost. A Dask cluster created using this method (when running on a machine that has an applicable port exposed) could be used by a sophisticated attacker to achieve remote code execution.GHSA-hwqr-f3v9-hwxr · CVE-2021-42343 ↗
UnknownUpgrade verdict
None of the 24 releases in this range publish notes, so breaking changes can't be ruled out.
In plain terms
- Some releases in this range publish no notes, so DepDelta can't rule out breaking changes.
- Their commits are listed with this comparison.
Must act 1 item needs code or config changes
Python requirement changed in 2021.3.1: >=3.6 → >=3.7.
Adds cloudpickle >=1.1.1, fsspec >=0.6.0, packaging >=20.0, partd >=0.3.10, toolz >=0.8.2.
Installs 7 packages in all (was 1).
What you must change
Security1
An issue was discovered in Dask (aka python-dask) through 2021.09.1. Single machine Dask clusters started with dask.distributed.LocalCluster or dask.distributed.Client (which defaults to using LocalCluster) would mistakenly configure their respective Dask workers to listen on external interfaces (typically with a randomly selected high port) rather than only on localhost. A Dask cluster created using this method (when running on a machine that has an applicable port exposed) could be used by a sophisticated attacker to achieve remote code execution.
API changes
Generated by DepDelta from published Python source. Not mixed with prose notes.
- added:
export append - added:
export delete - added:
export histogram2d - added:
export histogramdd - added:
export lib - added:
export rot90 - added:
export searchsorted - added:
export select - added:
export tri - added:
export tril_indices - added:
export tril_indices_from - added:
export triu_indices - added:
export triu_indices_from
- added:
function percentile(a, q, interpolation=…)
- changed:
function arange(start, stop, step, length, dtype) → function arange(start, stop, step, length, dtype, like=…) - added:
function getitem(obj, index) - added:
function linspace(start, stop, num, endpoint=…, dtype=…)
- removed:
field Array.chunks - removed:
function reshapelist(shape, seq) - removed:
variable concatenate_lookup - removed:
variable einsum_lookup - removed:
variable tensordot_lookup - changed:
function asanyarray(a) → function asanyarray(a, dtype=…, order=…, *, like=…) - changed:
function asarray(a, **kwargs) → function asarray(a, allow_unknown_chunksizes=…, dtype=…, order=…, *, like=…, **kwargs) - changed:
function broadcast_arrays(*args, **kwargs) → function broadcast_arrays(*args, subok=…) - changed:
function broadcast_to(x, shape, chunks=…) → function broadcast_to(x, shape, chunks=…, meta=…) - changed:
function elemwise(op, *args, **kwargs) → function elemwise(op, *args, out=…, where=…, dtype=…, name=…, **kwargs) - changed:
function from_array(x, chunks=…, name=…, lock=…, asarray=…, fancy=…, getitem=…, meta=…) → function from_array(x, chunks=…, name=…, lock=…, asarray=…, fancy=…, getitem=…, meta=…, inline_array=…) - changed:
function from_zarr(url, component=…, storage_options=…, chunks=…, name=…, **kwargs) → function from_zarr(url, component=…, storage_options=…, chunks=…, name=…, inline_array=…, **kwargs) - changed:
function insert_to_ooc(arr, out, lock=…, region=…, return_stored=…, load_stored=…, tok=…) → function insert_to_ooc(keys: list, chunks: tuple[tuple[int, ...], ...], out, name: str, *, lock: Lock | bool=…, region: slice | None=…, return_stored: bool=…, load_stored: bool=…) -> dict - changed:
function retrieve_from_ooc(keys, dsk_pre, dsk_post=…) → function retrieve_from_ooc(keys: Collection[Hashable], dsk_pre: Mapping, dsk_post: Mapping) -> dict - changed:
function store(sources, targets, lock=…, regions=…, compute=…, return_stored=…, **kwargs) → function store(sources: Array | Collection[Array], targets, lock: bool | Lock=…, regions: tuple[slice, ...] | Collection[tuple[slice, ...]] | None=…, compute: bool=…, return_stored: bool=…, **kwargs) - changed:
function to_hdf5(filename, *args, **kwargs) → function to_hdf5(filename, *args, chunks=…, **kwargs) - changed:
method Array.reshape(*shape) → method Array.reshape(*shape, merge_chunks=…) - added:
class BlockView - added:
constructor BlockView(array: Array) -> BlockView - added:
method BlockView.ravel() -> list[Array] - added:
property Array.chunks - added:
property BlockView.shape -> tuple[int, ...] - added:
property BlockView.size -> int
- removed:
function tril(m, k=…) - removed:
function triu(m, k=…) - changed:
function arange(*args, **kwargs) → function arange(*args, chunks=…, like=…, dtype=…, **kwargs) - changed:
function meshgrid(*xi, **kwargs) → function meshgrid(*xi, sparse=…, indexing=…, **kwargs) - added:
function tri(N, M=…, k=…, dtype=…, chunks=…, *, like=…)
- added:
variable concatenate_lookup - added:
variable divide_lookup - added:
variable einsum_lookup - added:
variable empty_lookup - added:
variable percentile_lookup - added:
variable tensordot_lookup
- changed:
function einsum(*operands, **kwargs) → function einsum(*operands, dtype=…, optimize=…, split_every=…, **kwargs)
- changed:
class gufunc(object) → class gufunc - changed:
constructor gufunc(pyfunc, **kwargs) → constructor gufunc(pyfunc, *, signature=…, vectorize=…, axes=…, axis=…, keepdims=…, output_sizes=…, output_dtypes=…, allow_rechunk=…, meta=…) - changed:
function apply_gufunc(func, signature, *args, **kwargs) → function apply_gufunc(func, signature, *args, axes=…, axis=…, keepdims=…, output_dtypes=…, output_sizes=…, vectorize=…, allow_rechunk=…, meta=…, **kwargs)
- added:
export stride_tricks
- added:
function sliding_window_view(x, window_shape, axis=…, *, subok=…, writeable=…) - added:
variable sliding_window_view
- removed:
function expand_key(k, dims, name=…, axes=…) - removed:
function fractional_slice(task, axes) - added:
function sliding_window_view(x, window_shape, axis=…)
- changed:
function merge_percentiles(finalq, qs, vals, interpolation=…, Ns=…) → function merge_percentiles(finalq, qs, vals, interpolation=…, Ns=…, raise_on_nan=…)
- changed:
class RandomState(object) → class RandomState
- removed:
variable divide_lookup - removed:
variable empty_lookup - added:
function nancumprod(x, axis, dtype=…, out=…, *, method=…) - added:
function nancumsum(x, axis, dtype=…, out=…, *, method=…) - added:
function nanprod(a, axis=…, dtype=…, keepdims=…, split_every=…, out=…)
- changed:
function reshape(x, shape) → function reshape(x, shape, merge_chunks=…)
- removed:
variable ALPHABET - removed:
variable alphabet - changed:
function aligned_coarsen_chunks(chunks: List[int], multiple: int) -> List[int] → function aligned_coarsen_chunks(chunks: List[int], multiple: int) -> Tuple[int] - changed:
function array(x, dtype=…, ndmin=…) → function array(x, dtype=…, ndmin=…, *, like=…) - changed:
function bincount(x, weights=…, minlength=…) → function bincount(x, weights=…, minlength=…, split_every=…) - changed:
function diff(a, n=…, axis=…) → function diff(a, n=…, axis=…, prepend=…, append=…) - changed:
function flip(m, axis) → function flip(m, axis=…) - changed:
function gradient(f, *varargs, **kwargs) → function gradient(f, *varargs, axis=…, **kwargs) - changed:
function ravel(array) → function ravel(array_like) - added:
function append(arr, values, axis=…) - added:
function delete(arr, obj, axis) - added:
function histogram2d(x, y, bins=…, range=…, normed=…, weights=…, density=…) - added:
function histogramdd(sample, bins, range=…, normed=…, weights=…, density=…) - added:
function rot90(m, k=…, axes=…) - added:
function searchsorted(a, v, side=…, sorter=…) - added:
function select(condlist, choicelist, default=…) - added:
function tril(m, k=…) - added:
function tril_indices(n, k=…, m=…, chunks=…) - added:
function tril_indices_from(arr, k=…) - added:
function triu(m, k=…) - added:
function triu_indices(n, k=…, m=…, chunks=…) - added:
function triu_indices_from(arr, k=…) - added:
function unique_no_structured_arr(ar, return_index=…, return_inverse=…, return_counts=…)
- changed:
function check_index(ind, dimension) → function check_index(axis, ind, dimension) - added:
function concatenate_array_chunks(x) - added:
function parse_assignment_indices(indices, shape) - added:
function setitem(x, v, indices) - added:
function setitem_array(out_name, array, indices, value)
- changed:
function to_tiledb(darray, uri, compute=…, return_stored=…, storage_options=…, **kwargs) → function to_tiledb(darray, uri, compute=…, return_stored=…, storage_options=…, key=…, **kwargs)
- changed:
class da_frompyfunc(object) → class da_frompyfunc - changed:
class ufunc(object) → class ufunc
- removed:
function empty_like_safe(a, shape, **kwargs) - removed:
function full_like_safe(a, fill_value, shape, **kwargs) - removed:
function ones_like_safe(a, shape, **kwargs) - removed:
function zeros_like_safe(a, shape, **kwargs) - removed:
variable AxisError - removed:
variable IS_NEP18_ACTIVE - changed:
function assert_eq(a, b, check_shape=…, check_graph=…, check_meta=…, **kwargs) → function assert_eq(a, b, check_shape=…, check_graph=…, check_meta=…, check_chunks=…, check_type=…, **kwargs) - added:
function arange_safe(*args, like, **kwargs) - added:
function array_safe(a, like, **kwargs) - added:
function asanyarray_safe(a, like, **kwargs) - added:
function asarray_safe(a, like, **kwargs) - added:
function scipy_linalg_safe(func_name, *args, **kwargs) - added:
function solve_triangular_safe(a, b, lower=…)
- removed:
function wrap_func_like_safe(func, func_like, *args, **kwargs) - changed:
function wrap(wrap_func, func, **kwargs) → function wrap(wrap_func, func, func_like=…, **kwargs)
- changed:
class StringAccessor(object) → class StringAccessor
- changed:
function decode(block, encoding, errors) → function decode(block, encoding, errors, line_delimiter) - changed:
function file_to_blocks(include_path, lazy_file) → function file_to_blocks(include_path, lazy_file, delimiter=…)
- changed:
class DaskMethodsMixin(object) → class DaskMethodsMixin - changed:
function compute(*args, **kwargs) → function compute(*args, traverse=…, optimize_graph=…, scheduler=…, get=…, **kwargs) - changed:
function optimize(*args, **kwargs) → function optimize(*args, traverse=…, **kwargs) - changed:
function persist(*args, **kwargs) → function persist(*args, traverse=…, optimize_graph=…, scheduler=…, **kwargs) - changed:
function visualize(*args, **kwargs) → function visualize(*args, filename=…, traverse=…, optimize_graph=…, maxval=…, **kwargs) - added:
function clone_key(key, seed) - added:
function get_collection_names(collection) -> Set[str] - added:
function get_name_from_key(key) -> str - added:
function replace_name_in_key(key, rename: Mapping[str, str])
- removed:
class BlockwiseIO(Blockwise) - removed:
class PackedFunctionCall - removed:
constructor BlockwiseIO(io_name, io_subgraph, output, output_indices, dsk, indices, numblocks, concatenate=…, new_axes=…, output_blocks=…, annotations=…) - removed:
constructor PackedFunctionCall(func) - changed:
constructor Blockwise(output, output_indices, dsk, indices, numblocks, concatenate=…, new_axes=…, output_blocks=…, annotations=…) → constructor Blockwise(output: str, output_indices: Iterable[str], dsk: Mapping[str, tuple], indices: Iterable[Tuple[Union[str, BlockwiseDep], Optional[Iterable[str]]]], numblocks: Mapping[str, Sequence[int]], concatenate: bool=…, new_axes: Mapping[str, int]=…, output_blocks: Set[Tuple[int, ...]]=…, annotations: Mapping[str, Any]=…, io_deps: Optional[Mapping[str, BlockwiseDep]]=…) - changed:
function make_blockwise_graph(func, output, out_indices, *arrind_pairs, **kwargs) → function make_blockwise_graph(func, output, out_indices, *arrind_pairs, numblocks=…, concatenate=…, new_axes=…, output_blocks=…, dims=…, deserializing=…, func_future_args=…, return_key_deps=…, io_deps=…, **kwargs) - changed:
method Blockwise.cull(keys, all_hlg_keys) → method Blockwise.cull(keys: set, all_hlg_keys: Iterable) -> Tuple[Layer, Mapping[Hashable, set]] - added:
class BlockwiseDep - added:
class BlockwiseDepDict(BlockwiseDep) - added:
constructor BlockwiseDepDict(mapping: dict, numblocks: Optional[Tuple[int, ...]]=…, produces_tasks: bool=…) - added:
field Blockwise.concatenate: Optional[bool] - added:
field Blockwise.dsk: Mapping[str, tuple] - added:
field Blockwise.indices: Tuple[Tuple[str, Optional[Tuple[str, ...]]], ...] - added:
field Blockwise.new_axes: Mapping[str, int] - added:
field Blockwise.numblocks: Mapping[str, Sequence[int]] - added:
field Blockwise.output: str - added:
field Blockwise.output_blocks: Optional[Set[Tuple[int, ...]]] - added:
field Blockwise.output_indices: Tuple[str, ...] - added:
field BlockwiseDep.numblocks: Tuple[int, ...] - added:
field BlockwiseDep.produces_tasks: bool - added:
method Blockwise.clone(keys: set, seed: Hashable, bind_to: Hashable=…) -> Tuple[Layer, bool] - added:
method BlockwiseDep.get(idx: Tuple[int, ...], default) -> Any
- removed:
export open_file - removed:
export open_files - removed:
variable fsspec - added:
function open_file(*arg, **kwargs) - added:
function open_files(*arg, **kwargs)
- changed:
class Callback(object) → class Callback - changed:
class add_callbacks(object) → class add_callbacks
- removed:
function dataclass_fields(x) - removed:
function is_dataclass(x) - removed:
variable PY_VERSION - removed:
variable unicode - added:
function prod(iterable, *, start=…)
- changed:
class set(object) → class set - changed:
function get(key, default=…, config=…) → function get(key, default=…, config=…, override_with=…) - added:
function deserialize(data) - added:
function serialize(data)
- changed:
class GlobalMethod(object) → class GlobalMethod
- changed:
class literal(object) → class literal
- added:
export dispatch - added:
export to_orc
- changed:
class Accessor(object) → class Accessor
- added:
class ShuffleGroupResult(SimpleSizeof, dict) - added:
function categorical_dtype_pandas(categories=…, ordered=…) - added:
function concat_pandas(dfs, axis=…, join=…, uniform=…, filter_warning=…, ignore_index=…, **kwargs) - added:
function get_parallel_type_dataframe(_) - added:
function get_parallel_type_frame(o) - added:
function get_parallel_type_index(_) - added:
function get_parallel_type_object(_) - added:
function get_parallel_type_series(_) - added:
function group_split_pandas(df, c, k, ignore_index=…) - added:
function hash_object_pandas(obj, index=…, encoding=…, hash_key=…, categorize=…) - added:
function is_categorical_dtype_pandas(obj) - added:
function make_meta_index(x, index=…) - added:
function make_meta_object(x, index=…) - added:
function make_meta_pandas(x, index=…) - added:
function make_meta_pandas_datetime_tz(x, index=…) - added:
function meta_nonempty_dataframe(x) - added:
function meta_nonempty_object(x) - added:
function percentile(a, q, interpolation=…) - added:
function tolist_pandas(obj) - added:
function union_categoricals_pandas(to_union, sort_categories=…, ignore_order=…) - added:
variable meta_object_types
- removed:
function categorical_dtype(meta, categories=…, ordered=…) - removed:
function categorical_dtype_pandas(categories=…, ordered=…) - removed:
variable categorical_dtype_dispatch
- removed:
function get_parallel_type_dataframe(_) - removed:
function get_parallel_type_frame(o) - removed:
function get_parallel_type_index(_) - removed:
function get_parallel_type_object(o) - removed:
function get_parallel_type_series(_) - removed:
function parallel_types() - removed:
variable Cache - removed:
variable get_parallel_type - changed:
function elemwise(op, *args, **kwargs) → function elemwise(op, *args, meta=…, out=…, transform_divisions=…, **kwargs) - changed:
function map_partitions(func, *args, meta=…, enforce_metadata=…, transform_divisions=…, **kwargs) → function map_partitions(func, *args, meta=…, enforce_metadata=…, transform_divisions=…, align_dataframes=…, **kwargs) - changed:
function new_dd_object(dsk, name, meta, divisions) → function new_dd_object(dsk, name, meta, divisions, parent_meta=…) - changed:
method DataFrame.groupby(by=…, **kwargs) → method DataFrame.groupby(by=…, group_keys=…, sort=…, observed=…, dropna=…, **kwargs) - changed:
method DataFrame.merge(right, how=…, on=…, left_on=…, right_on=…, left_index=…, right_index=…, suffixes=…, indicator=…, npartitions=…, shuffle=…) → method DataFrame.merge(right, how=…, on=…, left_on=…, right_on=…, left_index=…, right_index=…, suffixes=…, indicator=…, npartitions=…, shuffle=…, broadcast=…) - changed:
method DataFrame.to_bag(index=…) → method DataFrame.to_bag(index=…, format=…) - changed:
method Series.groupby(by=…, **kwargs) → method Series.groupby(by=…, group_keys=…, sort=…, observed=…, dropna=…, **kwargs) - changed:
method Series.to_bag(index=…) → method Series.to_bag(index=…, format=…) - changed:
method Series.value_counts(sort=…, ascending=…, dropna=…, split_every=…, split_out=…) → method Series.value_counts(sort=…, ascending=…, dropna=…, normalize=…, split_every=…, split_out=…) - added:
constructor DataFrame(dsk, name, meta, divisions) - added:
method DataFrame.sort_values(by, npartitions=…, ascending=…, na_position=…, sort_function=…, sort_function_kwargs=…, **kwargs) - added:
method DataFrame.to_orc(path, *args, **kwargs) - added:
property DataFrame.axes - added:
property Index.is_monotonic - added:
property Index.is_monotonic_decreasing - added:
property Index.is_monotonic_increasing - added:
property Series.axes - added:
property Series.is_monotonic - added:
property Series.is_monotonic_decreasing - added:
property Series.is_monotonic_increasing
- added:
function categorical_dtype(meta, categories=…, ordered=…) - added:
function concat(dfs, axis=…, join=…, uniform=…, filter_warning=…, ignore_index=…, **kwargs) - added:
function is_categorical_dtype(obj) - added:
function make_meta(x, index=…, parent_meta=…) - added:
function tolist(obj) - added:
function union_categoricals(to_union, sort_categories=…, ignore_order=…) - added:
variable categorical_dtype_dispatch - added:
variable concat_dispatch - added:
variable get_parallel_type - added:
variable group_split_dispatch - added:
variable hash_object_dispatch - added:
variable is_categorical_dtype_dispatch - added:
variable make_meta_dispatch - added:
variable make_meta_obj - added:
variable meta_nonempty - added:
variable tolist_dispatch - added:
variable union_categoricals_dispatch
- changed:
class Aggregation(object) → class Aggregation - added:
method SeriesGroupBy.head(n=…, split_every=…, split_out=…) - added:
method SeriesGroupBy.tail(n=…, split_every=…, split_out=…)
- added:
export read_orc - added:
export to_orc
- removed:
class BlockwiseReadCSV(BlockwiseIO) - removed:
class CSVSubgraph(Mapping) - removed:
constructor BlockwiseReadCSV(name, reader, blocks, is_first, head, header, kwargs, dtypes, columns, enforce, path) - removed:
constructor CSVSubgraph(name, reader, blocks, is_first, head, header, kwargs, dtypes, columns, enforce, path) - changed:
function read_pandas(reader, urlpath, blocksize=…, lineterminator=…, compression=…, sample=…, enforce=…, assume_missing=…, storage_options=…, include_path_column=…, **kwargs) → function read_pandas(reader, urlpath, blocksize=…, lineterminator=…, compression=…, sample=…, sample_rows=…, enforce=…, assume_missing=…, storage_options=…, include_path_column=…, **kwargs) - changed:
function text_blocks_to_pandas(reader, block_lists, header, head, kwargs, enforce=…, specified_dtypes=…, path=…, blocksize=…) → function text_blocks_to_pandas(reader, block_lists, header, head, kwargs, enforce=…, specified_dtypes=…, path=…, blocksize=…, urlpath=…) - added:
class CSVFunctionWrapper - added:
constructor CSVFunctionWrapper(full_columns, columns, colname, head, header, reader, dtypes, enforce, kwargs) - added:
function block_mask_last(block_lists) - added:
method CSVFunctionWrapper.project_columns(columns)
- added:
class HDFFunctionWrapper - added:
constructor HDFFunctionWrapper(columns, dim, lock, common_kwargs) - added:
method HDFFunctionWrapper.project_columns(columns)
- changed:
function to_bag(df, index=…) → function to_bag(df, index=…, format=…)
- changed:
function to_json(df, url_path, orient=…, lines=…, storage_options=…, compute=…, encoding=…, errors=…, compression=…, compute_kwargs=…, **kwargs) → function to_json(df, url_path, orient=…, lines=…, storage_options=…, compute=…, encoding=…, errors=…, compression=…, compute_kwargs=…, name_function=…, **kwargs)
- removed:
function read_orc(path, columns=…, storage_options=…) - added:
export read_orc - added:
export to_orc
- added:
class ArrowORCEngine - added:
static method ArrowORCEngine.read_metadata(fs, paths, columns, index, split_stripes, aggregate_files, **kwargs) - added:
static method ArrowORCEngine.read_partition(fs, parts, schema, columns, **kwargs) - added:
static method ArrowORCEngine.write_partition(df, path, fs, filename, **kwargs)
- added:
class ORCFunctionWrapper - added:
constructor ORCFunctionWrapper(fs, columns, schema, engine, index) - added:
function read_orc(path, engine=…, columns=…, index=…, split_stripes=…, aggregate_files=…, storage_options=…) - added:
function to_orc(df, path, engine=…, write_index=…, storage_options=…, compute=…, compute_kwargs=…) - added:
method ORCFunctionWrapper.project_columns(columns)
- added:
class ORCEngine - added:
static method ORCEngine.read_metadata(fs, paths, columns, index, split_stripes, aggregate_files, **kwargs) - added:
static method ORCEngine.read_partition(fs, part, columns, **kwargs) - added:
static method ORCEngine.write_partition(df, path, fs, filename, **kwargs)
- removed:
static method ArrowLegacyEngine.aggregate_metadata(meta_list, fs, out_path) - removed:
static method ArrowLegacyEngine.collect_file_metadata(path, fs, file_path) - removed:
variable pa_ds - removed:
variable preserve_ind_supported - removed:
variable read_row_groups_supported - removed:
variable schema_field_supported - changed:
static method ArrowDatasetEngine.read_metadata(fs, paths, categories=…, index=…, gather_statistics=…, filters=…, split_row_groups=…, read_from_paths=…, engine=…, **kwargs) → static method ArrowDatasetEngine.read_metadata(fs, paths, categories=…, index=…, gather_statistics=…, filters=…, split_row_groups=…, chunksize=…, aggregate_files=…, ignore_metadata_file=…, metadata_task_size=…, **kwargs) - changed:
static method ArrowDatasetEngine.read_partition(fs, piece, columns, index, categories=…, partitions=…, filters=…, schema=…, **kwargs) → static method ArrowDatasetEngine.read_partition(fs, pieces, columns, index, categories=…, partitions=…, filters=…, schema=…, **kwargs) - changed:
static method ArrowDatasetEngine.write_partition(df, path, fs, filename, partition_on, return_metadata, fmd=…, compression=…, index_cols=…, schema=…, head=…, **kwargs) → static method ArrowDatasetEngine.write_partition(df, path, fs, filename, partition_on, return_metadata, fmd=…, compression=…, index_cols=…, schema=…, head=…, custom_metadata=…, **kwargs) - added:
static method ArrowDatasetEngine.aggregate_metadata(meta_list, fs, out_path) - added:
static method ArrowDatasetEngine.collect_file_metadata(path, fs, file_path) - added:
static method ArrowDatasetEngine.multi_support() - added:
static method ArrowLegacyEngine.multi_support()
- changed:
function read_parquet(path, columns=…, filters=…, categories=…, index=…, storage_options=…, engine=…, gather_statistics=…, split_row_groups=…, read_from_paths=…, chunksize=…, **kwargs) → function read_parquet(path, columns=…, filters=…, categories=…, index=…, storage_options=…, engine=…, gather_statistics=…, ignore_metadata_file=…, metadata_task_size=…, split_row_groups=…, chunksize=…, aggregate_files=…, **kwargs) - changed:
function to_parquet(df, path, engine=…, compression=…, write_index=…, append=…, overwrite=…, ignore_divisions=…, partition_on=…, storage_options=…, write_metadata_file=…, compute=…, compute_kwargs=…, schema=…, **kwargs) → function to_parquet(df, path, engine=…, compression=…, write_index=…, append=…, overwrite=…, ignore_divisions=…, partition_on=…, storage_options=…, custom_metadata=…, write_metadata_file=…, compute=…, compute_kwargs=…, schema=…, name_function=…, **kwargs)
- changed:
static method FastParquetEngine.initialize_write(df, fs, path, append=…, partition_on=…, ignore_divisions=…, division_info=…, schema=…, **kwargs) → static method FastParquetEngine.initialize_write(df, fs, path, append=…, partition_on=…, ignore_divisions=…, division_info=…, schema=…, object_encoding=…, index_cols=…, **kwargs) - changed:
static method FastParquetEngine.read_metadata(fs, paths, categories=…, index=…, gather_statistics=…, filters=…, **kwargs) → static method FastParquetEngine.read_metadata(fs, paths, categories=…, index=…, gather_statistics=…, filters=…, split_row_groups=…, chunksize=…, aggregate_files=…, ignore_metadata_file=…, metadata_task_size=…, **kwargs) - changed:
static method FastParquetEngine.read_partition(fs, piece, columns, index, categories=…, pf=…, **kwargs) → static method FastParquetEngine.read_partition(fs, pieces, columns, index, categories=…, root_cats=…, root_file_scheme=…, base_path=…, **kwargs) - changed:
static method FastParquetEngine.write_partition(df, path, fs, filename, partition_on, return_metadata, fmd=…, compression=…, **kwargs) → static method FastParquetEngine.write_partition(df, path, fs, filename, partition_on, return_metadata, fmd=…, compression=…, custom_metadata=…, **kwargs) - added:
static method FastParquetEngine.multi_support()
- removed:
function concat(dfs, axis=…, join=…, uniform=…, filter_warning=…, ignore_index=…, **kwargs) - removed:
function concat_pandas(dfs, axis=…, join=…, uniform=…, filter_warning=…, ignore_index=…, **kwargs) - removed:
function tolist(obj) - removed:
function tolist_pandas(obj) - removed:
variable concat_dispatch - removed:
variable tolist_dispatch - changed:
function describe_numeric_aggregate(stats, name=…, is_timedelta_col=…) → function describe_numeric_aggregate(stats, name=…, is_timedelta_col=…, is_datetime_col=…) - changed:
function value_counts_aggregate(x, sort=…, ascending=…, **groupby_kwargs) → function value_counts_aggregate(x, sort=…, ascending=…, normalize=…, total_length=…, **groupby_kwargs) - added:
function cumprod_aggregate(x, y) - added:
function cumsum_aggregate(x, y) - added:
function monotonic_decreasing_aggregate(concatenated) - added:
function monotonic_decreasing_chunk(x) - added:
function monotonic_increasing_aggregate(concatenated) - added:
function monotonic_increasing_chunk(x) - added:
function wrap_kurtosis_reduction(array_kurtosis, index)
- changed:
function concat(dfs, axis=…, join=…, interleave_partitions=…, ignore_unknown_divisions=…, **kwargs) → function concat(dfs, axis=…, join=…, interleave_partitions=…, ignore_unknown_divisions=…, ignore_order=…, **kwargs) - changed:
function concat_and_check(dfs) → function concat_and_check(dfs, ignore_order=…) - changed:
function concat_indexed_dataframes(dfs, axis=…, join=…, **kwargs) → function concat_indexed_dataframes(dfs, axis=…, join=…, ignore_order=…, **kwargs) - changed:
function concat_unindexed_dataframes(dfs, **kwargs) → function concat_unindexed_dataframes(dfs, ignore_order=…, **kwargs) - changed:
function hash_join(lhs, left_on, rhs, right_on, how=…, npartitions=…, suffixes=…, shuffle=…, indicator=…) → function hash_join(lhs, left_on, rhs, right_on, how=…, npartitions=…, suffixes=…, shuffle=…, indicator=…, max_branch=…) - changed:
function merge(left, right, how=…, on=…, left_on=…, right_on=…, left_index=…, right_index=…, suffixes=…, indicator=…, npartitions=…, shuffle=…, max_branch=…) → function merge(left, right, how=…, on=…, left_on=…, right_on=…, left_index=…, right_index=…, suffixes=…, indicator=…, npartitions=…, shuffle=…, max_branch=…, broadcast=…) - changed:
function merge_chunk(lhs, *args, **kwargs) → function merge_chunk(lhs, *args, empty_index_dtype=…, categorical_columns=…, **kwargs) - changed:
function stack_partitions(dfs, divisions, join=…, **kwargs) → function stack_partitions(dfs, divisions, join=…, ignore_order=…, **kwargs) - added:
function broadcast_join(lhs, left_on, rhs, right_on, how=…, npartitions=…, suffixes=…, shuffle=…, indicator=…, parts_out=…)
- removed:
function optimize_read_parquet_getitem(dsk, keys) - added:
function optimize_dataframe_getitem(dsk, keys)
- removed:
function pandas_rolling_method(df, rolling_kwargs, name, *args, **kwargs) - changed:
class Rolling(object) → class Rolling - added:
class RollingGroupby(Rolling) - added:
constructor RollingGroupby(groupby, window=…, min_periods=…, center=…, win_type=…, axis=…) - added:
static method Rolling.pandas_rolling_method(df, rolling_kwargs, name, *args, **kwargs) - added:
static method RollingGroupby.pandas_rolling_method(df, rolling_kwargs, name, *args, groupby_kwargs=…, groupby_slice=…, **kwargs)
- removed:
class ShuffleLayer(SimpleShuffleLayer) - removed:
class SimpleShuffleLayer(Layer) - removed:
constructor ShuffleLayer(name, column, inputs, stage, npartitions, npartitions_input, nsplits, ignore_index, name_input, meta_input, parts_out=…, annotations=…) - removed:
constructor SimpleShuffleLayer(name, column, npartitions, npartitions_input, ignore_index, name_input, meta_input, parts_out=…, annotations=…) - removed:
method SimpleShuffleLayer.cull(keys, all_keys) - removed:
method SimpleShuffleLayer.get_output_keys() - removed:
method SimpleShuffleLayer.is_materialized() - changed:
class maybe_buffered_partd(object) → class maybe_buffered_partd - changed:
function rearrange_by_divisions(df, column, divisions, max_branch=…, shuffle=…) → function rearrange_by_divisions(df, column, divisions, max_branch=…, shuffle=…, ascending=…, na_position=…, duplicates=…) - changed:
function set_partitions_pre(s, divisions) → function set_partitions_pre(s, divisions, ascending=…, na_position=…) - added:
function sort_values(df, by, npartitions=…, ascending=…, na_position=…, upsample=…, partition_size=…, sort_function=…, sort_function_kwargs=…, **kwargs)
- changed:
class Resampler(object) → class Resampler
- removed:
function group_split_pandas(df, c, k, ignore_index=…) - removed:
function hash_object_pandas(obj, index=…, encoding=…, hash_key=…, categorize=…) - removed:
function make_meta_index(x, index=…) - removed:
function make_meta_object(x, index=…) - removed:
function make_meta_pandas(x, index=…) - removed:
function meta_nonempty_dataframe(x) - removed:
function meta_nonempty_object(x) - removed:
variable group_split_dispatch - removed:
variable hash_object_dispatch - removed:
variable make_meta - removed:
variable meta_nonempty - changed:
function assert_eq(a, b, check_names=…, check_dtypes=…, check_divisions=…, check_index=…, **kwargs) → function assert_eq(a, b, check_names=…, check_dtype=…, check_divisions=…, check_index=…, **kwargs) - added:
function is_float_na_dtype(t) - added:
variable meta_object_types
- added:
property Delayed.dask
- changed:
function visualize(profilers, file_path=…, show=…, save=…, **kwargs) → function visualize(profilers, filename=…, show=…, save=…, mode=…, **kwargs)
- removed:
variable msg
- added:
function bind(children: T, parents, *, omit=…, seed: Hashable=…, assume_layers: bool=…, split_every: SplitEvery=…) -> T - added:
function checkpoint(*collections, split_every: SplitEvery=…) -> Delayed - added:
function clone(*collections, omit=…, seed: Hashable=…, assume_layers: bool=…) - added:
function wait_on(*collections, split_every: SplitEvery=…)
- removed:
class BasicLayer(Layer) - removed:
constructor BasicLayer(mapping: Mapping, annotations=…) - removed:
method BasicLayer.is_materialized() - removed:
method Layer.pack_annotations() -> Mapping[str, Any] - removed:
static method Layer.expand_annotations(annotations, keys) -> Mapping[str, Any] - changed:
constructor HighLevelGraph(layers: Mapping[str, Layer], dependencies: Mapping[str, Set], key_dependencies: Optional[Mapping[Hashable, Set]]=…) → constructor HighLevelGraph(layers: Mapping[str, Mapping], dependencies: Mapping[str, AbstractSet], key_dependencies: Optional[Dict[Hashable, AbstractSet]]=…) - changed:
constructor Layer(annotations=…) → constructor Layer(annotations: Mapping[str, Any]=…, collection_annotations: Mapping[str, Any]=…) - changed:
method HighLevelGraph.cull(keys: Set) -> HighLevelGraph → method HighLevelGraph.cull(keys: Iterable) -> HighLevelGraph - changed:
method HighLevelGraph.get_all_dependencies() -> Mapping[Hashable, Set] → method HighLevelGraph.get_all_dependencies() -> Dict[Hashable, AbstractSet] - changed:
method HighLevelGraph.get_all_external_keys() -> Set → method HighLevelGraph.get_all_external_keys() -> set - changed:
method HighLevelGraph.keys() → method HighLevelGraph.keys() -> AbstractSet - changed:
method HighLevelGraph.keyset() -> Set → method HighLevelGraph.keyset() -> AbstractSet - changed:
method Layer.cull(keys: Set, all_hlg_keys: Iterable) -> Tuple[Layer, Mapping[Hashable, Set]] → method Layer.cull(keys: set, all_hlg_keys: Iterable) -> Tuple[Layer, Mapping[Hashable, set]] - changed:
method Layer.get_dependencies(key: Hashable, all_hlg_keys: Iterable) -> Set → method Layer.get_dependencies(key: Hashable, all_hlg_keys: Iterable) -> set - changed:
method Layer.get_output_keys() -> Set → method Layer.get_output_keys() -> AbstractSet - added:
class MaterializedLayer(Layer) - added:
constructor MaterializedLayer(mapping: Mapping, annotations=…) - added:
field HighLevelGraph.dependencies: Mapping[str, AbstractSet] - added:
field HighLevelGraph.key_dependencies: Dict[Hashable, AbstractSet] - added:
field HighLevelGraph.layers: Mapping[str, Layer] - added:
field Layer.annotations: Optional[Mapping[str, Any]] - added:
field Layer.collection_annotations: Optional[Mapping[str, Any]] - added:
method HighLevelGraph.cull_layers(layers: Iterable[str]) -> HighLevelGraph - added:
method HighLevelGraph.to_dict() -> dict - added:
method Layer.clone(keys: set, seed: Hashable, bind_to: Hashable=…) -> tuple[Layer, bool] - added:
method Layer.layer_info_dict() - added:
method MaterializedLayer.get_output_keys() - added:
method MaterializedLayer.is_materialized()
- added:
class ArrayOverlapLayer(Layer) - added:
class BlockwiseCreateArray(Blockwise) - added:
class BroadcastJoinLayer(DataFrameLayer) - added:
class CallableLazyImport - added:
class CreateArrayDeps(BlockwiseDep) - added:
class DataFrameIOLayer(Blockwise, DataFrameLayer) - added:
class DataFrameLayer(Layer) - added:
class ShuffleLayer(SimpleShuffleLayer) - added:
class SimpleShuffleLayer(DataFrameLayer) - added:
constructor ArrayOverlapLayer(name, axes, chunks, numblocks, token) - added:
constructor BlockwiseCreateArray(name, func, shape, chunks) - added:
constructor BroadcastJoinLayer(name, npartitions, lhs_name, lhs_npartitions, rhs_name, rhs_npartitions, parts_out=…, annotations=…, **merge_kwargs) - added:
constructor CallableLazyImport(function_path) - added:
constructor CreateArrayDeps(chunks: tuple) - added:
constructor DataFrameIOLayer(name, columns, inputs, io_func, label=…, produces_tasks=…, annotations=…) - added:
constructor ShuffleLayer(name, column, inputs, stage, npartitions, npartitions_input, nsplits, ignore_index, name_input, meta_input, parts_out=…, annotations=…) - added:
constructor SimpleShuffleLayer(name, column, npartitions, npartitions_input, ignore_index, name_input, meta_input, parts_out=…, annotations=…) - added:
function fractional_slice(task, axes) - added:
function reshapelist(shape, seq) - added:
method ArrayOverlapLayer.get_output_keys() - added:
method ArrayOverlapLayer.is_materialized() - added:
method BroadcastJoinLayer.cull(keys, all_keys) - added:
method BroadcastJoinLayer.get_output_keys() - added:
method BroadcastJoinLayer.is_materialized() - added:
method DataFrameIOLayer.project_columns(columns) - added:
method DataFrameLayer.project_columns(output_columns) - added:
method ShuffleLayer.get_split_keys() - added:
method SimpleShuffleLayer.cull(keys, all_keys) - added:
method SimpleShuffleLayer.get_output_keys() - added:
method SimpleShuffleLayer.get_split_keys() - added:
method SimpleShuffleLayer.is_materialized()
- removed:
function apply_sync(func, args=…, kwds=…, callback=…) - removed:
variable DEBUG - changed:
function get_async(apply_async, num_workers, dsk, result, cache=…, get_id=…, rerun_exceptions_locally=…, pack_exception=…, raise_exception=…, callbacks=…, dumps=…, loads=…, **kwargs) → function get_async(submit, num_workers, dsk, result, cache=…, get_id=…, rerun_exceptions_locally=…, pack_exception=…, raise_exception=…, callbacks=…, dumps=…, loads=…, chunksize=…, **kwargs) - added:
class MultiprocessingPoolExecutor(Executor) - added:
class SynchronousExecutor(Executor) - added:
constructor MultiprocessingPoolExecutor(pool) - added:
function batch_execute_tasks(it) - added:
function get_apply_async(apply_async, num_workers, *args, **kwargs) - added:
function submit_apply_async(apply_async, fn, *args, **kwargs) - added:
method MultiprocessingPoolExecutor.submit(fn, *args, **kwargs) - added:
method SynchronousExecutor.submit(fn, *args, **kwargs) - added:
variable synchronous_executor
- changed:
function get(dsk, keys, num_workers=…, func_loads=…, func_dumps=…, optimize_graph=…, pool=…, **kwargs) → function get(dsk, keys, num_workers=…, func_loads=…, func_dumps=…, optimize_graph=…, pool=…, chunksize=…, **kwargs)
- changed:
class SubgraphCallable(object) → class SubgraphCallable
- changed:
class StrComparable(object) → class StrComparable - added:
function diagnostics(dsk, o=…, dependencies=…) - added:
variable OrderInfo
- changed:
class RewriteRule(object) → class RewriteRule - changed:
class RuleSet(object) → class RuleSet - changed:
class Token(object) → class Token - changed:
class Traverser(object) → class Traverser
- added:
class SimpleSizeof - added:
function sizeof_blocked(d)
- changed:
class Dispatch(object) → class Dispatch - changed:
class IndexCallable(object) → class IndexCallable - changed:
class MethodCache(object) → class MethodCache - changed:
class OperatorMethodMixin(object) → class OperatorMethodMixin - changed:
class SerializableLock(object) → class SerializableLock - changed:
class itemgetter(object) → class itemgetter - changed:
class methodcaller(object) → class methodcaller - changed:
function ensure_dict(d) → function ensure_dict(d: Mapping[K, V], *, copy: bool=…) -> dict[K, V] - changed:
function format_bytes(n) → function format_bytes(n: int) -> str - changed:
function stringify(obj, exclusive: Optional[Iterable]=…) → function stringify(obj, exclusive: Iterable | None=…) - changed:
function typename(typ) → function typename(typ, short=…) - added:
class cached_property(_cached_property) - added:
function is_cupy_type(x) - added:
variable K - added:
variable V
- changed:
class GetFunctionTestMixin(object) → class GetFunctionTestMixin - added:
function hlg_layer(hlg: HighLevelGraph, prefix: str) -> Layer - added:
function hlg_layer_topological(hlg: HighLevelGraph, i: int) -> Layer - added:
function import_or_none(name)
- added:
export FILTERS - added:
export TEMPLATE_PATHS - added:
export get_environment - added:
export get_template - added:
function get_environment(*args, **kwargs) - added:
function get_template(*args, **kwargs) - added:
variable FILTERS - added:
variable TEMPLATE_PATHS - added:
variable exception - added:
variable msg
- added:
function get_environment() -> Environment - added:
function get_template(name: str) -> Template - added:
variable FILTERS - added:
variable TEMPLATE_PATHS