Skip to content

Upgrading pyarrow 0.9 to 0.10

Breaking0.9.0.post1 → 0.10.0Breaking11Security1Platform1Deprecated1New91Changed99Fixed159Other122

Full comparisonBack to pyarrow
BreakingUpgrade verdict

11 breaking changes and a platform requirement change and a security fix.

In plain terms
  • Plasma no longer exposes flatbuffers in its public interface, so code using it must change.
  • The package no longer works on Python 3.4; you need a newer Python version.
  • The timestamps_to_ms option has been removed, so code passing it will break.
  • Renames garrow_decimal128_array_get_value to garrow_decimal128_array_format_value in GLib.
Must act Top 5 of 14 that need code or config changes

What you must change

Breaking11
Removes flatbuffers from the Plasma public API.
0.10.0CHANGELOG.mdSummarizedView source
Drops Python 3.4 support.
0.10.0CHANGELOG.mdSummarizedView source
Removes the timestamps_to_ms parameter.
timestamps_to_ms
0.10.0CHANGELOG.mdSummarizedView source
Renames garrow_decimal128_array_get_value to garrow_decimal128_array_format_value in GLib.
garrow_decimal128_array_get_valuegarrow_decimal128_array_format_value
0.10.0CHANGELOG.mdSummarizedView source
Removes ArrayBuilder::Init, cleans up Resize, and removes PoolBuffer from the public API.
ArrayBuilder::InitArrayBuilder::ResizePoolBuffer
0.10.0CHANGELOG.mdSummarizedView source
Removes the "enable_" prefix from GLib Meson options.
0.10.0CHANGELOG.mdSummarizedView source
Removes the PoolBuffer class from the C++ public API and hides implementation details behind factory functions.
PoolBuffer
0.10.0CHANGELOG.mdSummarizedView source
Changes vector getMinorType to use MinorType instead of Types.MinorType.
getMinorTypeMinorTypeTypes.MinorType
0.10.0CHANGELOG.mdSummarizedView source
Makes out arguments last in ReadMessage.
ReadMessage
0.10.0CHANGELOG.mdSummarizedView source
Removes public attributes from Rust Schema and Field and adds accessors.
SchemaField
0.10.0CHANGELOG.mdSummarizedView source
Makes the Plasma Python client release method private.
0.10.0CHANGELOG.mdSummarizedView source
Security1
Handles untrusted inputs.
0.10.0CHANGELOG.mdSummarizedView source
Platform1
Bumps the Cython requirement to 0.27+.
Cython
0.10.0CHANGELOG.mdSummarizedView source

Deprecations to plan for

Deprecated1
Exposes thread pool size setting to Python and deprecates "nthreads" where possible.
nthreads
0.10.0CHANGELOG.mdSummarizedView source

API changes

Generated by DepDelta from published Python source. Not mixed with prose notes.

  • removed: export NAType
  • removed: export TimestampType
  • added: export DictionaryValue
  • added: export HalfFloatValue
  • added: export NULL
  • added: export ProxyMemoryPool
  • added: export StructValue
  • added: export Time32Value
  • added: export Time64Value
  • added: export UnionValue
  • added: export compat
  • changed: function read_feather(source, columns=…, nthreads=…) → function read_feather(source, columns=…, nthreads=…, use_threads=…)
  • changed: method FeatherReader.read(columns=…, nthreads=…) → method FeatherReader.read(*args, **kwargs)
  • added: class FeatherDataset(object)
  • added: constructor FeatherDataset(path_or_paths, validate_schema=…)
  • added: function read_table(source, columns=…)
  • added: method FeatherDataset.read_pandas(columns=…, nthreads=…, use_threads=…)
  • added: method FeatherDataset.read_table(columns=…)
  • added: method FeatherDataset.validate_schemas(piece, table)
  • added: method FeatherReader.read_pandas(columns=…, nthreads=…, use_threads=…)
  • added: method FeatherReader.read_table(columns=…)
  • added: function indent(text, prefix)
  • changed: constructor HadoopFileSystem(host=…, port=…, user=…, kerb_ticket=…, driver=…) → constructor HadoopFileSystem(host=…, port=…, user=…, kerb_ticket=…, driver=…, extra_conf=…)
  • changed: function connect(host=…, port=…, user=…, kerb_ticket=…, driver=…) → function connect(host=…, port=…, user=…, kerb_ticket=…, driver=…, extra_conf=…)
  • changed: function deserialize_pandas(buf, nthreads=…) → function deserialize_pandas(buf, nthreads=…, use_threads=…)
  • added: function array(jvm_array)
  • added: function field(jvm_field)
  • added: function jvm_buffer(arrowbuf)
  • changed: function dataframe_to_arrays(df, schema, preserve_index, nthreads=…) → function dataframe_to_arrays(df, schema, preserve_index, nthreads=…, columns=…)
  • changed: function table_to_blockmanager(options, table, memory_pool, nthreads=…, categories=…) → function table_to_blockmanager(options, table, memory_pool, categories=…)
  • changed: constructor ParquetDataset(path_or_paths, filesystem=…, schema=…, metadata=…, split_row_groups=…, validate_schema=…) → constructor ParquetDataset(path_or_paths, filesystem=…, schema=…, metadata=…, split_row_groups=…, validate_schema=…, filters=…, metadata_nthreads=…)
  • changed: constructor ParquetManifest(dirpath, filesystem=…, pathsep=…, partition_scheme=…) → constructor ParquetManifest(dirpath, filesystem=…, pathsep=…, partition_scheme=…, metadata_nthreads=…)
  • added: method ParquetPartitions.filter_accepts_partition(part_key, filter, level)
  • added: function build_plasma_tensorflow_op()
  • added: function start_plasma_store(plasma_store_memory, use_valgrind=…, use_profiler=…, use_one_memory_mapped_file=…, plasma_directory=…, use_hugepages=…)
  • added: variable TF_PLASMA_OP_PATH
  • added: variable tf_plasma_op
  • added: function is_primitive(t)