Skip to content

Upgrading pyarrow 3 to 4

Breaking3.0.0 → 4.0.1Breaking10Deprecated2New221Changed134Fixed199Other198

Full comparisonBack to pyarrow
BreakingUpgrade verdict

10 breaking changes.

In plain terms
  • Removes args from FeatherReader$create v2.
  • Deletes Scan() in R.
  • Makes bool no longer implement ArrowNativeType in Rust.
  • Removes the concurrency field from GlobalLimitExec and SortExec in Rust DataFusion.
Must act Top 5 of 12 that need code or config changes

What you must change

Breaking10
Removes args from FeatherReader$create v2.
FeatherReader$create
4.0.0CHANGELOG.mdSummarizedView source
Deletes Scan() in R.
Scan()
4.0.0CHANGELOG.mdSummarizedView source
Makes bool no longer implement ArrowNativeType in Rust.
ArrowNativeType
4.0.0CHANGELOG.mdSummarizedView source
Removes the concurrency field from GlobalLimitExec and SortExec in Rust DataFusion.
GlobalLimitExecSortExec
4.0.0CHANGELOG.mdSummarizedView source
Removes args from FeatherReader$create in R.
FeatherReader$create
4.0.0CHANGELOG.mdSummarizedView source
[Rust] Removes OffsetSize::prefix.
OffsetSize::prefix
4.0.0CHANGELOG.mdSummarizedView source
Drops GNU Autotools support in GLib.
4.0.0CHANGELOG.mdSummarizedView source
Reading null columns as dictionary is no longer possible in the C++ Dataset API.
4.0.0CHANGELOG.mdSummarizedView source
Removes deprecated IO classes in C++.
4.0.0CHANGELOG.mdSummarizedView source
Refuses implicit cast of str to unsigned integer in Python.
4.0.0CHANGELOG.mdSummarizedView source

Deprecations to plan for

Deprecated2
Makes legacy pyarrow.filesystem and pyarrow.serialize warnings more visible by changing DeprecationWarning to FutureWarning.
pyarrow.filesystempyarrow.serialize
4.0.0CHANGELOG.mdSummarizedView source
Deprecates ReadBatchSpaced in the C++ implementation.
ReadBatchSpaced
4.0.0CHANGELOG.mdSummarizedView source

API changes

Generated by DepDelta from published Python source. Not mixed with prose notes.

  • added: export ArrowCancelled
  • added: export DenseUnionType
  • added: export RuntimeInfo
  • added: export SparseUnionType
  • added: export dense_union
  • added: export enable_signal_handlers
  • added: export runtime_info
  • added: export sparse_union
  • added: function match_substring_regex(array, pattern)
  • added: class ORCWriter
  • added: constructor ORCWriter(where)
  • added: function write_table(table, where)
  • added: method ORCWriter.close()
  • added: method ORCWriter.write(table)
  • changed: constructor ParquetWriter(where, schema, filesystem=…, flavor=…, version=…, use_dictionary=…, compression=…, write_statistics=…, use_deprecated_int96_timestamps=…, compression_level=…, use_byte_stream_split=…, writer_engine_version=…, data_page_version=…, **options) → constructor ParquetWriter(where, schema, filesystem=…, flavor=…, version=…, use_dictionary=…, compression=…, write_statistics=…, use_deprecated_int96_timestamps=…, compression_level=…, use_byte_stream_split=…, writer_engine_version=…, data_page_version=…, use_compliant_nested_type=…, **options)
  • changed: function read_pandas(source, columns=…, use_threads=…, memory_map=…, metadata=…, filters=…, buffer_size=…, use_legacy_dataset=…, ignore_prefixes=…) → function read_pandas(source, columns=…, **kwargs)
  • changed: function write_table(table, where, row_group_size=…, version=…, use_dictionary=…, compression=…, write_statistics=…, use_deprecated_int96_timestamps=…, coerce_timestamps=…, allow_truncated_timestamps=…, data_page_size=…, flavor=…, filesystem=…, compression_level=…, use_byte_stream_split=…, data_page_version=…, **kwargs) → function write_table(table, where, row_group_size=…, version=…, use_dictionary=…, compression=…, write_statistics=…, use_deprecated_int96_timestamps=…, coerce_timestamps=…, allow_truncated_timestamps=…, data_page_size=…, flavor=…, filesystem=…, compression_level=…, use_byte_stream_split=…, data_page_version=…, use_compliant_nested_type=…, **kwargs)
  • added: class InvalidVersion(ValueError)
  • added: class LegacyVersion(_BaseVersion)
  • added: class Version(_BaseVersion)
  • added: constructor LegacyVersion(version)
  • added: constructor Version(version)
  • added: function parse(version)
  • added: property LegacyVersion.base_version
  • added: property LegacyVersion.dev
  • added: property LegacyVersion.epoch
  • added: property LegacyVersion.is_devrelease
  • added: property LegacyVersion.is_postrelease
  • added: property LegacyVersion.is_prerelease
  • added: property LegacyVersion.local
  • added: property LegacyVersion.post
  • added: property LegacyVersion.pre
  • added: property LegacyVersion.public
  • added: property LegacyVersion.release
  • added: property Version.base_version
  • added: property Version.dev
  • added: property Version.epoch
  • added: property Version.is_devrelease
  • added: property Version.is_postrelease
  • added: property Version.is_prerelease
  • added: property Version.local
  • added: property Version.major
  • added: property Version.micro
  • added: property Version.minor
  • added: property Version.post
  • added: property Version.pre
  • added: property Version.public
  • added: property Version.release
  • added: variable VERSION_PATTERN