Upgrading pyarrow 0.14 to 0.15
Breaking0.14.1 → 0.15.1Breaking11Security2Deprecated1New201Changed119Fixed305Other133
SecurityIt was discovered that the C++ implementation (which underlies the R, Python and Ruby implementations) of Apache Arrow 0.14.0 to 0.14.1 had a uninitialized memory bug when building arrays with null values in some cases. This can lead to uninitialized memory being unintentionally shared if Arrow Arrays are transmitted over the wire (for instance with Flight) or persisted in the streaming IPC and file formats.GHSA-8cw2-jv5c-c825 · CVE-2019-12408 ↗
BreakingUpgrade verdict
11 breaking changes and 2 security fixes.
In plain terms
- Removes the arrow::Column class across C++, Python, GLib, Ruby, MATLAB and R.
- Gandiva expressions using 'div' must switch to 'divide'.
- DataFusion no longer depends on the serde crate.
- Python code that still uses APIs deprecated in 0.14.x or before will stop working.
Must act Top 5 of 14 that need code or config changes
What you must change
Breaking11
Removes the arrow::Column class across C++, Python, GLib, Ruby, MATLAB and R.
arrow::ColumnRemoves the 'div' alias for 'divide' in C++ Gandiva.
divdivideRemoves the serde crate dependency from Rust DataFusion.
serdeRemoves Python APIs deprecated in 0.14.x and earlier.
[C++] Renames the Argsort kernel to SortToIndices.
ArgsortSortToIndicesRemoves the useless Java class ByteArrayWrapper.
ByteArrayWrapperRemoves Arrow::Struct in the Ruby bindings and uses Hash instead.
Arrow::StructHashRemoves the ARROW_NO_DEFAULT_MEMORY_POOL macro in C++.
ARROW_NO_DEFAULT_MEMORY_POOLRemoves the equals API from Java ValueVector.
ValueVectorRemoves the dependency on RangeEqualsVisitor from the Java ValueVector interface.
ValueVectorRangeEqualsVisitorArray/ChunkedArray.to_pandas now always returns a pandas.Series, with a mechanism to preserve column names as Series.name.
Array.to_pandasChunkedArray.to_pandaspandas.SeriesRecordBatchTableSeries.nameSecurity2
It was discovered that the C++ implementation (which underlies the R, Python and Ruby implementations) of Apache Arrow 0.14.0 to 0.14.1 had a uninitialized memory bug when building arrays with null values in some cases. This can lead to uninitialized memory being unintentionally shared if Arrow Arrays are transmitted over the wire (for instance with Flight) or persisted in the streaming IPC and file formats.
While investigating UBSAN errors in https://github.com/apache/arrow/pull/5365 it was discovered Apache Arrow versions 0.12.0 to 0.14.1, left memory Array data uninitialized when reading RLE null data from parquet. This affected the C++, Python, Ruby and R implementations. The uninitialized memory could potentially be shared if are transmitted over the wire (for instance with Flight) or persisted in the streaming IPC and file formats.
Deprecations to plan for
Deprecated1
Deprecates and renames the Python add_metadata methods.
add_metadataAPI changes
Generated by DepDelta from published Python source. Not mixed with prose notes.
- removed:
export Column - removed:
export PlasmaObjectExists - removed:
export column - removed:
function open_file(source) - removed:
function open_stream(source) - added:
export BufferedInputStream - added:
export BufferedOutputStream - added:
export LargeBinaryArray - added:
export LargeBinaryValue - added:
export LargeListArray - added:
export LargeListType - added:
export LargeListValue - added:
export LargeStringArray - added:
export LargeStringValue - added:
export PyExtensionType - added:
export SparseTensorCOO - added:
export SparseTensorCSR - added:
export jemalloc_set_decay_ms - added:
export large_binary - added:
export large_list - added:
export large_string - added:
export large_utf8 - added:
export record_batch - added:
export register_extension_type - added:
export unregister_extension_type
- changed:
function check_chunked_overflow(col) → function check_chunked_overflow(name, col)
- changed:
constructor RecordBatchFileWriter(sink, schema) → constructor RecordBatchFileWriter(sink, schema, use_legacy_format=…) - changed:
constructor RecordBatchStreamWriter(sink, schema) → constructor RecordBatchStreamWriter(sink, schema, use_legacy_format=…)
- added:
function make_tz_aware(series, tz)
- changed:
constructor ParquetDataset(path_or_paths, filesystem=…, schema=…, metadata=…, split_row_groups=…, validate_schema=…, filters=…, metadata_nthreads=…, memory_map=…) → constructor ParquetDataset(path_or_paths, filesystem=…, schema=…, metadata=…, split_row_groups=…, validate_schema=…, filters=…, metadata_nthreads=…, read_dictionary=…, memory_map=…, buffer_size=…) - changed:
constructor ParquetDatasetPiece(path, open_file_func=…, row_group=…, partition_keys=…) → constructor ParquetDatasetPiece(path, open_file_func=…, file_options=…, row_group=…, partition_keys=…) - changed:
constructor ParquetFile(source, metadata=…, common_metadata=…, memory_map=…) → constructor ParquetFile(source, metadata=…, common_metadata=…, read_dictionary=…, memory_map=…, buffer_size=…) - changed:
constructor ParquetWriter(where, schema, filesystem=…, flavor=…, version=…, use_dictionary=…, compression=…, write_statistics=…, use_deprecated_int96_timestamps=…, **options) → constructor ParquetWriter(where, schema, filesystem=…, flavor=…, version=…, use_dictionary=…, compression=…, write_statistics=…, use_deprecated_int96_timestamps=…, compression_level=…, **options) - changed:
function read_pandas(source, columns=…, use_threads=…, memory_map=…, metadata=…, filters=…) → function read_pandas(source, columns=…, use_threads=…, memory_map=…, metadata=…, filters=…, buffer_size=…) - changed:
function read_table(source, columns=…, use_threads=…, metadata=…, use_pandas_metadata=…, memory_map=…, filesystem=…, filters=…) → function read_table(source, columns=…, use_threads=…, metadata=…, use_pandas_metadata=…, memory_map=…, read_dictionary=…, filesystem=…, filters=…, buffer_size=…) - changed:
function write_table(table, where, row_group_size=…, version=…, use_dictionary=…, compression=…, write_statistics=…, use_deprecated_int96_timestamps=…, coerce_timestamps=…, allow_truncated_timestamps=…, data_page_size=…, flavor=…, filesystem=…, **kwargs) → function write_table(table, where, row_group_size=…, version=…, use_dictionary=…, compression=…, write_statistics=…, use_deprecated_int96_timestamps=…, coerce_timestamps=…, allow_truncated_timestamps=…, data_page_size=…, flavor=…, filesystem=…, compression_level=…, **kwargs) - changed:
function write_to_dataset(table, root_path, partition_cols=…, filesystem=…, preserve_index=…, **kwargs) → function write_to_dataset(table, root_path, partition_cols=…, partition_filename_cb=…, filesystem=…, **kwargs) - changed:
method ParquetDatasetPiece.get_metadata(open_file_func=…) → method ParquetDatasetPiece.get_metadata() - changed:
method ParquetDatasetPiece.read(columns=…, use_threads=…, partitions=…, open_file_func=…, file=…, use_pandas_metadata=…) → method ParquetDatasetPiece.read(columns=…, use_threads=…, partitions=…, file=…, use_pandas_metadata=…) - added:
field ParquetDataset.buffer_size - added:
field ParquetDataset.common_metadata - added:
field ParquetDataset.fs - added:
field ParquetDataset.memory_map - added:
field ParquetDataset.read_dictionary - added:
method ParquetFile.read_row_groups(row_groups, columns=…, use_threads=…, use_pandas_metadata=…)
- added:
function is_large_binary(t) - added:
function is_large_list(t) - added:
function is_large_string(t) - added:
function is_large_unicode(t)