Upgrading pyarrow 0.10 to 0.11
Breaking0.10.0 → 0.11.1Breaking9Platform3Deprecated1New158Changed151Fixed202Other117
BreakingUpgrade verdict
9 breaking changes and 3 platform requirement changes.
In plain terms
- Drops older APIs that were deprecated in version 0.10, so code using them must be updated.
- GLib users must update code that used the old "writeable" spelling.
- C++ code that used the parquet_cpp namespace must now use parquet.
- Removes the RandomAccessSource::Seek method, which could cause thread safety problems.
Must act Top 5 of 13 that need code or config changes
Adds numpy >=1.14, six >=1.0.0.
Installs 2 packages in all (was 0).
What you must change
Breaking9
Removes API deprecations from 0.10 in C++ and Python.
Renames "writeable" to "writable" in the GLib bindings.
Renames the public namespace to parquet from parquet_cpp.
parquetparquet_cppRemoves the RandomAccessSource::Seek method, which could cause thread safety problems.
RandomAccessSource::SeekRemoves the namespace arrow::test from the C++ codebase.
arrow::testChanges column reader methods to be array-oriented rather than scalar.
Decouples serialized file internals from the ParquetFileReader public API.
ParquetFileReaderUpdates the C++ TypedColumnReader::ReadBatch method signature.
TypedColumnReader::ReadBatchRequires bytes-like input to NativeFile.write.
NativeFile.writePlatform3
Updates conda packages to the new numpy requirement.
numpyFixes pyarrow 0.10 requiring a newer numpy version than its requirements specified.
Raises the global numpy requirement to 1.14.
numpyDeprecations to plan for
Deprecated1
Renames Writeable references to Writable, adds backwards compatibility and deprecations.
WriteableWritableAPI changes
Generated by DepDelta from published Python source. Not mixed with prose notes.
- removed:
function parse_version(root) - removed:
variable frombuffer - added:
function parse_git(root, **kwargs)
- changed:
function read_feather(source, columns=…, nthreads=…, use_threads=…) → function read_feather(source, columns=…, use_threads=…) - changed:
method FeatherDataset.read_pandas(columns=…, nthreads=…, use_threads=…) → method FeatherDataset.read_pandas(columns=…, use_threads=…) - changed:
method FeatherReader.read_pandas(columns=…, nthreads=…, use_threads=…) → method FeatherReader.read_pandas(columns=…, use_threads=…)
- changed:
method DaskFileSystem.walk(top_path) → method DaskFileSystem.walk(path) - changed:
method FileSystem.read_parquet(path, columns=…, metadata=…, schema=…, nthreads=…, use_pandas_metadata=…) → method FileSystem.read_parquet(path, columns=…, metadata=…, schema=…, use_threads=…, nthreads=…, use_pandas_metadata=…) - changed:
method LocalFileSystem.walk(top_dir) → method LocalFileSystem.walk(path)
- changed:
function deserialize_pandas(buf, nthreads=…, use_threads=…) → function deserialize_pandas(buf, use_threads=…)
- changed:
function dataframe_to_arrays(df, schema, preserve_index, nthreads=…, columns=…) → function dataframe_to_arrays(df, schema, preserve_index, nthreads=…, columns=…, safe=…)
- removed:
function is_path(x) - changed:
function read_pandas(source, columns=…, nthreads=…, metadata=…) → function read_pandas(source, columns=…, use_threads=…, nthreads=…, metadata=…) - changed:
function read_table(source, columns=…, nthreads=…, metadata=…, use_pandas_metadata=…) → function read_table(source, columns=…, use_threads=…, metadata=…, use_pandas_metadata=…, nthreads=…) - changed:
function write_table(table, where, row_group_size=…, version=…, use_dictionary=…, compression=…, use_deprecated_int96_timestamps=…, coerce_timestamps=…, flavor=…, **kwargs) → function write_table(table, where, row_group_size=…, version=…, use_dictionary=…, compression=…, use_deprecated_int96_timestamps=…, coerce_timestamps=…, allow_truncated_timestamps=…, flavor=…, **kwargs) - changed:
method ParquetDataset.read(columns=…, nthreads=…, use_pandas_metadata=…) → method ParquetDataset.read(columns=…, use_threads=…, use_pandas_metadata=…) - changed:
method ParquetDatasetPiece.read(columns=…, nthreads=…, partitions=…, open_file_func=…, file=…, use_pandas_metadata=…) → method ParquetDatasetPiece.read(columns=…, use_threads=…, partitions=…, open_file_func=…, file=…, use_pandas_metadata=…) - changed:
method ParquetFile.read(columns=…, nthreads=…, use_pandas_metadata=…) → method ParquetFile.read(columns=…, use_threads=…, use_pandas_metadata=…) - changed:
method ParquetFile.read_row_group(i, columns=…, nthreads=…, use_pandas_metadata=…) → method ParquetFile.read_row_group(i, columns=…, nthreads=…, use_threads=…, use_pandas_metadata=…)