Skip to content

Upgrading pyarrow 6 to 7

Breaking6.0.1 → 7.0.0Breaking10Platform3New157Changed91Fixed192Other203

Full comparisonBack to pyarrow
BreakingUpgrade verdict

10 breaking changes and 3 platform requirement changes.

In plain terms
  • Removes columns from the JavaScript implementation.
  • Makes JS Table and RecordBatch no longer extend Vector to shrink the library.
  • Renames the BitUtil and LittleEndianArray namespaces.
  • Removes the deprecated Scanner::Scan method.
Must act Top 5 of 13 that need code or config changes

Python requirement changed in 7.0.0: >=3.6 → >=3.7.

What you must change

Breaking10
Removes columns from the JavaScript implementation.
7.0.0CHANGELOG.mdSummarizedView source
Makes JS Table and RecordBatch no longer extend Vector to shrink the library.
TableRecordBatchVector
7.0.0CHANGELOG.mdSummarizedView source
Renames the BitUtil and LittleEndianArray namespaces.
BitUtilLittleEndianArray
7.0.0CHANGELOG.mdSummarizedView source
Removes the deprecated Scanner::Scan method.
Scanner::Scan
7.0.0CHANGELOG.mdSummarizedView source
Removes Datum::COLLECTION.
Datum::COLLECTION
7.0.0CHANGELOG.mdSummarizedView source
Removes RelBase from the IR.
RelBase
7.0.0CHANGELOG.mdSummarizedView source
Renames GArrowSortKey::name to GArrowSortKey::target in GLib and Ruby.
GArrowSortKey::nameGArrowSortKey::target
7.0.0CHANGELOG.mdSummarizedView source
Migrates Go array types to arrow package types.
array.Interfacearray.Recordarray.Columnarray.Chunkedarray.Tablearrow.Arrayarrow.Recordarrow.Column
7.0.0CHANGELOG.mdSummarizedView source
Disallows comparing zoned and naive timestamps in C++.
7.0.0CHANGELOG.mdSummarizedView source
Removes the schema property from the IR Source node.
Source
7.0.0CHANGELOG.mdSummarizedView source
Platform3
Drops Python 3.6 support.
7.0.0CHANGELOG.mdSummarizedView source
Pins the numpy build dependency using oldest-supported-numpy.
oldest-supported-numpy
7.0.0CHANGELOG.mdSummarizedView source
Increases the minimum NodeJS version to 16 for the JS release.
7.0.0CHANGELOG.mdSummarizedView source

API changes

Generated by DepDelta from published Python source. Not mixed with prose notes.

  • removed: variable FileSystem
  • removed: variable HadoopFileSystem
  • removed: variable LocalFileSystem
  • removed: variable SerializationContext
  • removed: variable SerializedPyObject
  • removed: variable localfs
  • added: export TableGroupBy
  • added: export supported_memory_backends
  • added: function show_info()
  • removed: function count_substring(array, pattern, *, ignore_case=…)
  • removed: function count_substring_regex(array, pattern, *, ignore_case=…)
  • removed: function filter(data, mask, null_selection_behavior=…)
  • removed: function find_substring(array, pattern, *, ignore_case=…)
  • removed: function find_substring_regex(array, pattern, *, ignore_case=…)
  • removed: function match_like(array, pattern, *, ignore_case=…)
  • removed: function match_substring(array, pattern, *, ignore_case=…)
  • removed: function match_substring_regex(array, pattern, *, ignore_case=…)
  • removed: function mode(array, n=…, *, skip_nulls=…, min_count=…)
  • added: function field(name)
  • added: function scalar(value)
  • removed: function field(name)
  • removed: function scalar(value)
  • changed: function write_dataset(data, base_dir, basename_template=…, format=…, partitioning=…, partitioning_flavor=…, schema=…, filesystem=…, file_options=…, use_threads=…, max_partitions=…, file_visitor=…, existing_data_behavior=…) → function write_dataset(data, base_dir, basename_template=…, format=…, partitioning=…, partitioning_flavor=…, schema=…, filesystem=…, file_options=…, use_threads=…, max_partitions=…, max_open_files=…, max_rows_per_file=…, min_rows_per_group=…, max_rows_per_group=…, file_visitor=…, existing_data_behavior=…)
  • changed: function read_table(source, columns=…, memory_map=…) → function read_table(source, columns=…, memory_map=…, use_threads=…)
  • changed: constructor ORCWriter(where) → constructor ORCWriter(where, *, file_version=…, batch_size=…, stripe_size=…, compression=…, compression_block_size=…, compression_strategy=…, row_index_stride=…, padding_tolerance=…, dictionary_key_size_threshold=…, bloom_filter_columns=…, bloom_filter_fpp=…)
  • changed: function write_table(table, where) → function write_table(table, where, *, file_version=…, batch_size=…, stripe_size=…, compression=…, compression_block_size=…, compression_strategy=…, row_index_stride=…, padding_tolerance=…, dictionary_key_size_threshold=…, bloom_filter_columns=…, bloom_filter_fpp=…)
  • added: field ORCWriter.is_open
  • added: function read_table(source, columns=…, filesystem=…)
  • added: property ORCFile.compression
  • added: property ORCFile.compression_size
  • added: property ORCFile.content_length
  • added: property ORCFile.file_footer_length
  • added: property ORCFile.file_length
  • added: property ORCFile.file_postscript_length
  • added: property ORCFile.file_version
  • added: property ORCFile.nstripe_statistics
  • added: property ORCFile.row_index_stride
  • added: property ORCFile.software_version
  • added: property ORCFile.stripe_statistics_length
  • added: property ORCFile.writer
  • added: property ORCFile.writer_version
  • changed: constructor ParquetWriter(where, schema, filesystem=…, flavor=…, version=…, use_dictionary=…, compression=…, write_statistics=…, use_deprecated_int96_timestamps=…, compression_level=…, use_byte_stream_split=…, writer_engine_version=…, data_page_version=…, use_compliant_nested_type=…, **options) → constructor ParquetWriter(where, schema, filesystem=…, flavor=…, version=…, use_dictionary=…, compression=…, write_statistics=…, use_deprecated_int96_timestamps=…, compression_level=…, use_byte_stream_split=…, column_encoding=…, writer_engine_version=…, data_page_version=…, use_compliant_nested_type=…, **options)
  • changed: function write_table(table, where, row_group_size=…, version=…, use_dictionary=…, compression=…, write_statistics=…, use_deprecated_int96_timestamps=…, coerce_timestamps=…, allow_truncated_timestamps=…, data_page_size=…, flavor=…, filesystem=…, compression_level=…, use_byte_stream_split=…, data_page_version=…, use_compliant_nested_type=…, **kwargs) → function write_table(table, where, row_group_size=…, version=…, use_dictionary=…, compression=…, write_statistics=…, use_deprecated_int96_timestamps=…, coerce_timestamps=…, allow_truncated_timestamps=…, data_page_size=…, flavor=…, filesystem=…, compression_level=…, use_byte_stream_split=…, column_encoding=…, data_page_version=…, use_compliant_nested_type=…, **kwargs)
  • added: method ParquetWriter.write(table_or_batch, row_group_size=…)
  • added: method ParquetWriter.write_batch(batch, row_group_size=…)
  • added: class ClassDoc(NumpyDocString)
  • added: class FunctionDoc(NumpyDocString)
  • added: class NumpyDocString(Mapping)
  • added: class ObjDoc(NumpyDocString)
  • added: class ParseError(Exception)
  • added: class Reader
  • added: constructor ClassDoc(cls, doc=…, modulename=…, func_doc=…, config=…)
  • added: constructor FunctionDoc(func, role=…, doc=…, config=…)
  • added: constructor NumpyDocString(docstring, config=…)
  • added: constructor ObjDoc(obj, doc=…, config=…)
  • added: constructor Reader(data)
  • added: field ClassDoc.extra_public_methods
  • added: field NumpyDocString.empty_description
  • added: field NumpyDocString.sections
  • added: function dedent_lines(lines)
  • added: function get_doc_object(obj, what=…, doc=…, config=…)
  • added: function strip_blank_lines(l)
  • added: method FunctionDoc.get_func()
  • added: method Reader.eof()
  • added: method Reader.is_empty()
  • added: method Reader.peek(n=…)
  • added: method Reader.read()
  • added: method Reader.read_to_condition(condition_func)
  • added: method Reader.read_to_next_empty_line()
  • added: method Reader.read_to_next_unindented_line()
  • added: method Reader.reset()
  • added: method Reader.seek_next_non_empty_line()
  • added: property ClassDoc.methods
  • added: property ClassDoc.properties
  • added: variable Parameter