• Earthmover's focus: Building the cloud platform for multidimensional array data (tensors), as there's a need in weather, climate, geospatial, and AI fields.
• Defining "structured data": Typically considered tabular with rows and columns, conforming to a pre-defined data model. CSV is more compact and efficient for storage and processing compared to JSON. Structured data eliminates redundant information.
• Multidimensional arrays and NetCDF data model: Well-established in scientific data with variables as multidimensional arrays. NetCDF allows for efficient representation with small coordinate variables and orthogonal indexing. Flattening multidimensional data into tables leads to redundant storage and poor query performance.
• Benchmark results: Comparing Xarray + Zarr + Icechunk with DuckDB + Parquet shows Xarray is much faster for typical weather forecast data queries. Xarray's result is a coordinate-aware array for easy visualization and further processing, while DuckDB's result is a dataframe that's difficult to reconstruct.
• "Tensors as columns" vs. "Tensors inside Tables": There's a partial analogy between NetCDF variables and table columns, but they are fundamentally incompatible. Arrow and some geospatial databases can store tensors inside tables, but there are limitations. Zarr is designed to handle large contiguous tensors.
• When to use tensors vs. tables: Use tables for data that maps well to the relational model. Use tensors for physical world data like weather, geospatial, bioimaging, and more.
• What's next for the array data ecosystem: Tensor-specific data systems are less mature than tabular ones. Xarray and its ecosystem are advancing, like with Icechunk adding ACID transactions. Earthmover Platform offers an end-to-end cloud data management solution for arrays. There are few commercial cloud services for array data, so engineering teams need to build from scratch.