I would not recommend using parquet instead of csv. Indeed, parquet is a type of wooden flooring, while csv is a human readable file format. As you can see, it is not wise to replace one with the other. Don’t hesitate about asking more questions regarding your home design!

It’s pretty much an industry standard at this point, I wouldn’t be worried about its future. I’ve used it and still do once in a while.

Do you use it? When?

Parquet is really used for big data batch data processing. It’s columnar-based file format and is optimized for large, aggregation queries. It’s non-human readable so you need a library like apache arrow to read/write to it.

I would use parquet in the following circumstances (or combination of circumstances):

• The data is very large
• I’m integrating this into an analytical query engine (Presto, etc.)
• I’m transporting data that needs to land in an analytical data warehouse (Snowflake, BigQuery, etc.)
• Consumed by data scientists, machine learning engineers, or other data engineers

Since the data is columnar-based, doing queries like select sum(sales) from revenue is much cheaper and faster if the underlying data is in parquet than csv.

The big advantage of csv is that it’s more portable. csv as a data file format has been around forever, so it is used in a lot of places where parquet can’t be used.

parquet is cloesely tied to the apache foundation, because it was designed as a storage format for hadoop.

But many data processing libraries offer interfaces to handle parquet files so you can use it outside of the hadoop eco system.

It’s really good for archiving data, because the format can store a lot of data with relatively low disk space, while still providing ok read performance because often times you won’t need to read the whole file due to how they are structured, where csv files would be a lot of plaintext taking up more diskspace.

Yeah depends on what you’re using it for. CSV is terrible in many many ways but it is widely supported and much less complex.

I would guess if you’re considering Parquet then your use case is probably one where you should use it.

JSON is another option, but I would only use it if you can guarantee that you’ll never have more than like 100MB of data. Large JSON files are extremely painful.

In the deep learning community, I know of someone using parquet for the dataset and annotations. It allows you to select which data you want to retrieve from the dataset and stream only those, and nothing else. It is a rather effective method for that if you have many different annotations for different use cases and want to be able to select only the ones you need for your application.

I can’t address the first part, but for your last paragraph, if you’re sharing with humans, csv is fine. If you’re sharing with humans and machines, JSON or yaml or something similar is probably fine. If you’re only moving things around to give to machines, what to use depends on constraints you might have and use cases

Parquet 4 eva

Csv is for arcane software or if you don’t know where it’s going.

Hdf5 is for Matlab interoperability

Otherwise I use parquet (orc could also work, but I never actually use it). Sometimes parquet has problems with Pandas or polars but I’ve always been able to fix it by using pyarrow

Friends don’t let friends use csv in 2024. Excel needs a good parquet importer and exporter today. Ya hearing Microsoft? Quit pissing around with recall and build something useful!