Karsten Weinert compares two databases:
On my laptop with 16 GB RAM, I would like to perform a matrix-vector multiplication with a sparse matrix of around 10 million columns and 2500 rows. The matrix has approximately only 2% non-zero entries, but this are still 500 million numbers and the column/row information, too large to work comfortably in-memory.
A while ago, I tried using
sqlitefor this task. It kind of worked, but was too slow to be useful. This weekend, I revisited the problem and tried usingduckdb.
Read on for the results. I’ve heard enough positives about DuckDB over the past few weeks that it makes me want to try it out. H/T R-Bloggers.