Ggtcf

I want to split a sparse matrix (type: scipy.sparse.csr.csr_matrix) into N parts in the right order and iterative over them to use them as inputs for classification prediction.

However, if I try to convert the sparse matrix to a dense matrix with csr_matrix.toarray(), I get a MemoryError. The size of the converted array would take up 70gb of RAM, tested with the method in this thread (Error Converting Sparse Matrix to Array with scipy.sparse.csc_matrix.toarray())

So I can't use numpy.array_split() to split the array, because it only works on dense matrices.

Is there another way to split/slice the sparse matrix into N sparse matrices?

Thanks.

Additional edit:

So the chunking goes like this with a sparse array X_test:

# X_test is a sparse matrix with feature vectors

chunk_results = []
X_dense = csr_matrix.toarray(X_test)
X_test_chunks = np.array_split(X_dense, 20)
for chunk in X_test_chunks:
 chunk_results.append(classifier.predict(chunk))
prediction = np.concatenate(chunk_results)

Here an example of a conversion for a small sparse matrix to dense:

# sparse
(0, 0) -0.5
(0, 1) 3.8570557155110414
(0, 2) -1.975755301731886
(1, 0) -3.5
(1, 1) 6.54336961554629
(1, 2) -3.311314222363026

# dense
[[-0.5 3.85705572 -1.9757553 ]
 [-3.5 6.54336962 -3.31131422]]

Each of those two inner arrays in the dense matrix is a Feature-Vector representing one object. Basically, assuming we would try to classify those by doing the chunk split technique with conversion on this example, we would take n=2 and get [-0.5 3.85705572 -1.9757553] and [-3.5 6.54336962 -3.31131422] as two chunks. In case we would have a bigger matrix with more entries, there would be several of these vector-arrays per chunk.

SOLUTION

I made a workaround by dividing the number of rows of the sparse matrix with the desired number of chunks, and slice the matrix in chunks of rows of that number as hpaulj suggested:

Blockquote
np.array_split does sliced indexing of the dense array, [arr[i:j] for i,j in ....]. So you could do the same sort of indexing on a csr matrix. Sparse slicing isn't as fast as the dense version, but it works (for the right sparse format).