[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Sc-devel] [commit?] FileReader ability to skip rows (start offset, subsampling)

Hi -

I'm working with some very large CSV data files at the moment. sclang
ends up running out of memory if using CSVFileReader on a file that's
too big.

I've modified it to be able to "subsample" a data file. For example,
pass in the parameter "subsample:10" and it only adds every 10th row
to the array it's returning. Also a "startRow" parameter which is
particularly handy when you want to skip the preamble or headers of a
text data file.

Diff attached. OK to commit?


looks fine for me.

best, a
Alberto de Campo
Bergstrasse 59/33
A-8020 Graz, Austria
e-mail : decampo@xxxxxx