I've recently come up against a file of sql that's too big for SQuirreL SQL to handle... I keep getting a Java Memory Heap error when I try and paste it all in. The quick answer appears to be csplit. To break a giant file into files of 10,000 lines each, you can call this:

csplit tokenLinks.sql 10000 {100} <<< Don't use!

That basically says to chunk the sql file into 100 groups of 10,000 lines each.

The problem here is if your file doesn't have 100 groups of 10,000 lines... I used 100 b/c the number of lines often changed from file to file, and I didn't want to have long last files if I pitched low. That is, 15 groups of 10,000 lines isn't really enough if the file's got 200k lines.

So csplit gives you an "out of range" error if you shoot too high and then, get this, erasing all the files it made. Nice. So it's worthless.

Not so fast. [Chapter 35] 35.10 Splitting Files by Context: csplit:

Unfortunately, if you tell csplit to create more files than it's able to, this produces an 'out of range' error. Furthermore, when csplit encounters an error, it exits by removing any files it created along the way. (A bug, if you ask me.) This is where the -k option comes in. Specify -k to keep the files around, even when the 'out of range' message occurs.


csplit -k tokenLinks.sql 10000 {100}

Happy and reponsitive [sic]. And, at least on my iBook running 10.4, the last file does have the last entry, so nothing's missed.

Labels: