Make CSV files do whatever you want, really fast
July 5, 2007 PC Tips, Reviews, Web Programming No Comments
The following is a sponsored review of software that helps you work with large CSV files for data transfer and migration.
If you work in IT, you’ve probably had to deal with csv files at one time or another. The idea of a .csv file is, of course, that it is simply data separated by commas, with a carriage return indicating the end of a particular table entry and the beginning of the next. The site has a couple of versions of the CSV reader available for download, one written in Java and one in .NET. I went with the .NET version first, since it seemed to be a demo of the more full-featured downloadable version.
Ok, the first thing I figured out that wasn’t immediately obvious to me (although it does clearly say on the site that it’s a .NET class) was that I needed Visual Studio installed. Since my hard drive recently died, I don’t really have any of my old software on my computer, so I guess I now have the impetus needed to spend an hour watching Visual Studio re-install.
While I was waiting on the install, I perused the pages on the CSV Reader site to learn more about the product. What impressed me off the bat was how flexible the class seems to be–you can use it to parse records and insert them for web viewing, into an SQL database, etc. Also, the speed is impressive, as the site states “[r]ough benchmarks on a 2 Ghz processor, parsing common comma separated columns is 20 MB, or 390,000 rows with 5 columns each, almost 2,000,000 cells total, of data parsing per second.” Now, this might not be something everyone needs, but if you’ve ever used Excel to its breaking point (admittedly much higher in 2007 than in 2003) you’ll be glad you have the extra speed and ability to parse huge data files.
Also noted from the CSV Reader site is the fact that it removes the need to use Excel as an intermediary when working with .csv files. In fact, CSV Reader can work with .xls (Excel’s native format) and the ever-popular .xml format directly, alleviating the need for slower, GUI-heavy programs to do the heavy crunching work.
One of the things that I really liked about CSV Reader was the demo sample projects included with the demo, as well as the code samples available on the site itself. Also, the pricing is pretty good, starting at $150 (if you just need to use the class in-house to convert data) all the way up to royalty-free distribution rights along with the C# source code for a mere $750. Not bad pricing, considering that, once again, this is clearly targeted at DBAs and others with absolutely huge data streams that they need to deal with, not the home user with a knitting club membership database that needs to be moved from Excel to SQL.