Smalltalk › Cincom › VisualWorks

Reading CSV files

_‹ Previous Topic Next Topic _›

Classic

List

Threaded

23 messages Options

Stefan Unterweger

Reading CSV files

Hello,

I need to read data in from a CSV file. Is there already some package
that does this? A quick search in the public repository did not reveal
such a thing, but maybe I've just been searching using the wrong
keywords. I thought I'd better ask before wasting my time re-inventing
the wheel...

Thanks,
Stefan
_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

jarober

Re: Reading CSV files

VRFileReader can do that

reader := VRFileReader new.
reader fieldSeparator: $,.
all := (reader read: 'myCSVFile.csv' into: SomeObject) modelCollection.

SomeObject is assumed to have as many instance variable as there are columns of data in the file. It will assume CR as teh recordSeparator, but you can tell it what to use as well

On Aug 3, 2011, at 11:14 AM, Stefan Unterweger wrote:

> Hello,
>
> I need to read data in from a CSV file. Is there already some package
> that does this? A quick search in the public repository did not reveal
> such a thing, but maybe I've just been searching using the wrong
> keywords. I thought I'd better ask before wasting my time re-inventing
> the wheel...
>
>
> Thanks,
> Stefan
> _______________________________________________
> vwnc mailing list
> [hidden email]
> http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

James Robertson
http://www.jarober.com
[hidden email]

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

Robert Taylor

Re: Reading CSV files

In reply to this post by Stefan Unterweger

On 8/3/2011 8:14 AM, Stefan Unterweger wrote:

It's not much of a wheel. I pulled this out of a method I used to read
a customers .csv file (aFile).

| myFile myStream myLine allLines |

myFile := aFile asFilename.
myStream := myFile readStream lineEndCR.
allLines := List new.
"The next line skips the row of column headers in the .csv file."
myStream upTo: Character cr.
[[myStream atEnd] whileFalse: [
myLine := myStream upTo: Character cr.
myLine notEmpty ifTrue: [allLines add: myLine]
]]
ensure: [myStream close].
^allLines

Regards,
Bob Taylor
Taylor Software
_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

Boris Popov, DeepCove Labs (SNN)

Re: Reading CSV files

That just reads the lines, the tricky bits are splitting the tokens and
respecting qualifiers like "s to wrap fields containing separators in
their value.

tokensBasedOn: separator quotedBy: quote

|quoted values stream token |
values := OrderedCollection new.
stream := self readStream.
quoted := false.
token := (String new: 10) writeStream.
[stream atEnd] whileFalse:
[|next|
next := stream next.
quoted
ifTrue:
[next = quote
ifFalse: [token nextPut: next]
ifTrue:
[stream peek = quote
ifTrue:
[token nextPut:
stream next.]
ifFalse: [quoted :=
false]]]
ifFalse:
[next = separator ifTrue:
[values add: token contents.
token reset]
ifFalse:
[next = quote
ifFalse: [token
nextPut: next]
ifTrue: [quoted
:= true]]]
].
values add: token contents.
^values

-Boris

-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On
Behalf Of Robert Taylor
Sent: Wednesday, August 03, 2011 11:53 AM
To: [hidden email]
Subject: Re: [vwnc] Reading CSV files

On 8/3/2011 8:14 AM, Stefan Unterweger wrote:
> Hello,
>
> I need to read data in from a CSV file. Is there already some package
> that does this? A quick search in the public repository did not reveal

> such a thing, but maybe I've just been searching using the wrong
> keywords. I thought I'd better ask before wasting my time re-inventing

> the wheel...
>
>
> Thanks,
> Stefan
> _______________________________________________
> vwnc mailing list
> [hidden email]
> http://lists.cs.uiuc.edu/mailman/listinfo/vwnc
>

Janko Mivšek

Re: Reading CSV files

In reply to this post by Stefan Unterweger

S, Stefan Unterweger piše:

> I need to read data in from a CSV file. Is there already some package
> that does this? A quick search in the public repository did not reveal
> such a thing, but maybe I've just been searching using the wrong
> keywords. I thought I'd better ask before wasting my time re-inventing
> the wheel...

DelimitedFile in package Aida-Support can read and write CSV files, in
many codepages and with settable delimiter.

Best regards
Janko

--
Janko Mivšek
Aida/Web
Smalltalk Web Application Server
http://www.aidaweb.si
_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

Holger Guhl

Re: Reading CSV files

In reply to this post by Stefan Unterweger

Stefan,
before reinventing the wheel you might give GHCsvImportExport a try. You'll find it in
Contributions\Heeg.
The focus is on streaming (best if you have huge files) and it's well prepared for reading quoted
cell values (important if the export wrote out embedded CRs). CsvReader class #onFileNamed:
auto-detects unicode encoding and switches accordingly, provided a BOM is present.
There is also a writer part that supports creating CSV files. Have a look at CsvWriter class
#example to get an impression on stream-out and converting capabilities.
If you have questions on that tiny piece, I am glad to answer.

Cheers
Holger

Am 03.08.2011 17:14, schrieb Stefan Unterweger:

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

Dave Stevenson-3

Re: Reading CSV files

Perhaps it would have been easier to ask who did *not* have a .csv reader handy.

Dave Stevenson
[hidden email]

From: Holger Guhl <[hidden email]>
To: Stefan Unterweger <[hidden email]>
Cc: [hidden email]
Sent: Wed, August 3, 2011 1:10:54 PM
Subject: Re: [vwnc] Reading CSV files

Stefan,
before reinventing the wheel you might give GHCsvImportExport a try. You'll find it in
Contributions\Heeg.
The focus is on streaming (best if you have huge files) and it's well prepared for reading quoted
cell values (important if the export wrote out embedded CRs). CsvReader class #onFileNamed:
auto-detects unicode encoding and switches accordingly, provided a BOM is present.
There is also a writer part that supports creating CSV files. Have a look at CsvWriter class
#example to get an impression on stream-out and converting capabilities.
If you have questions on that tiny piece, I am glad to answer.

Cheers
Holger

Am 03.08.2011 17:14, schrieb Stefan Unterweger:

> http://lists.cs.uiuc.edu/mailman/listinfo/vwnc
>
>

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

Boris Popov, DeepCove Labs (SNN)

Re: Reading CSV files

Heh, indeed, so much for not reinventing the wheel, eh?

-Boris

From: [hidden email] [mailto:[hidden email]] On Behalf Of Dave Stevenson
Sent: Thursday, August 04, 2011 3:30 PM
To: [hidden email]
Subject: Re: [vwnc] Reading CSV files

Perhaps it would have been easier to ask who did *not* have a .csv reader handy.

Dave Stevenson
[hidden email]

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

Jon Paynter-2

Re: Reading CSV files

On Thu, Aug 4, 2011 at 12:31 PM, Boris Popov, DeepCove Labs <[hidden email]> wrote:

Heh, indeed, so much for not reinventing the wheel, eh?

-Boris

From: [hidden email] [mailto:[hidden email]] On Behalf Of Dave Stevenson
Sent: Thursday, August 04, 2011 3:30 PM

To: [hidden email]
Subject: Re: [vwnc] Reading CSV files

Perhaps it would have been easier to ask who did *not* have a .csv reader handy.

Dave Stevenson
[hidden email]

good point there....

but count me as one who doesnt have a pre-made one handy. I just roll my own whenever its needed:
aStream nextLine asArrayOfSubstringsSeparatedBy: $,

or something close to that - depending on the needs.

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

Boris Popov, DeepCove Labs (SNN)

Re: Reading CSV files

Jon,

How,about,”something,like,this?”

;)

-Boris

From: Jon Paynter [mailto:[hidden email]]
Sent: Thursday, August 04, 2011 4:29 PM
To: Boris Popov, DeepCove Labs
Cc: Dave Stevenson; [hidden email]
Subject: Re: [vwnc] Reading CSV files

On Thu, Aug 4, 2011 at 12:31 PM, Boris Popov, DeepCove Labs <[hidden email]> wrote:

Heh, indeed, so much for not reinventing the wheel, eh?

-Boris

From: [hidden email] [mailto:[hidden email]] On Behalf Of Dave Stevenson
Sent: Thursday, August 04, 2011 3:30 PM

To: [hidden email]

Subject: Re: [vwnc] Reading CSV files

Perhaps it would have been easier to ask who did *not* have a .csv reader handy.

Dave Stevenson
[hidden email]

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

Jon Paynter-2

Re: Reading CSV files

On Thu, Aug 4, 2011 at 1:37 PM, Boris Popov, DeepCove Labs <[hidden email]> wrote:

Jon,

How,about,”something,like,this?”

;)

-Boris

yeah that would break my simpleminded method. Which is why I said "depending on the needs"

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

Travis Griggs-4

Re: Reading CSV files

In reply to this post by Boris Popov, DeepCove Labs (SNN)

On Aug 4, 2011, at 1:37 PM, Boris Popov, DeepCove Labs wrote:

Jon,

How,about,”something,like,this?”

I'm curious what the xStreams solution would look like...

Travis Griggs

Objologist

Simplicity is the Ultimate Sophistication -- Leonardo da Vinci

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

Claus Kick

Re: Reading CSV files

In reply to this post by Stefan Unterweger

Now the question is: why was the user unable to find the packages which are available?

--
Claus Kick

"Wenn Sie mich suchen: Ich halte mich in der Nähe des Wahnsinns auf.
Genauer gesagt auf der schmalen Linie zwischen Wahnsinn und Panik.
Gleich um die Ecke von Todesangst, nicht weit weg von Irrwitz und Idiotie."

"If you are looking for me: I am somewhere near to lunacy.
More clearly, on the narrow path between lunacy and panic.
Right around the corner of fear of death,
not far away from idiocy and insanity."

Von: "Boris Popov, DeepCove Labs" <[hidden email]>
Gesendet: 04.08.2011 21:31:52
An: "Dave Stevenson" <[hidden email]>, [hidden email]
Betreff: Re: [vwnc] Reading CSV files

Heh, indeed, so much for not reinventing the wheel, eh?

-Boris

From: [hidden email] [mailto:[hidden email]] On Behalf Of Dave Stevenson
Sent: Thursday, August 04, 2011 3:30 PM
To: [hidden email]
Subject: Re: [vwnc] Reading CSV files

Perhaps it would have been easier to ask who did *not* have a .csv reader handy.

Dave Stevenson
[hidden email]

From: Holger Guhl <[hidden email]>
To: Stefan Unterweger <[hidden email]>
Cc: [hidden email]
Sent: Wed, August 3, 2011 1:10:54 PM
Subject: Re: [vwnc] Reading CSV files

Stefan,
before reinventing the wheel you might give GHCsvImportExport a try. You'll find it in
Contributions\Heeg.
The focus is on streaming (best if you have huge files) and it's well prepared for reading quoted
cell values (important if the export wrote out embedded CRs). CsvReader class #onFileNamed:
auto-detects unicode encoding and switches accordingly, provided a BOM is present.
There is also a writer part that supports creating CSV files. Have a look at CsvWriter class
#example to get an impression on stream-out and converting capabilities.
If you have questions on that tiny piece, I am glad to answer.

Cheers
Holger

Am 03.08.2011 17:14, schrieb Stefan Unterweger:

> Hello,
>
> I need to read data in from a CSV file. Is there already some package
> that does this? A quick search in the public repository did not reveal
> such a thing, but maybe I've just been searching using the wrong
> keywords. I thought I'd better ask before wasting my time re-inventing
> the wheel...
>
>
> Thanks,
> Stefan
> _______________________________________________
> vwnc mailing list
> [hidden email]
> http://lists.cs.uiuc.edu/mailman/listinfo/vwnc
>
>

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

Michael Lucas-Smith-2

Re: Reading CSV files

In reply to this post by Travis Griggs-4

Dunno about the xStreams solution, but there are a few considerations in the Xtreams solution:

First off, how memory intensive does it need to be? Let's imagine that every cell on every line is a terabyte long and go with the most pathelogical case we can muster. We'll also ignore quoting and escaping and come back to that later:

('test,this,out

some,more,please' reading ending: Character cr) slicing do: [:line |

(line ending: $,) slicing do: [:cell |

Transcript writing cr; write: cell]].

The cell is a stream in this case, so we can read from it and process it however we want. In the case where the cell is a terabyte long, assuming the Transcript truncates or goes to an external resource, we will not exhaust the machines memory and crash. This is a neat feature to have.

In the case where you don't have such concerns about memory usage, there's this variant:

('test,this,out

some,more,please' reading ending: Character cr) slicing collect: [:line |

(line ending: $,) slicing collect: #rest].

This is your typical, "Give me back an array of arrays" type thing.

It's interesting to compare the Xtreams approach versus, say, simple string manipulations:

('test,this,out

some,more,please' tokensBasedOn: Character cr) collect: [:line | line tokensBasedOn: $,]

Either you need streaming because you're concerned about resource limitations, in which case the top example is the most appropriate, or you don't need streaming because you intend to have all of the data in memory anyway, in which case the middle example is not that useful when the string example is concise and "good enough".

Finally, let's talk about doing it properly. How do we handle quoting and escaping, eg: Boris's example. To put it simply, no one liner is going to be convenient for that, you need to either write a full program or go half-way to that and write a grammar. But since there are already packages to parse CSV, possibly even with tests? I'm not going to create a grammar for it here in this email. If, however, someone informs me that there is no appropriate CSV parser with tests, I'll look in to adding one to Xtreams-Grammar for you to use.

Cheers,

Michael

On Aug 4, 2011, at 1:52 PM, Travis Griggs wrote:

On Aug 4, 2011, at 1:37 PM, Boris Popov, DeepCove Labs wrote:

Jon,

How,about,”something,like,this?”

I'm curious what the xStreams solution would look like...

--
Travis Griggs
Objologist
Simplicity is the Ultimate Sophistication -- Leonardo da Vinci

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

Stefan Unterweger

Re: Reading CSV files

In reply to this post by Holger Guhl

* Holger Guhl on Wed, Aug 03, 2011 at 08:10:54PM +0200:
> before reinventing the wheel you might give GHCsvImportExport a try.
> You'll find it in Contributions\Heeg.
> The focus is on streaming (best if you have huge files) and it's
> well prepared for reading quoted cell values (important if the
> export wrote out embedded CRs). CsvReader class #onFileNamed:
> auto-detects unicode encoding and switches accordingly, provided a
> BOM is present.

That is just what I need -- thanks.
_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

Stefan Unterweger

Re: Reading CSV files

In reply to this post by Boris Popov, DeepCove Labs (SNN)

* Boris Popov, DeepCove Labs on Thu, Aug 04, 2011 at 01:37:39PM -0700:
> How,about,"something,like,this?"
>
> ;)
>
> -Boris

Yeah, that's exactly the kind of data that I had in mind, plus the
Unicode stuff that Holger mentioned. Otherwise, it had indeed been
quicker to just roll out my own. :o)

Thanks for all the suggestions.
Stefan
_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

Holger Guhl

Re: Reading CSV files

In reply to this post by Boris Popov, DeepCove Labs (SNN)

I claim to be first :-)
GHCsvImportExport was "invented" in 1995. I implemented it for testing and importing purposes in a database project, after that I forgot it for years. Then all of a sudden it showed up again because some customer was reporting some troubles. I never found out how they got that inofficial tool into their hands, but nevermind, I fixed it. Strange as the world is, more and more projects showed up that used that tiny tool, so we decided to publish it as Heeg Contrib. But of course, the topic is so easy and CSV is a very convenient and general format, that it is too natural to implement the little code.

Cheers,
Holger

Am 04.08.2011 21:31, schrieb Boris Popov, DeepCove Labs:

Heh, indeed, so much for not reinventing the wheel, eh?

-Boris

From: [hidden email] [[hidden email]] On Behalf Of Dave Stevenson
Sent: Thursday, August 04, 2011 3:30 PM
To: [hidden email]
Subject: Re: [vwnc] Reading CSV files

Perhaps it would have been easier to ask who did *not* have a .csv reader handy.

Dave Stevenson
[hidden email]

From: Holger Guhl <[hidden email]>
To: Stefan Unterweger <[hidden email]>
Cc: [hidden email]
Sent: Wed, August 3, 2011 1:10:54 PM
Subject: Re: [vwnc] Reading CSV files

Stefan,
before reinventing the wheel you might give GHCsvImportExport a try. You'll find it in
Contributions\Heeg.
The focus is on streaming (best if you have huge files) and it's well prepared for reading quoted
cell values (important if the export wrote out embedded CRs). CsvReader class #onFileNamed:
auto-detects unicode encoding and switches accordingly, provided a BOM is present.
There is also a writer part that supports creating CSV files. Have a look at CsvWriter class
#example to get an impression on stream-out and converting capabilities.
If you have questions on that tiny piece, I am glad to answer.

Cheers
Holger

Am 03.08.2011 17:14, schrieb Stefan Unterweger:
> Hello,
>
> I need to read data in from a CSV file. Is there already some package
> that does this? A quick search in the public repository did not reveal
> such a thing, but maybe I've just been searching using the wrong
> keywords. I thought I'd better ask before wasting my time re-inventing
> the wheel...
>
>
> Thanks,
> Stefan
> _______________________________________________
> vwnc mailing list
> [hidden email]
> http://lists.cs.uiuc.edu/mailman/listinfo/vwnc
>
>

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc
_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

Holger Guhl

Re: Reading CSV files

In reply to this post by Claus Kick

There are two arguments for Stefan's question:
1. The user loves to discuss his question with the community. He trusts the community and the common
knowledge so much that he wants to hear what other users say, hoping for the best possible
recommendation without having to test all options himself. Why not?
2. In the face of the prominent launcher operation to open the Parcel Manager, users forget that
there is another option to load parcels just by name pattern. With launcher menu "System> Load
Parcels named..." you can enter a pattern, e.g. "*CSV*". Hopefully, the publisher has chosen a name
that contains the topic, and not some "MyBestTool" or just a brand name, then you'll see the desired
stuff. Ok, one could ask why such a search function is not part of the Parcel Manager, but I'm not
in complaining mode today.

Cheers,
Holger

Am 04.08.2011 22:55, schrieb Claus Kick:

> Now the question is: why was the user unable to find the packages which are available?
>
> --
> Claus Kick
>
> "Wenn Sie mich suchen: Ich halte mich in der Nähe des Wahnsinns auf.
> Genauer gesagt auf der schmalen Linie zwischen Wahnsinn und Panik.
> Gleich um die Ecke von Todesangst, nicht weit weg von Irrwitz und Idiotie."
>
> "If you are looking for me: I am somewhere near to lunacy.
> More clearly, on the narrow path between lunacy and panic.
> Right around the corner of fear of death,
> not far away from idiocy and insanity."

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

Dave Stevenson-3

Re: Reading CSV files

No reason why the parcel mgr can't also scan parcel comments in the background - even cache a keyword index.

Dave Stevenson
[hidden email]

From: Holger Guhl <[hidden email]>
To: Claus Kick <[hidden email]>
Cc: [hidden email]
Sent: Fri, August 5, 2011 6:09:47 AM
Subject: Re: [vwnc] Reading CSV files

There are two arguments for Stefan's question:
1. The user loves to discuss his question with the community. He trusts the community and the common
knowledge so much that he wants to hear what other users say, hoping for the best possible
recommendation without having to test all options himself. Why not?
2. In the face of the prominent launcher operation to open the Parcel Manager, users forget that
there is another option to load parcels just by name pattern. With launcher menu "System> Load
Parcels named..." you can enter a pattern, e.g. "*CSV*". Hopefully, the publisher has chosen a name
that contains the topic, and not some "MyBestTool" or just a brand name, then you'll see the desired
stuff. Ok, one could ask why such a search function is not part of the Parcel Manager, but I'm not
in complaining mode today.

Cheers,
Holger

Am 04.08.2011 22:55, schrieb Claus Kick:

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

Carl Gundel

[7.8] Modal dialog?

In reply to this post by Claus Kick

How can I create a window which looks like a dialog, which makes only the single window it was opened from to be disabled, and which acts like a non dialog in any other respect? In other words, events are handled as if it were not a dialog at all. Other windows which are not the parent of my dialog can be activated and used while the dialog is still open.

1) Looks like a dialog (has same frame and buttons, non-resizable, etc.)
2) Only its parent is disabled
3) Behaves like a non-dialog
4) Other windows are responsive

Thanks,

-Carl Gundel
http://www.libertybasic.com
http://www.runbasic.com
_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc