ASCII Streams with mixed Line-End conventions

Hi there,

we're importing some data from csv files. There's a lot of good reasons not to do it, but anyways... ;-)

Just yesterday I got a complaint from a user who couldn't figure out how to import their data. It turns out their source system exports data with mixed CrLf and Lf line-ends.

I know there are quite a few things we could do here. Like pre-process the file, look fo the first occurence of either Cr, CrLf or Lf and make sure to unify them befor starting the CSV import. But maybe someone has an even easier way to do it, maybe even with on-board tools of VAST?

Joachim

Hi Joachim,

The #nextLine method of #CfsReadFileStream has the following comment:

nextLine

"Answer the elements between the current position and the next lineDelimiter. 

 

If #shouldSearchForAllStandardDelimiters answers false 

(likely, the user has specified an explicit line delimiter via #lineDelimiter:)

then we ONLY search for that delimiter. However, if it answers true

(likely the stream current delimiter is the default one), then we try to look

for any of the standard delimiters: Cr, Lf, and CrLf. This is useful when we are

reading streams that could have been written on different platforms.

Answers:

<ByteArray | String>"

It seems to me that covers you case.  I haven't checked but I guess other streams may have similar methods.

Lou

On Monday, September 21, 2020 at 4:02:17 AM UTC-4 [hidden email] wrote:

Hi there,

we're importing some data from csv files. There's a lot of good reasons not to do it, but anyways... ;-)

Just yesterday I got a complaint from a user who couldn't figure out how to import their data. It turns out their source system exports data with mixed CrLf and Lf line-ends.

I know there are quite a few things we could do here. Like pre-process the file, look fo the first occurence of either Cr, CrLf or Lf and make sure to unify them befor starting the CSV import. But maybe someone has an even easier way to do it, maybe even with on-board tools of VAST?

Joachim

Thanks Lou,

I was somewhat assuming that if I set #lineDelimiter (a variable that our users can modify in the Applications' GUI), the Stream will only look for this one. The comment suggests I can set (or override) #shouldSearchForAllStandardDelimiters and combine the two somehow.

I will have to play with this....

BUT of course, Richard is bringing up a very valid point. We're using NeoCSVReader to parse CSV fields from these files. So the challenge here will be to find the right way to fiddle the line-end detection into that.... In the worst of cases, we'll have to do the "line reading" outside of NeoCSVReader and only feed each single line into the Reader instead of using #upToEnd....

Sigh. So much trouble for a single exception to a simple rule....

Anyways: Thanks Lou and Richard for your input. You showed me both a possible solution to take a closer look at as well as a very important caveat...

Joachim

[hidden email] schrieb am Montag, 21. September 2020 um 15:02:47 UTC+2:

Hi Joachim,

The #nextLine method of #CfsReadFileStream has the following comment:

nextLine

"Answer the elements between the current position and the next lineDelimiter. 

 

If #shouldSearchForAllStandardDelimiters answers false 

(likely, the user has specified an explicit line delimiter via #lineDelimiter:)

then we ONLY search for that delimiter. However, if it answers true

(likely the stream current delimiter is the default one), then we try to look

for any of the standard delimiters: Cr, Lf, and CrLf. This is useful when we are

reading streams that could have been written on different platforms.

Answers:

<ByteArray | String>"

It seems to me that covers you case.  I haven't checked but I guess other streams may have similar methods.

Lou

On Monday, September 21, 2020 at 4:02:17 AM UTC-4 [hidden email] wrote:

Hi there,

we're importing some data from csv files. There's a lot of good reasons not to do it, but anyways... ;-)

Just yesterday I got a complaint from a user who couldn't figure out how to import their data. It turns out their source system exports data with mixed CrLf and Lf line-ends.

I know there are quite a few things we could do here. Like pre-process the file, look fo the first occurence of either Cr, CrLf or Lf and make sure to unify them befor starting the CSV import. But maybe someone has an even easier way to do it, maybe even with on-board tools of VAST?

Joachim

Hi Joachim,

Perhaps you can modify #nextLine or extend the class with a new method that handles the embedded line end character.  perhaps something as simple as counting the double quotes and continuing to read if there is an odd number.

Lou

On Tuesday, September 22, 2020 at 2:51:12 AM UTC-4 [hidden email] wrote:

Thanks Lou,

I was somewhat assuming that if I set #lineDelimiter (a variable that our users can modify in the Applications' GUI), the Stream will only look for this one. The comment suggests I can set (or override) #shouldSearchForAllStandardDelimiters and combine the two somehow.

I will have to play with this....

BUT of course, Richard is bringing up a very valid point. We're using NeoCSVReader to parse CSV fields from these files. So the challenge here will be to find the right way to fiddle the line-end detection into that.... In the worst of cases, we'll have to do the "line reading" outside of NeoCSVReader and only feed each single line into the Reader instead of using #upToEnd....

Sigh. So much trouble for a single exception to a simple rule....

Anyways: Thanks Lou and Richard for your input. You showed me both a possible solution to take a closer look at as well as a very important caveat...

Joachim

[hidden email] schrieb am Montag, 21. September 2020 um 15:02:47 UTC+2:

Hi Joachim,

The #nextLine method of #CfsReadFileStream has the following comment:

nextLine

"Answer the elements between the current position and the next lineDelimiter. 

 

If #shouldSearchForAllStandardDelimiters answers false 

(likely, the user has specified an explicit line delimiter via #lineDelimiter:)

then we ONLY search for that delimiter. However, if it answers true

(likely the stream current delimiter is the default one), then we try to look

for any of the standard delimiters: Cr, Lf, and CrLf. This is useful when we are

reading streams that could have been written on different platforms.

Answers:

<ByteArray | String>"

It seems to me that covers you case.  I haven't checked but I guess other streams may have similar methods.

Lou

On Monday, September 21, 2020 at 4:02:17 AM UTC-4 [hidden email] wrote:

Hi there,

we're importing some data from csv files. There's a lot of good reasons not to do it, but anyways... ;-)

Just yesterday I got a complaint from a user who couldn't figure out how to import their data. It turns out their source system exports data with mixed CrLf and Lf line-ends.

I know there are quite a few things we could do here. Like pre-process the file, look fo the first occurence of either Cr, CrLf or Lf and make sure to unify them befor starting the CSV import. But maybe someone has an even easier way to do it, maybe even with on-board tools of VAST?

Joachim

--
You received this message because you are subscribed to the Google Groups "VA Smalltalk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/va-smalltalk/d6244a91-501c-461e-aa08-7a585b3d488dn%40googlegroups.com.

Hi guys, 

Joachim, yes, that behavior on #nextLine and #shouldSearchForAllStandardDelimiters is somehow new (91 or 92) and I hope it can meet your requirements. If not, then I would like to hear about it.  Not sure if I understand the question NeoCSVReader. If that's still an issue, could you clarify?

Louis: no, please, don't modify base apps. 

Best,

Mariano

On Tue, Sep 22, 2020 at 10:48 AM Louis LaBrunda <[hidden email]> wrote:

Hi Joachim,

Perhaps you can modify #nextLine or extend the class with a new method that handles the embedded line end character.  perhaps something as simple as counting the double quotes and continuing to read if there is an odd number.

Lou

On Tuesday, September 22, 2020 at 2:51:12 AM UTC-4 [hidden email] wrote:

Thanks Lou,

I was somewhat assuming that if I set #lineDelimiter (a variable that our users can modify in the Applications' GUI), the Stream will only look for this one. The comment suggests I can set (or override) #shouldSearchForAllStandardDelimiters and combine the two somehow.

I will have to play with this....

BUT of course, Richard is bringing up a very valid point. We're using NeoCSVReader to parse CSV fields from these files. So the challenge here will be to find the right way to fiddle the line-end detection into that.... In the worst of cases, we'll have to do the "line reading" outside of NeoCSVReader and only feed each single line into the Reader instead of using #upToEnd....

Sigh. So much trouble for a single exception to a simple rule....

Anyways: Thanks Lou and Richard for your input. You showed me both a possible solution to take a closer look at as well as a very important caveat...

Joachim

[hidden email] schrieb am Montag, 21. September 2020 um 15:02:47 UTC+2:

Hi Joachim,

The #nextLine method of #CfsReadFileStream has the following comment:

nextLine

"Answer the elements between the current position and the next lineDelimiter. 

 

If #shouldSearchForAllStandardDelimiters answers false 

(likely, the user has specified an explicit line delimiter via #lineDelimiter:)

then we ONLY search for that delimiter. However, if it answers true

(likely the stream current delimiter is the default one), then we try to look

for any of the standard delimiters: Cr, Lf, and CrLf. This is useful when we are

reading streams that could have been written on different platforms.

Answers:

<ByteArray | String>"

It seems to me that covers you case.  I haven't checked but I guess other streams may have similar methods.

Lou

On Monday, September 21, 2020 at 4:02:17 AM UTC-4 [hidden email] wrote:

Hi there,

we're importing some data from csv files. There's a lot of good reasons not to do it, but anyways... ;-)

Just yesterday I got a complaint from a user who couldn't figure out how to import their data. It turns out their source system exports data with mixed CrLf and Lf line-ends.

I know there are quite a few things we could do here. Like pre-process the file, look fo the first occurence of either Cr, CrLf or Lf and make sure to unify them befor starting the CSV import. But maybe someone has an even easier way to do it, maybe even with on-board tools of VAST?

Joachim

--
You received this message because you are subscribed to the Google Groups "VA Smalltalk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/va-smalltalk/d6244a91-501c-461e-aa08-7a585b3d488dn%40googlegroups.com.

--

Mariano Martinez Peck
Software Engineer, Instantiations Inc.

Email: [hidden email]

Twitter: https://twitter.com/MartinezPeck

LinkedIn: www.linkedin.com/in/mariano-martinez-peck

Blog: https://marianopeck.wordpress.com/

Hi guys, 

Joachim, yes, that behavior on #nextLine and #shouldSearchForAllStandardDelimiters is somehow new (91 or 92) and I hope it can meet your requirements. If not, then I would like to hear about it.  Not sure if I understand the question NeoCSVReader. If that's still an issue, could you clarify?

Louis: no, please, don't modify base apps. 

Best,

Mariano

On Tue, Sep 22, 2020 at 10:48 AM Louis LaBrunda <[hidden email]> wrote:

Hi Joachim,

Perhaps you can modify #nextLine or extend the class with a new method that handles the embedded line end character.  perhaps something as simple as counting the double quotes and continuing to read if there is an odd number.

Lou

On Tuesday, September 22, 2020 at 2:51:12 AM UTC-4 [hidden email] wrote:

Thanks Lou,

I was somewhat assuming that if I set #lineDelimiter (a variable that our users can modify in the Applications' GUI), the Stream will only look for this one. The comment suggests I can set (or override) #shouldSearchForAllStandardDelimiters and combine the two somehow.

I will have to play with this....

BUT of course, Richard is bringing up a very valid point. We're using NeoCSVReader to parse CSV fields from these files. So the challenge here will be to find the right way to fiddle the line-end detection into that.... In the worst of cases, we'll have to do the "line reading" outside of NeoCSVReader and only feed each single line into the Reader instead of using #upToEnd....

Sigh. So much trouble for a single exception to a simple rule....

Anyways: Thanks Lou and Richard for your input. You showed me both a possible solution to take a closer look at as well as a very important caveat...

Joachim

[hidden email] schrieb am Montag, 21. September 2020 um 15:02:47 UTC+2:

Hi Joachim,

The #nextLine method of #CfsReadFileStream has the following comment:

nextLine

"Answer the elements between the current position and the next lineDelimiter. 

 

If #shouldSearchForAllStandardDelimiters answers false 

(likely, the user has specified an explicit line delimiter via #lineDelimiter:)

then we ONLY search for that delimiter. However, if it answers true

(likely the stream current delimiter is the default one), then we try to look

for any of the standard delimiters: Cr, Lf, and CrLf. This is useful when we are

reading streams that could have been written on different platforms.

Answers:

<ByteArray | String>"

It seems to me that covers you case.  I haven't checked but I guess other streams may have similar methods.

Lou

On Monday, September 21, 2020 at 4:02:17 AM UTC-4 [hidden email] wrote:

Hi there,

we're importing some data from csv files. There's a lot of good reasons not to do it, but anyways... ;-)

Just yesterday I got a complaint from a user who couldn't figure out how to import their data. It turns out their source system exports data with mixed CrLf and Lf line-ends.

I know there are quite a few things we could do here. Like pre-process the file, look fo the first occurence of either Cr, CrLf or Lf and make sure to unify them befor starting the CSV import. But maybe someone has an even easier way to do it, maybe even with on-board tools of VAST?

Joachim

--
You received this message because you are subscribed to the Google Groups "VA Smalltalk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/va-smalltalk/d6244a91-501c-461e-aa08-7a585b3d488dn%40googlegroups.com.

--

Mariano Martinez Peck
Software Engineer, Instantiations Inc.

Email: [hidden email]

Twitter: https://twitter.com/MartinezPeck

LinkedIn: www.linkedin.com/in/mariano-martinez-peck

Blog: https://marianopeck.wordpress.com/

Mariano,

Thanks. First of all, I can confirm #nextLine works as expected on teh example file that contained lines ending in both <cr><lf> and <lf>. So the key is not to set a lineDelimiter on the Stream.

To my surpise, NeoCSV also scans for cr/crlf/lf and does everything in the way it should.

The problem in this case is that I had implemented some clever code that first reads the first line of the File and tries to determine the lineDelimiter and then sets it on the input Stream. From then on, the Stream didn't handle any other lineDeliimiter than the one I had told it. So I was knocking out the cleverness of the Streams. I thought I was more clever. I learned something about this illusion today ;-)

Anyways: thank you Lou, Richard, Mariano for joining me in this journey and pushing me in the right direction.

Conclusion: PositionableStream, Cfs*FileStream and NeoCSVReader all handle line endings perfectly. If you expect Windows or Unix line endings and even a mix of these, just let them do their work and don't interfere. You're in the way if you try to be more clever ;-)

Joachim

[hidden email] schrieb am Donnerstag, 24. September 2020 um 22:00:01 UTC+2:

Hi guys, 

Joachim, yes, that behavior on #nextLine and #shouldSearchForAllStandardDelimiters is somehow new (91 or 92) and I hope it can meet your requirements. If not, then I would like to hear about it.  Not sure if I understand the question NeoCSVReader. If that's still an issue, could you clarify?

Louis: no, please, don't modify base apps. 

Best,

Mariano

On Tue, Sep 22, 2020 at 10:48 AM Louis LaBrunda <[hidden email]> wrote:

Hi Joachim,

Perhaps you can modify #nextLine or extend the class with a new method that handles the embedded line end character.  perhaps something as simple as counting the double quotes and continuing to read if there is an odd number.

Lou

On Tuesday, September 22, 2020 at 2:51:12 AM UTC-4 [hidden email] wrote:

Thanks Lou,

I was somewhat assuming that if I set #lineDelimiter (a variable that our users can modify in the Applications' GUI), the Stream will only look for this one. The comment suggests I can set (or override) #shouldSearchForAllStandardDelimiters and combine the two somehow.

I will have to play with this....

BUT of course, Richard is bringing up a very valid point. We're using NeoCSVReader to parse CSV fields from these files. So the challenge here will be to find the right way to fiddle the line-end detection into that.... In the worst of cases, we'll have to do the "line reading" outside of NeoCSVReader and only feed each single line into the Reader instead of using #upToEnd....

Sigh. So much trouble for a single exception to a simple rule....

Anyways: Thanks Lou and Richard for your input. You showed me both a possible solution to take a closer look at as well as a very important caveat...

Joachim

[hidden email] schrieb am Montag, 21. September 2020 um 15:02:47 UTC+2:

Hi Joachim,

The #nextLine method of #CfsReadFileStream has the following comment:

nextLine

"Answer the elements between the current position and the next lineDelimiter. 

 

If #shouldSearchForAllStandardDelimiters answers false 

(likely, the user has specified an explicit line delimiter via #lineDelimiter:)

then we ONLY search for that delimiter. However, if it answers true

(likely the stream current delimiter is the default one), then we try to look

for any of the standard delimiters: Cr, Lf, and CrLf. This is useful when we are

reading streams that could have been written on different platforms.

Answers:

<ByteArray | String>"

It seems to me that covers you case.  I haven't checked but I guess other streams may have similar methods.

Lou

On Monday, September 21, 2020 at 4:02:17 AM UTC-4 [hidden email] wrote:

Hi there,

we're importing some data from csv files. There's a lot of good reasons not to do it, but anyways... ;-)

Just yesterday I got a complaint from a user who couldn't figure out how to import their data. It turns out their source system exports data with mixed CrLf and Lf line-ends.

I know there are quite a few things we could do here. Like pre-process the file, look fo the first occurence of either Cr, CrLf or Lf and make sure to unify them befor starting the CSV import. But maybe someone has an even easier way to do it, maybe even with on-board tools of VAST?

Joachim

--
You received this message because you are subscribed to the Google Groups "VA Smalltalk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/va-smalltalk/d6244a91-501c-461e-aa08-7a585b3d488dn%40googlegroups.com.

--

Mariano Martinez Peck
Software Engineer, Instantiations Inc.

Email: [hidden email]

Twitter: https://twitter.com/MartinezPeck

LinkedIn: www.linkedin.com/in/mariano-martinez-peck

Blog: https://marianopeck.wordpress.com/

--
You received this message because you are subscribed to the Google Groups "VA Smalltalk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/va-smalltalk/b2eafb48-bebf-454b-b67e-e355f492b314n%40googlegroups.com.

On Thu, Sep 24, 2020, 23:36 Joachim Tuchel <[hidden email]> wrote:

Mariano,

Thanks. First of all, I can confirm #nextLine works as expected on teh example file that contained lines ending in both <cr><lf> and <lf>. So the key is not to set a lineDelimiter on the Stream.

To my surpise, NeoCSV also scans for cr/crlf/lf and does everything in the way it should.

The problem in this case is that I had implemented some clever code that first reads the first line of the File and tries to determine the lineDelimiter and then sets it on the input Stream. From then on, the Stream didn't handle any other lineDeliimiter than the one I had told it. So I was knocking out the cleverness of the Streams. I thought I was more clever. I learned something about this illusion today ;-)

One of the greatest failings of programmers is when they try to be clever. There are myriad counter-examples. I'll tell you my favourite, if you're interested.

Anyways: thank you Lou, Richard, Mariano for joining me in this journey and pushing me in the right direction.

Conclusion: PositionableStream, Cfs*FileStream and NeoCSVReader all handle line endings perfectly. If you expect Windows or Unix line endings and even a mix of these, just let them do their work and don't interfere. You're in the way if you try to be more clever ;-)

Joachim

[hidden email] schrieb am Donnerstag, 24. September 2020 um 22:00:01 UTC+2:

Hi guys, 

Joachim, yes, that behavior on #nextLine and #shouldSearchForAllStandardDelimiters is somehow new (91 or 92) and I hope it can meet your requirements. If not, then I would like to hear about it.  Not sure if I understand the question NeoCSVReader. If that's still an issue, could you clarify?

Louis: no, please, don't modify base apps. 

Best,

Mariano

On Tue, Sep 22, 2020 at 10:48 AM Louis LaBrunda <[hidden email]> wrote:

Hi Joachim,

Perhaps you can modify #nextLine or extend the class with a new method that handles the embedded line end character.  perhaps something as simple as counting the double quotes and continuing to read if there is an odd number.

Lou

On Tuesday, September 22, 2020 at 2:51:12 AM UTC-4 [hidden email] wrote:

Thanks Lou,

I was somewhat assuming that if I set #lineDelimiter (a variable that our users can modify in the Applications' GUI), the Stream will only look for this one. The comment suggests I can set (or override) #shouldSearchForAllStandardDelimiters and combine the two somehow.

I will have to play with this....

BUT of course, Richard is bringing up a very valid point. We're using NeoCSVReader to parse CSV fields from these files. So the challenge here will be to find the right way to fiddle the line-end detection into that.... In the worst of cases, we'll have to do the "line reading" outside of NeoCSVReader and only feed each single line into the Reader instead of using #upToEnd....

Sigh. So much trouble for a single exception to a simple rule....

Anyways: Thanks Lou and Richard for your input. You showed me both a possible solution to take a closer look at as well as a very important caveat...

Joachim

[hidden email] schrieb am Montag, 21. September 2020 um 15:02:47 UTC+2:

Hi Joachim,

The #nextLine method of #CfsReadFileStream has the following comment:

nextLine

"Answer the elements between the current position and the next lineDelimiter. 

 

If #shouldSearchForAllStandardDelimiters answers false 

(likely, the user has specified an explicit line delimiter via #lineDelimiter:)

then we ONLY search for that delimiter. However, if it answers true

(likely the stream current delimiter is the default one), then we try to look

for any of the standard delimiters: Cr, Lf, and CrLf. This is useful when we are

reading streams that could have been written on different platforms.

Answers:

<ByteArray | String>"

It seems to me that covers you case.  I haven't checked but I guess other streams may have similar methods.

Lou

On Monday, September 21, 2020 at 4:02:17 AM UTC-4 [hidden email] wrote:

Hi there,

we're importing some data from csv files. There's a lot of good reasons not to do it, but anyways... ;-)

Just yesterday I got a complaint from a user who couldn't figure out how to import their data. It turns out their source system exports data with mixed CrLf and Lf line-ends.

I know there are quite a few things we could do here. Like pre-process the file, look fo the first occurence of either Cr, CrLf or Lf and make sure to unify them befor starting the CSV import. But maybe someone has an even easier way to do it, maybe even with on-board tools of VAST?

Joachim

--
You received this message because you are subscribed to the Google Groups "VA Smalltalk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/va-smalltalk/d6244a91-501c-461e-aa08-7a585b3d488dn%40googlegroups.com.

--

Mariano Martinez Peck
Software Engineer, Instantiations Inc.

Email: [hidden email]

Twitter: https://twitter.com/MartinezPeck

LinkedIn: www.linkedin.com/in/mariano-martinez-peck

Blog: https://marianopeck.wordpress.com/

--

You received this message because you are subscribed to a topic in the Google Groups "VA Smalltalk" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/va-smalltalk/pifWoYORMxY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/va-smalltalk/b2eafb48-bebf-454b-b67e-e355f492b314n%40googlegroups.com.

Hi Joachim, 

Happy yo read everything is working. I think you indeed needed to be smarter before and that's exactly why we improved this in 9.1. Just for the record, the developer case was:

"63547: Improve #nextLine to work with all known delimiters (Cr, Lf and CrLf)"

Cheers,

On Fri, Sep 25, 2020 at 3:36 AM Joachim Tuchel <[hidden email]> wrote:

Mariano,

Thanks. First of all, I can confirm #nextLine works as expected on teh example file that contained lines ending in both <cr><lf> and <lf>. So the key is not to set a lineDelimiter on the Stream.

To my surpise, NeoCSV also scans for cr/crlf/lf and does everything in the way it should.

The problem in this case is that I had implemented some clever code that first reads the first line of the File and tries to determine the lineDelimiter and then sets it on the input Stream. From then on, the Stream didn't handle any other lineDeliimiter than the one I had told it. So I was knocking out the cleverness of the Streams. I thought I was more clever. I learned something about this illusion today ;-)

Anyways: thank you Lou, Richard, Mariano for joining me in this journey and pushing me in the right direction.

Conclusion: PositionableStream, Cfs*FileStream and NeoCSVReader all handle line endings perfectly. If you expect Windows or Unix line endings and even a mix of these, just let them do their work and don't interfere. You're in the way if you try to be more clever ;-)

Joachim

[hidden email] schrieb am Donnerstag, 24. September 2020 um 22:00:01 UTC+2:

Hi guys, 

Joachim, yes, that behavior on #nextLine and #shouldSearchForAllStandardDelimiters is somehow new (91 or 92) and I hope it can meet your requirements. If not, then I would like to hear about it.  Not sure if I understand the question NeoCSVReader. If that's still an issue, could you clarify?

Louis: no, please, don't modify base apps. 

Best,

Mariano

On Tue, Sep 22, 2020 at 10:48 AM Louis LaBrunda <[hidden email]> wrote:

Hi Joachim,

Perhaps you can modify #nextLine or extend the class with a new method that handles the embedded line end character.  perhaps something as simple as counting the double quotes and continuing to read if there is an odd number.

Lou

On Tuesday, September 22, 2020 at 2:51:12 AM UTC-4 [hidden email] wrote:

Thanks Lou,

I was somewhat assuming that if I set #lineDelimiter (a variable that our users can modify in the Applications' GUI), the Stream will only look for this one. The comment suggests I can set (or override) #shouldSearchForAllStandardDelimiters and combine the two somehow.

I will have to play with this....

BUT of course, Richard is bringing up a very valid point. We're using NeoCSVReader to parse CSV fields from these files. So the challenge here will be to find the right way to fiddle the line-end detection into that.... In the worst of cases, we'll have to do the "line reading" outside of NeoCSVReader and only feed each single line into the Reader instead of using #upToEnd....

Sigh. So much trouble for a single exception to a simple rule....

Anyways: Thanks Lou and Richard for your input. You showed me both a possible solution to take a closer look at as well as a very important caveat...

Joachim

[hidden email] schrieb am Montag, 21. September 2020 um 15:02:47 UTC+2:

Hi Joachim,

The #nextLine method of #CfsReadFileStream has the following comment:

nextLine

"Answer the elements between the current position and the next lineDelimiter. 

 

If #shouldSearchForAllStandardDelimiters answers false 

(likely, the user has specified an explicit line delimiter via #lineDelimiter:)

then we ONLY search for that delimiter. However, if it answers true

(likely the stream current delimiter is the default one), then we try to look

for any of the standard delimiters: Cr, Lf, and CrLf. This is useful when we are

reading streams that could have been written on different platforms.

Answers:

<ByteArray | String>"

It seems to me that covers you case.  I haven't checked but I guess other streams may have similar methods.

Lou

On Monday, September 21, 2020 at 4:02:17 AM UTC-4 [hidden email] wrote:

Hi there,

we're importing some data from csv files. There's a lot of good reasons not to do it, but anyways... ;-)

Just yesterday I got a complaint from a user who couldn't figure out how to import their data. It turns out their source system exports data with mixed CrLf and Lf line-ends.

I know there are quite a few things we could do here. Like pre-process the file, look fo the first occurence of either Cr, CrLf or Lf and make sure to unify them befor starting the CSV import. But maybe someone has an even easier way to do it, maybe even with on-board tools of VAST?

Joachim

--
You received this message because you are subscribed to the Google Groups "VA Smalltalk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/va-smalltalk/d6244a91-501c-461e-aa08-7a585b3d488dn%40googlegroups.com.

--

Mariano Martinez Peck
Software Engineer, Instantiations Inc.

Email: [hidden email]

Twitter: https://twitter.com/MartinezPeck

LinkedIn: www.linkedin.com/in/mariano-martinez-peck

Blog: https://marianopeck.wordpress.com/

--
You received this message because you are subscribed to the Google Groups "VA Smalltalk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].

To view this discussion on the web visit https://groups.google.com/d/msgid/va-smalltalk/b2eafb48-bebf-454b-b67e-e355f492b314n%40googlegroups.com.

--

Mariano Martinez Peck
Software Engineer, Instantiations Inc.

Email: [hidden email]

Twitter: https://twitter.com/MartinezPeck

LinkedIn: www.linkedin.com/in/mariano-martinez-peck

Blog: https://marianopeck.wordpress.com/