NeoCSVReader and wrong number of fieldAccessors

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
25 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Re: NeoCSVReader and wrong number of fieldAccessors

Kasper Osterbye
As it happened, I ran into the exact same scenario as Joachim just the other day,
that is, the external provider of my csv had added some new columns. In my case
manifested itself in an error that an integer field was not an integer (because new
columns were added in the middle).

Reading through this whole thread leaves me with the feeling that no matter what Sven
adds, there is still a risk for error. Nevertheless, my suggestion would be to add a 
functionality to #skipHeaders, or make a sister method: 
#assertAndSkipHeaders: numberOfColumns onFailDo: aBlock given the actual number of headers
That would give me a way to handle the error up front. 

This will only be interesting if your data has headers of cause.

Thanks for NeoCSV which I use all the time!

Best,

Kasper 
Reply | Threaded
Open this post in threaded view
|

Re: NeoCSVReader and wrong number of fieldAccessors

jtuchel
Kasper,

I think this is somewhat close to another thing I am describing here: https://github.com/svenvc/NeoCSV/issues/20

The problem with extending NeoCSV endlessly is that some of the things we need with "real-life" CSV files is the fact that they are not CSV files at all, so it is hard to tell if it is a good idea to litter NeoCSV with methods for edge cases that have literally to do with the fact that part of our files are not CSV at all...

I somehow have the feeling some of what we need is a subclass of Stream that knows about constraints like "a line-break is only the end of a record if it is not part of a quoted field". So I am somewhat torn between wanting that stuff in NeoCSV and not wanting to mix csv parsing with handling of stupid ideas people have when exporting data in some CSV-like file.

Maybe it is a good idea to collect a few concepts that have been mentioned in these threads:
  • Sometimes we want to skip lines (header, footer) without interpreting their contents and without any side effects (not CSV-compliant)
  • Sometimes we want to ignore "additional data" after the end of a defined number of columns (not CSV-compliant)
  • Sometimes we need to know which line/column couldn't be parsed correctly (related to CSV and non-CSV)
    A plus would be if we could add the column name to the error message like in "The column 'amount' in line 34 cannot be interpreted as monetary amount" - but this is surely quite some work!
  • Sometimes we want to interpret columns by the column names in the header line (which may or may not be the first line of the file, only the former being CSV-compliant)

Of course this all doesn't mean I am not a fan of NeoCSV. It is well-written, works very well for "real" CSV and performs very well for my use cases. Most of the things we are talking about here are problems that arise when a CSV-file is not a CSV-file...


Joachim




Am 22.01.21 um 07:42 schrieb Kasper Osterbye:
As it happened, I ran into the exact same scenario as Joachim just the other day,
that is, the external provider of my csv had added some new columns. In my case
manifested itself in an error that an integer field was not an integer (because new
columns were added in the middle).

Reading through this whole thread leaves me with the feeling that no matter what Sven
adds, there is still a risk for error. Nevertheless, my suggestion would be to add a 
functionality to #skipHeaders, or make a sister method: 
#assertAndSkipHeaders: numberOfColumns onFailDo: aBlock given the actual number of headers
That would give me a way to handle the error up front. 

This will only be interesting if your data has headers of cause.

Thanks for NeoCSV which I use all the time!

Best,

Kasper 


-- 
-----------------------------------------------------------------------
Objektfabrik Joachim Tuchel          [hidden email]
Fliederweg 1                         http://www.objektfabrik.de
D-71640 Ludwigsburg                  http://joachimtuchel.wordpress.com
Telefon: +49 7141 56 10 86 0         Fax: +49 7141 56 10 86 1


Reply | Threaded
Open this post in threaded view
|

Re: NeoCSVReader and wrong number of fieldAccessors

Tim Mackinnon
In reply to this post by Kasper Osterbye
I’m not doing any CSV processing at the moment, but have in the past - so was interested in this thread.

@Kasper, can’t you just use #readHeader upfront, and do the assertion yourself, and then proceed to loop through your records? It would seem that the Neo caters for what you are suggesting - and if you want to add a helper method extension you have the building blocks to already do this?

The only flaw I can think of, is if there is no header present then I can’t recall what Neo does - ideally throws an exception so you can decide what to do - potentially continue if the number of columns is what you expect and the data matches the columns - or you fail with an error that a header is required. But I think you would always need to do some basic initial checks when processing CSV due to the nature of the format?

Tim

On Fri, 22 Jan 2021, at 6:42 AM, Kasper Osterbye wrote:
As it happened, I ran into the exact same scenario as Joachim just the other day,
that is, the external provider of my csv had added some new columns. In my case
manifested itself in an error that an integer field was not an integer (because new
columns were added in the middle).

Reading through this whole thread leaves me with the feeling that no matter what Sven
adds, there is still a risk for error. Nevertheless, my suggestion would be to add a 
functionality to #skipHeaders, or make a sister method: 
#assertAndSkipHeaders: numberOfColumns onFailDo: aBlock given the actual number of headers
That would give me a way to handle the error up front. 

This will only be interesting if your data has headers of cause.

Thanks for NeoCSV which I use all the time!

Best,

Kasper 

Reply | Threaded
Open this post in threaded view
|

Re: NeoCSVReader and wrong number of fieldAccessors

jtuchel
Tim,




Am 22.01.21 um 10:22 schrieb Tim Mackinnon:
I’m not doing any CSV processing at the moment, but have in the past - so was interested in this thread.

@Kasper, can’t you just use #readHeader upfront, and do the assertion yourself, and then proceed to loop through your records? It would seem that the Neo caters for what you are suggesting - and if you want to add a helper method extension you have the building blocks to already do this?

This is a good idea. One caveat, however: #readHeader in its current implementation does 2 things:

  • read the line respecting each field (thereby, respect line breaks within quoted fields - perfect for this purpose)
  • update the number of Columns for further reading (assuming #readHeader's purpose is to interpret the header line)
This second thing is in our way, because it may influence the way the following lines will be interpreted. That is ecactly why I created an issue on github (https://github.com/svenvc/NeoCSV/issues/20).
A method that reads a line without any side effects (other than pushing the position pointer forward to the next line) would come in handy for such scenarios. But you can always argue that this has nothing to do with CSV, because in CSV all lines have the same number of columns, each of them containing the same kind of information, and there may be exactly one header line. Anything else is just some file that may contain CSV-y stuff in it. So I am really not sure if NeoCSV should build lots of stuff for such files. I'd love to have this, but I'd understand if Sven refused to integrate it.... ;-)


The only flaw I can think of, is if there is no header present then I can’t recall what Neo does - ideally throws an exception so you can decide what to do - potentially continue if the number of columns is what you expect and the data matches the columns - or you fail with an error that a header is required. But I think you would always need to do some basic initial checks when processing CSV due to the nature of the format?

Right. You'd always have to write some specific logic for this particular file format and make NeoCSV ignore the right stuff...


Joachim




Tim

On Fri, 22 Jan 2021, at 6:42 AM, Kasper Osterbye wrote:
As it happened, I ran into the exact same scenario as Joachim just the other day,
that is, the external provider of my csv had added some new columns. In my case
manifested itself in an error that an integer field was not an integer (because new
columns were added in the middle).

Reading through this whole thread leaves me with the feeling that no matter what Sven
adds, there is still a risk for error. Nevertheless, my suggestion would be to add a 
functionality to #skipHeaders, or make a sister method: 
#assertAndSkipHeaders: numberOfColumns onFailDo: aBlock given the actual number of headers
That would give me a way to handle the error up front. 

This will only be interesting if your data has headers of cause.

Thanks for NeoCSV which I use all the time!

Best,

Kasper 


-- 
-----------------------------------------------------------------------
Objektfabrik Joachim Tuchel          [hidden email]
Fliederweg 1                         http://www.objektfabrik.de
D-71640 Ludwigsburg                  http://joachimtuchel.wordpress.com
Telefon: +49 7141 56 10 86 0         Fax: +49 7141 56 10 86 1


Reply | Threaded
Open this post in threaded view
|

Re: NeoCSVReader and wrong number of fieldAccessors

Sven Van Caekenberghe-2
There is now the following commit:

https://github.com/svenvc/NeoCSV/commit/0acc2270b382f52533c478f2f1585341e390d4b5

which should address a couple of issues.

> On 22 Jan 2021, at 12:15, [hidden email] wrote:
>
> Tim,
>
>
>
>
> Am 22.01.21 um 10:22 schrieb Tim Mackinnon:
>> I’m not doing any CSV processing at the moment, but have in the past - so was interested in this thread.
>>
>> @Kasper, can’t you just use #readHeader upfront, and do the assertion yourself, and then proceed to loop through your records? It would seem that the Neo caters for what you are suggesting - and if you want to add a helper method extension you have the building blocks to already do this?
>>
> This is a good idea. One caveat, however: #readHeader in its current implementation does 2 things:
>
> • read the line respecting each field (thereby, respect line breaks within quoted fields - perfect for this purpose)
> • update the number of Columns for further reading (assuming #readHeader's purpose is to interpret the header line)
> This second thing is in our way, because it may influence the way the following lines will be interpreted. That is ecactly why I created an issue on github (https://github.com/svenvc/NeoCSV/issues/20).
> A method that reads a line without any side effects (other than pushing the position pointer forward to the next line) would come in handy for such scenarios. But you can always argue that this has nothing to do with CSV, because in CSV all lines have the same number of columns, each of them containing the same kind of information, and there may be exactly one header line. Anything else is just some file that may contain CSV-y stuff in it. So I am really not sure if NeoCSV should build lots of stuff for such files. I'd love to have this, but I'd understand if Sven refused to integrate it.... ;-)
>
>
>> The only flaw I can think of, is if there is no header present then I can’t recall what Neo does - ideally throws an exception so you can decide what to do - potentially continue if the number of columns is what you expect and the data matches the columns - or you fail with an error that a header is required. But I think you would always need to do some basic initial checks when processing CSV due to the nature of the format?
> Right. You'd always have to write some specific logic for this particular file format and make NeoCSV ignore the right stuff...
>
>
>
> Joachim
>
>
>
>
>
>>
>> Tim
>>
>> On Fri, 22 Jan 2021, at 6:42 AM, Kasper Osterbye wrote:
>>> As it happened, I ran into the exact same scenario as Joachim just the other day,
>>> that is, the external provider of my csv had added some new columns. In my case
>>> manifested itself in an error that an integer field was not an integer (because new
>>> columns were added in the middle).
>>>
>>> Reading through this whole thread leaves me with the feeling that no matter what Sven
>>> adds, there is still a risk for error. Nevertheless, my suggestion would be to add a
>>> functionality to #skipHeaders, or make a sister method:
>>> #assertAndSkipHeaders: numberOfColumns onFailDo: aBlock given the actual number of headers
>>> That would give me a way to handle the error up front.
>>>
>>> This will only be interesting if your data has headers of cause.
>>>
>>> Thanks for NeoCSV which I use all the time!
>>>
>>> Best,
>>>
>>> Kasper
>>
>
>
> --
> -----------------------------------------------------------------------
> Objektfabrik Joachim Tuchel          
> mailto:[hidden email]
>
> Fliederweg 1                        
> http://www.objektfabrik.de
>
> D-71640 Ludwigsburg                  
> http://joachimtuchel.wordpress.com
>
> Telefon: +49 7141 56 10 86 0         Fax: +49 7141 56 10 86 1
>
>
>
>
12