shape file reading

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

shape file reading

Hans Baveco

Hi Hernan

 

I downloaded this shapefile directly via the browser. Indeed it won’t load, the type of shape file returned is

self longAt: 33 bigEndian: false -> 577660194

 

 

where it should have answered an Integer between 0 and 31

 

shapeType

         "position 32, Integer between 0 and 31, Little Endian"

 

         ^shpBytes longAt: 33 bigEndian: false

 

 

No idea what could be the cause – apparently the header of the file (the first 100 bytes) is not ok...

 

Hans

 

 

 

 

You mean some types are missing? Because I get this error when I try to read some SHP files: "Types > 20 are not yet accounted for"

 

I wrote this script to reproduce:

 

 | shpE legend urlRoot urlESRIFilePrefix urlESRIFileShp urlPath response fileRef |

" Download Shapefile resources "
urlRoot := 'https://github.com/nvkelso/natural-earth-vector/blob/master/110m_cultural/'.
urlESRIFilePrefix := 'ne_110m_populated_places'.
urlESRIFileShp := urlESRIFilePrefix , '.shp'.
urlPath := urlRoot , urlESRIFilePrefix.

#('.shx' '.dbf' '.shp') do: [ : ext |
| url |
url :=  (urlPath , ext , '?raw=true') asZnUrl.
(response := ZnEasy get: url) isSuccess
ifFalse: [ self error: 'Cannot download file' ].
fileRef := FileSystem disk workingDirectory / (urlESRIFilePrefix , ext).
fileRef writeStreamDo: [ : stream |
stream nextPutAll: ((ZnByteEncoder newForEncoding: 'koi8r') decodeBytes: response contents ) ] ].

 

Once downloaded:

 

| urlESRIFilePrefix urlESRIFileShp shpE |
urlESRIFilePrefix := 'ne_110m_populated_places'.
urlESRIFileShp := urlESRIFilePrefix , '.shp'.

" Load and display it in Morphic "
shpE := ShapeEnsemble fromFile: urlESRIFileShp.

 

Maybe it's related with the fact they are encoded as koi8r?

I checked with Zinc downloading first from the browser:

 

| shpFile |                                                                            
shpFile := 'ne_110m_admin_0_countries.shp' asFileReference.
[ shpFile binaryReadStreamDo: [ : in |
                (ZnCharacterEncoder detectEncoding: in upToEnd) identifier ] ]
        on: ZnCharacterEncodingError
        do: [ nil ].

 

 

Cheers,

 

Hernán

 

 

Reply | Threaded
Open this post in threaded view
|

Re: shape file reading

Hans Baveco

Well, it appears that simple direct downloading does not download the real shapefile. Taking care to actually get a shapefile, the result loads without problems.

ne_110m_populated_places.shp is a PointShape with 243 shapes (points). If you display it on a Form (#displayMorphic) you won’t see much (243 hardly visible dots).

 

With an application for vector graphics (Roassal?) that can read & display the vector data we would get much nicer maps...

 

Hans

 

From: Baveco, Hans <[hidden email]>
Sent: vrijdag 5 juni 2020 12:44
To: Any question about pharo is welcome <[hidden email]>
Subject: [Pharo-users] shape file reading

 

Hi Hernan

 

I downloaded this shapefile directly via the browser. Indeed it won’t load, the type of shape file returned is

self longAt: 33 bigEndian: false -> 577660194

 

 

where it should have answered an Integer between 0 and 31

 

shapeType

         "position 32, Integer between 0 and 31, Little Endian"

 

         ^shpBytes longAt: 33 bigEndian: false

 

 

No idea what could be the cause – apparently the header of the file (the first 100 bytes) is not ok...

 

Hans

 

 

 

 

You mean some types are missing? Because I get this error when I try to read some SHP files: "Types > 20 are not yet accounted for"

 

I wrote this script to reproduce:

 

 | shpE legend urlRoot urlESRIFilePrefix urlESRIFileShp urlPath response fileRef |

" Download Shapefile resources "
urlRoot := 'https://github.com/nvkelso/natural-earth-vector/blob/master/110m_cultural/'.
urlESRIFilePrefix := 'ne_110m_populated_places'.
urlESRIFileShp := urlESRIFilePrefix , '.shp'.
urlPath := urlRoot , urlESRIFilePrefix.

#('.shx' '.dbf' '.shp') do: [ : ext |
| url |
url :=  (urlPath , ext , '?raw=true') asZnUrl.
(response := ZnEasy get: url) isSuccess
ifFalse: [ self error: 'Cannot download file' ].
fileRef := FileSystem disk workingDirectory / (urlESRIFilePrefix , ext).
fileRef writeStreamDo: [ : stream |
stream nextPutAll: ((ZnByteEncoder newForEncoding: 'koi8r') decodeBytes: response contents ) ] ].

 

Once downloaded:

 

| urlESRIFilePrefix urlESRIFileShp shpE |
urlESRIFilePrefix := 'ne_110m_populated_places'.
urlESRIFileShp := urlESRIFilePrefix , '.shp'.

" Load and display it in Morphic "
shpE := ShapeEnsemble fromFile: urlESRIFileShp.

 

Maybe it's related with the fact they are encoded as koi8r?

I checked with Zinc downloading first from the browser:

 

| shpFile |                                                                            
shpFile := 'ne_110m_admin_0_countries.shp' asFileReference.
[ shpFile binaryReadStreamDo: [ : in |
                (ZnCharacterEncoder detectEncoding: in upToEnd) identifier ] ]
        on: ZnCharacterEncodingError
        do: [ nil ].

 

 

Cheers,

 

Hernán

 

 

Reply | Threaded
Open this post in threaded view
|

Re: shape file reading

Thierry Goubier
Hi Hans,

Roassal has support for loading OSM raw data and OSM tiles (map
backgrounds) (WGS84 projection). See
https://www.youtube.com/watch?v=6zoGIsQaGUo&t=429s. We've also
experimented with animations as well:
https://www.youtube.com/watch?v=sMB7DG-lbiU; we've also tried in the
past to work out of GRASS-produced rasters for complex geographic
analysis, such as runoff flooding, projected on osm tiles.

Regards,

Thierry

Le ven. 5 juin 2020 à 14:27, Baveco, Hans <[hidden email]> a écrit :

>
> Well, it appears that simple direct downloading does not download the real shapefile. Taking care to actually get a shapefile, the result loads without problems.
>
> ne_110m_populated_places.shp is a PointShape with 243 shapes (points). If you display it on a Form (#displayMorphic) you won’t see much (243 hardly visible dots).
>
>
>
> With an application for vector graphics (Roassal?) that can read & display the vector data we would get much nicer maps...
>
>
>
> Hans
>
>
>
> From: Baveco, Hans <[hidden email]>
> Sent: vrijdag 5 juni 2020 12:44
> To: Any question about pharo is welcome <[hidden email]>
> Subject: [Pharo-users] shape file reading
>
>
>
> Hi Hernan
>
>
>
> I downloaded this shapefile directly via the browser. Indeed it won’t load, the type of shape file returned is
>
> self longAt: 33 bigEndian: false -> 577660194
>
>
>
>
>
> where it should have answered an Integer between 0 and 31
>
>
>
> shapeType
>
>          "position 32, Integer between 0 and 31, Little Endian"
>
>
>
>          ^shpBytes longAt: 33 bigEndian: false
>
>
>
>
>
> No idea what could be the cause – apparently the header of the file (the first 100 bytes) is not ok...
>
>
>
> Hans
>
>
>
>
>
>
>
>
>
> You mean some types are missing? Because I get this error when I try to read some SHP files: "Types > 20 are not yet accounted for"
>
>
>
> I wrote this script to reproduce:
>
>
>
>  | shpE legend urlRoot urlESRIFilePrefix urlESRIFileShp urlPath response fileRef |
>
> " Download Shapefile resources "
> urlRoot := 'https://github.com/nvkelso/natural-earth-vector/blob/master/110m_cultural/'.
> urlESRIFilePrefix := 'ne_110m_populated_places'.
> urlESRIFileShp := urlESRIFilePrefix , '.shp'.
> urlPath := urlRoot , urlESRIFilePrefix.
>
> #('.shx' '.dbf' '.shp') do: [ : ext |
> | url |
> url :=  (urlPath , ext , '?raw=true') asZnUrl.
> (response := ZnEasy get: url) isSuccess
> ifFalse: [ self error: 'Cannot download file' ].
> fileRef := FileSystem disk workingDirectory / (urlESRIFilePrefix , ext).
> fileRef writeStreamDo: [ : stream |
> stream nextPutAll: ((ZnByteEncoder newForEncoding: 'koi8r') decodeBytes: response contents ) ] ].
>
>
>
> Once downloaded:
>
>
>
> | urlESRIFilePrefix urlESRIFileShp shpE |
> urlESRIFilePrefix := 'ne_110m_populated_places'.
> urlESRIFileShp := urlESRIFilePrefix , '.shp'.
>
> " Load and display it in Morphic "
> shpE := ShapeEnsemble fromFile: urlESRIFileShp.
>
>
>
> Maybe it's related with the fact they are encoded as koi8r?
>
> I checked with Zinc downloading first from the browser:
>
>
>
> | shpFile |
> shpFile := 'ne_110m_admin_0_countries.shp' asFileReference.
> [ shpFile binaryReadStreamDo: [ : in |
>                 (ZnCharacterEncoder detectEncoding: in upToEnd) identifier ] ]
>         on: ZnCharacterEncodingError
>         do: [ nil ].
>
>
>
>
>
> Cheers,
>
>
>
> Hernán
>
>
>
>

Reply | Threaded
Open this post in threaded view
|

Re: shape file reading

hernanmd
In reply to this post by Hans Baveco


El vie., 5 jun. 2020 a las 9:27, Baveco, Hans (<[hidden email]>) escribió:

Well, it appears that simple direct downloading does not download the real shapefile. Taking care to actually get a shapefile, the result loads without problems.

ne_110m_populated_places.shp is a PointShape with 243 shapes (points). If you display it on a Form (#displayMorphic) you won’t see much (243 hardly visible dots).

 


I checked and my previous download method was completely wrong. Now switching from ZnEasy to ZnClient to download and it works nicely:

 | shpE legend urlRoot urlESRIFilePrefix urlESRIFileShp urlPath response fileRef |

" Download Shapefile resources "
urlRoot := 'https://github.com/nvkelso/natural-earth-vector/blob/master/110m_cultural/'.
urlESRIFilePrefix := 'ne_110m_populated_places'.
urlESRIFileShp := urlESRIFilePrefix , '.shp'.
urlPath := urlRoot , urlESRIFilePrefix.

#('.shx' '.dbf' '.shp' '.cpg' '.prj') do: [ : ext |
  ZnClient new
   url: (urlPath , ext) asZnUrl;
   queryAt: 'raw' put: 'true';
   numberOfRetries: 2;
   enforceHttpSuccess: true;
   downloadTo: urlESRIFilePrefix , ext;
   get ].

" Load and display it in Morphic "
shpE := ShapeEnsemble fromFile: urlESRIFileShp.
" List data fields "
shpE dataFields inspect.
" List all shape records "
shpE shapeRecords inspect.
" Set the current attribute "
shpE attribute: 'NAME'.

legend := ColorLegend mapValuesToRandom: shpE valuesOfCurrentAttribute.
shpE legend: legend.

shpE displayMorphic.
 

With an application for vector graphics (Roassal?) that can read & display the vector data we would get much nicer maps...

 

Hans

 

From: Baveco, Hans <[hidden email]>
Sent: vrijdag 5 juni 2020 12:44
To: Any question about pharo is welcome <[hidden email]>
Subject: [Pharo-users] shape file reading

 

Hi Hernan

 

I downloaded this shapefile directly via the browser. Indeed it won’t load, the type of shape file returned is

self longAt: 33 bigEndian: false -> 577660194

 

 

where it should have answered an Integer between 0 and 31

 

shapeType

         "position 32, Integer between 0 and 31, Little Endian"

 

         ^shpBytes longAt: 33 bigEndian: false

 

 

No idea what could be the cause – apparently the header of the file (the first 100 bytes) is not ok...

 

Hans

 

 

 

 

You mean some types are missing? Because I get this error when I try to read some SHP files: "Types > 20 are not yet accounted for"

 

I wrote this script to reproduce:

 

 | shpE legend urlRoot urlESRIFilePrefix urlESRIFileShp urlPath response fileRef |

" Download Shapefile resources "
urlRoot := 'https://github.com/nvkelso/natural-earth-vector/blob/master/110m_cultural/'.
urlESRIFilePrefix := 'ne_110m_populated_places'.
urlESRIFileShp := urlESRIFilePrefix , '.shp'.
urlPath := urlRoot , urlESRIFilePrefix.

#('.shx' '.dbf' '.shp') do: [ : ext |
| url |
url :=  (urlPath , ext , '?raw=true') asZnUrl.
(response := ZnEasy get: url) isSuccess
ifFalse: [ self error: 'Cannot download file' ].
fileRef := FileSystem disk workingDirectory / (urlESRIFilePrefix , ext).
fileRef writeStreamDo: [ : stream |
stream nextPutAll: ((ZnByteEncoder newForEncoding: 'koi8r') decodeBytes: response contents ) ] ].

 

Once downloaded:

 

| urlESRIFilePrefix urlESRIFileShp shpE |
urlESRIFilePrefix := 'ne_110m_populated_places'.
urlESRIFileShp := urlESRIFilePrefix , '.shp'.

" Load and display it in Morphic "
shpE := ShapeEnsemble fromFile: urlESRIFileShp.

 

Maybe it's related with the fact they are encoded as koi8r?

I checked with Zinc downloading first from the browser:

 

| shpFile |                                                                            
shpFile := 'ne_110m_admin_0_countries.shp' asFileReference.
[ shpFile binaryReadStreamDo: [ : in |
                (ZnCharacterEncoder detectEncoding: in upToEnd) identifier ] ]
        on: ZnCharacterEncodingError
        do: [ nil ].

 

 

Cheers,

 

Hernán

 

 

Reply | Threaded
Open this post in threaded view
|

Re: shape file reading

Stéphane Ducasse
Please add this to the readme :)


On 5 Jun 2020, at 23:19, Hernán Morales Durand <[hidden email]> wrote:



El vie., 5 jun. 2020 a las 9:27, Baveco, Hans (<[hidden email]>) escribió:

Well, it appears that simple direct downloading does not download the real shapefile. Taking care to actually get a shapefile, the result loads without problems.

ne_110m_populated_places.shp is a PointShape with 243 shapes (points). If you display it on a Form (#displayMorphic) you won’t see much (243 hardly visible dots).

 


I checked and my previous download method was completely wrong. Now switching from ZnEasy to ZnClient to download and it works nicely:

 | shpE legend urlRoot urlESRIFilePrefix urlESRIFileShp urlPath response fileRef |

" Download Shapefile resources "
urlRoot := 'https://github.com/nvkelso/natural-earth-vector/blob/master/110m_cultural/'.
urlESRIFilePrefix := 'ne_110m_populated_places'.
urlESRIFileShp := urlESRIFilePrefix , '.shp'.
urlPath := urlRoot , urlESRIFilePrefix.

#('.shx' '.dbf' '.shp' '.cpg' '.prj') do: [ : ext |
  ZnClient new
   url: (urlPath , ext) asZnUrl;
   queryAt: 'raw' put: 'true';
   numberOfRetries: 2;
   enforceHttpSuccess: true;
   downloadTo: urlESRIFilePrefix , ext;
   get ].

" Load and display it in Morphic "
shpE := ShapeEnsemble fromFile: urlESRIFileShp.
" List data fields "
shpE dataFields inspect.
" List all shape records "
shpE shapeRecords inspect.
" Set the current attribute "
shpE attribute: 'NAME'.

legend := ColorLegend mapValuesToRandom: shpE valuesOfCurrentAttribute.
shpE legend: legend.

shpE displayMorphic.
 

With an application for vector graphics (Roassal?) that can read & display the vector data we would get much nicer maps...

 

Hans

 

From: Baveco, Hans <[hidden email]>
Sent: vrijdag 5 juni 2020 12:44
To: Any question about pharo is welcome <[hidden email]>
Subject: [Pharo-users] shape file reading

 

Hi Hernan

 

I downloaded this shapefile directly via the browser. Indeed it won’t load, the type of shape file returned is

self longAt: 33 bigEndian: false -> 577660194

 

 

where it should have answered an Integer between 0 and 31

 

shapeType

         "position 32, Integer between 0 and 31, Little Endian"

 

         ^shpBytes longAt: 33 bigEndian: false

 

 

No idea what could be the cause – apparently the header of the file (the first 100 bytes) is not ok...

 

Hans

 

 

 

 

You mean some types are missing? Because I get this error when I try to read some SHP files: "Types > 20 are not yet accounted for"

 

I wrote this script to reproduce:

 

 | shpE legend urlRoot urlESRIFilePrefix urlESRIFileShp urlPath response fileRef |

" Download Shapefile resources "
urlRoot := 'https://github.com/nvkelso/natural-earth-vector/blob/master/110m_cultural/'.
urlESRIFilePrefix := 'ne_110m_populated_places'.
urlESRIFileShp := urlESRIFilePrefix , '.shp'.
urlPath := urlRoot , urlESRIFilePrefix.

#('.shx' '.dbf' '.shp') do: [ : ext |
| url |
url :=  (urlPath , ext , '?raw=true') asZnUrl.
(response := ZnEasy get: url) isSuccess
ifFalse: [ self error: 'Cannot download file' ].
fileRef := FileSystem disk workingDirectory / (urlESRIFilePrefix , ext).
fileRef writeStreamDo: [ : stream |
stream nextPutAll: ((ZnByteEncoder newForEncoding: 'koi8r') decodeBytes: response contents ) ] ].

 

Once downloaded:

 

| urlESRIFilePrefix urlESRIFileShp shpE |
urlESRIFilePrefix := 'ne_110m_populated_places'.
urlESRIFileShp := urlESRIFilePrefix , '.shp'.

" Load and display it in Morphic "
shpE := ShapeEnsemble fromFile: urlESRIFileShp.

 

Maybe it's related with the fact they are encoded as koi8r?

I checked with Zinc downloading first from the browser:

 

| shpFile |                                                                            
shpFile := 'ne_110m_admin_0_countries.shp' asFileReference.
[ shpFile binaryReadStreamDo: [ : in |
                (ZnCharacterEncoder detectEncoding: in upToEnd) identifier ] ]
        on: ZnCharacterEncodingError
        do: [ nil ].

 

 

Cheers,

 

Hernán

 

 


--------------------------------------------
Stéphane Ducasse
03 59 35 87 52
Assistant: Aurore Dalle 
FAX 03 59 57 78 50
TEL 03 59 35 86 16
S. Ducasse - Inria
40, avenue Halley, 
Parc Scientifique de la Haute Borne, Bât.A, Park Plaza
Villeneuve d'Ascq 59650
France

Reply | Threaded
Open this post in threaded view
|

Re: shape file reading

hernanmd
Done!

Cheers

Hernán

El sáb., 6 jun. 2020 a las 3:17, Stéphane Ducasse (<[hidden email]>) escribió:
Please add this to the readme :)


On 5 Jun 2020, at 23:19, Hernán Morales Durand <[hidden email]> wrote:



El vie., 5 jun. 2020 a las 9:27, Baveco, Hans (<[hidden email]>) escribió:

Well, it appears that simple direct downloading does not download the real shapefile. Taking care to actually get a shapefile, the result loads without problems.

ne_110m_populated_places.shp is a PointShape with 243 shapes (points). If you display it on a Form (#displayMorphic) you won’t see much (243 hardly visible dots).

 


I checked and my previous download method was completely wrong. Now switching from ZnEasy to ZnClient to download and it works nicely:

 | shpE legend urlRoot urlESRIFilePrefix urlESRIFileShp urlPath response fileRef |

" Download Shapefile resources "
urlRoot := 'https://github.com/nvkelso/natural-earth-vector/blob/master/110m_cultural/'.
urlESRIFilePrefix := 'ne_110m_populated_places'.
urlESRIFileShp := urlESRIFilePrefix , '.shp'.
urlPath := urlRoot , urlESRIFilePrefix.

#('.shx' '.dbf' '.shp' '.cpg' '.prj') do: [ : ext |
  ZnClient new
   url: (urlPath , ext) asZnUrl;
   queryAt: 'raw' put: 'true';
   numberOfRetries: 2;
   enforceHttpSuccess: true;
   downloadTo: urlESRIFilePrefix , ext;
   get ].

" Load and display it in Morphic "
shpE := ShapeEnsemble fromFile: urlESRIFileShp.
" List data fields "
shpE dataFields inspect.
" List all shape records "
shpE shapeRecords inspect.
" Set the current attribute "
shpE attribute: 'NAME'.

legend := ColorLegend mapValuesToRandom: shpE valuesOfCurrentAttribute.
shpE legend: legend.

shpE displayMorphic.
 

With an application for vector graphics (Roassal?) that can read & display the vector data we would get much nicer maps...

 

Hans

 

From: Baveco, Hans <[hidden email]>
Sent: vrijdag 5 juni 2020 12:44
To: Any question about pharo is welcome <[hidden email]>
Subject: [Pharo-users] shape file reading

 

Hi Hernan

 

I downloaded this shapefile directly via the browser. Indeed it won’t load, the type of shape file returned is

self longAt: 33 bigEndian: false -> 577660194

 

 

where it should have answered an Integer between 0 and 31

 

shapeType

         "position 32, Integer between 0 and 31, Little Endian"

 

         ^shpBytes longAt: 33 bigEndian: false

 

 

No idea what could be the cause – apparently the header of the file (the first 100 bytes) is not ok...

 

Hans

 

 

 

 

You mean some types are missing? Because I get this error when I try to read some SHP files: "Types > 20 are not yet accounted for"

 

I wrote this script to reproduce:

 

 | shpE legend urlRoot urlESRIFilePrefix urlESRIFileShp urlPath response fileRef |

" Download Shapefile resources "
urlRoot := 'https://github.com/nvkelso/natural-earth-vector/blob/master/110m_cultural/'.
urlESRIFilePrefix := 'ne_110m_populated_places'.
urlESRIFileShp := urlESRIFilePrefix , '.shp'.
urlPath := urlRoot , urlESRIFilePrefix.

#('.shx' '.dbf' '.shp') do: [ : ext |
| url |
url :=  (urlPath , ext , '?raw=true') asZnUrl.
(response := ZnEasy get: url) isSuccess
ifFalse: [ self error: 'Cannot download file' ].
fileRef := FileSystem disk workingDirectory / (urlESRIFilePrefix , ext).
fileRef writeStreamDo: [ : stream |
stream nextPutAll: ((ZnByteEncoder newForEncoding: 'koi8r') decodeBytes: response contents ) ] ].

 

Once downloaded:

 

| urlESRIFilePrefix urlESRIFileShp shpE |
urlESRIFilePrefix := 'ne_110m_populated_places'.
urlESRIFileShp := urlESRIFilePrefix , '.shp'.

" Load and display it in Morphic "
shpE := ShapeEnsemble fromFile: urlESRIFileShp.

 

Maybe it's related with the fact they are encoded as koi8r?

I checked with Zinc downloading first from the browser:

 

| shpFile |                                                                            
shpFile := 'ne_110m_admin_0_countries.shp' asFileReference.
[ shpFile binaryReadStreamDo: [ : in |
                (ZnCharacterEncoder detectEncoding: in upToEnd) identifier ] ]
        on: ZnCharacterEncodingError
        do: [ nil ].

 

 

Cheers,

 

Hernán

 

 


--------------------------------------------
Stéphane Ducasse
03 59 35 87 52
Assistant: Aurore Dalle 
FAX 03 59 57 78 50
TEL 03 59 35 86 16
S. Ducasse - Inria
40, avenue Halley, 
Parc Scientifique de la Haute Borne, Bât.A, Park Plaza
Villeneuve d'Ascq 59650
France