File size limit
Locked
 
|
Hi
We are using VSE 3.0 for over 15 years and in this
period, files are getting bigger and bigger. We are unable to read files
larger than 4 gigs. At writing, there is no problem.
I know this is because of the "maxInteger"
value.
Is there a way to bypass that limit (except
splitting larger files in two) ?
Thanks
*** this signature added by listserv ***
*** Visit http://www.listserv.dfn.de/archives/vswe-l.html ***
*** for archive browsing and VSWE-L membership management ***
|
|
|
Hi!
The Win32 API does support 64 bit file pointers.
Currently the methods calling the API use zero for the
upper 32 bit. This limits the files to 4GB.
I think it should be possible to enhance the File
related classes to support the 64 bit file pointers. See: http://msdn.microsoft.com/en-us/library/windows/desktop/aa365541(v=vs.85).aspx Perhaps somebody already did this and is willing to
share it?
Regards,
Andreas
Andreas Rosenberg | eMail:
[hidden email] *** this signature added by listserv *** *** Visit http://www.listserv.dfn.de/archives/vswe-l.html *** *** for archive browsing and VSWE-L membership management *** |
|
|
Try
FileHandle >>position: anInteger
*** this signature added by listserv ***
*** Visit http://www.listserv.dfn.de/archives/vswe-l.html ***
*** for archive browsing and VSWE-L membership management ***
| large return | large := WinLargeInteger fromInteger: anInteger. return := KernelLibrary setFilePointer: self asParameter offset: (large bytesAtOffset: 0 count: 4) offsetHigh: (large bytesAtOffset: 4 count: 4) move: FileBegin. return = 16rFFFFFFFF ifTrue: [^self osError] On Mon, Feb 13, 2012 at 12:07 PM, Andreas Rosenberg <[hidden email]> wrote:
|
|
|
In reply to this post by Andreas Rosenberg
Thanks, i'll wait for an answer
*** this signature added by listserv *** *** Visit http://www.listserv.dfn.de/archives/vswe-l.html *** *** for archive browsing and VSWE-L membership management *** |
|
|
Be careful. I haven't tried the hint myself yet.
On Mon, Feb 13, 2012 at 1:06 PM, Chantal Thibodeau <[hidden email]> wrote:
*** this signature added by listserv *** *** Visit http://www.listserv.dfn.de/archives/vswe-l.html *** *** for archive browsing and VSWE-L membership management *** |
|
|
More info.
There are two additional methods you should also change: FileHandle >> #endByte | large success | large := WinLargeInteger new. success := KernelLibrary getFileSize: self asParameter fileSizeHigh: large. success ifFalse: [self osError]. ^large asUnsignedInteger KernelDLL >> #getFileSize: aFileHandle fileSizeHigh: aWinLargeInteger <api: GetFileSizeEx ulong struct boolean> ^self invalidArgument
On Mon, Feb 13, 2012 at 1:21 PM, Leandro Caniglia <[hidden email]> wrote: Be careful. I haven't tried the hint myself yet. |
|
|
Thanks, I'm looking into that (the first
modification wasn't enough).
I'm using a <nextLine> ou <upTo:> to
parse the file.
I will get back to you later this
afternoon.
And thanks again
*** this signature added by listserv *** *** Visit http://www.listserv.dfn.de/archives/vswe-l.html *** *** for archive browsing and VSWE-L membership management *** |
|
|
Hi, In addition to the methods already mentioned FileHandle>>endByte, FileHandle>>position:,
we have also changed the methods FileHandle>>position, FileHandle>>readInto:atPage:pageSize FileHandle>>writeFrom:toPage:for:pageSize
using the same type of enhancements as given by Leandro.
position "(Private) Private - Answer the current file position."
| pos high error | high := ExternalLong fromInteger: 0. ( pos := KernelLibrary setFilePointer: self asParameter offset: 0 offsetHigh: high asParameter move: FileCurrent ) = 16rFFFFFFFF ifTrue: [ #sdgHack. "^self osError" (error := self lastOsError) ~= 0 ifTrue: [ ^self osError: error in: #SetFilePointer ] ]. high asInteger = 0 ifTrue: [ "keine high-bytes" ^pos ]. ^(high asInteger bitShift: 32) + pos
readInto: aString atPage: anInteger pageSize: sizeInteger "(Private) Private - Read a page (sizeInteger bytes) or less (if at end of file) at page number anInteger from the receiver file into aString. Answer the number of bytes read. aString size must be >= sizeInteger."
| long | self position: anInteger * sizeInteger. long := ExternalLong fromInteger: 0. ( KernelLibrary readFile: self asParameter buffer: aString bytesToRead: sizeInteger bytesRead: long asParameter overlapped: nil ) ifFalse: [ #sdgHack. "^self osError" ^self osErrorIn: #ReadFile ]. ^long asInteger
writeFrom: aString toPage: anInteger for: size pageSize: sizeInteger "(Private) Private - Write size bytes of aString to the receiver file at page number anInteger. Page size is sizeInteger."
| long | self position: anInteger * sizeInteger. long := ExternalLong fromInteger: 0. ( KernelLibrary writeFile: self asParameter buffer: aString bytesToWrite: size bytesWritten: long asParameter overlapped: nil ) ifFalse: [ #sdgHack. "^self osError" ^self osErrorIn: #WriteFile ]
Manfred.
Von: Using Visual Smalltalk for Windows/Enterprise [mailto:[hidden email]]
Im Auftrag von Chantal Thibodeau
Thanks, I'm looking into that (the first modification wasn't enough).
I'm using a <nextLine> ou <upTo:> to parse the file.
I will get back to you later this afternoon.
And thanks again
*** this signature added by listserv *** *** Visit http://www.listserv.dfn.de/archives/vswe-l.html *** *** for archive browsing and VSWE-L membership management *** Treffen Sie AEB vom 13.-15. März 2012 auf der LogiMAT in Stuttgart. Halle 7, Stand 261. Vereinbaren Sie jetzt einen Termin und erhalten eine Eintrittskarte: www.aeb.de/logimat *** this signature added by listserv *** *** Visit http://www.listserv.dfn.de/archives/vswe-l.html *** *** for archive browsing and VSWE-L membership management *** |
|
|
Thanks Manfred!
I liked your catch of FileHandle>>#position and have a question: Is this method ever used?
*** this signature added by listserv ***
*** Visit http://www.listserv.dfn.de/archives/vswe-l.html ***
*** for archive browsing and VSWE-L membership management ***
/Leandro On Mon, Feb 13, 2012 at 6:17 PM, Möbus. Manfred <[hidden email]> wrote:
|
|
|
Hi Leandro, don’t know, guess we just scanned the class for methods using file pointers, and so fixed it.
Manfred
Von: Using Visual Smalltalk for Windows/Enterprise [mailto:[hidden email]]
Im Auftrag von Leandro Caniglia
Thanks Manfred!
I liked your catch of FileHandle>>#position and have a question: Is this method ever used?
/Leandro Treffen Sie AEB vom 13.-15. März 2012 auf der LogiMAT in Stuttgart. Halle 7, Stand 261. Vereinbaren Sie jetzt einen Termin und erhalten eine Eintrittskarte: www.aeb.de/logimat *** this signature added by listserv *** *** Visit http://www.listserv.dfn.de/archives/vswe-l.html *** *** for archive browsing and VSWE-L membership management *** |
|
|
Thank to you all guys.
I tried yesterday, with an additional modificaiton
to I don't remember wich method, but without your last methods.
I was able to read up to about 8 gigs, but no
more. I will try later today with your last modifications. But 8
gigs is already far better than 4 gigs.
I'll keep you informed.
*** this signature added by listserv *** *** Visit http://www.listserv.dfn.de/archives/vswe-l.html *** *** for archive browsing and VSWE-L membership management *** |
|
|
Chantal,
I've tested the code (mine + Manfred's) with 4, 8 and 12 GB and it worked just fine.
*** this signature added by listserv ***
*** Visit http://www.listserv.dfn.de/archives/vswe-l.html ***
*** for archive browsing and VSWE-L membership management ***
/Leandro On Tue, Feb 14, 2012 at 9:15 AM, Chantal Thibodeau <[hidden email]> wrote:
|
|
|
Here is the (approximate) code of my test
testLargeFile | filename long file a b pos | filename := 'largefile.bin'. long := (ExternalLong fromInteger: -1) asUnsignedInteger * 3. "12 GB" file := File newFile: filename. [ file position: long; nextPut: $a; nextPut: $b] ensure: [file close]. self assert: (File fromPath: filename) size = (long + 2). file := File pathNameReadOnly: filename. [ file position: long. a := file next. b := file next. pos := file file handle position] ensure: [ file close. File remove: filename]. self assert: a = $a; assert: b = $b; assert: pos = (long + 2) On Tue, Feb 14, 2012 at 9:52 AM, Leandro Caniglia <[hidden email]> wrote: Chantal, |
|
|
In reply to this post by Leandro Caniglia
Thanks, i'm trilled to test it
*** this signature added by listserv *** *** Visit http://www.listserv.dfn.de/archives/vswe-l.html *** *** for archive browsing and VSWE-L membership management *** |
|
|
In reply to this post by Leandro Caniglia
Happy Valentine's Day to you all, it works, it
works !
I was using just a <nextLine> all the way
through the file and I reach the last line.
Chantal
*** this signature added by listserv *** *** Visit http://www.listserv.dfn.de/archives/vswe-l.html *** *** for archive browsing and VSWE-L membership management *** |
|
|
Eureka!
On Tue, Feb 14, 2012 at 11:06 AM, Chantal Thibodeau <[hidden email]> wrote:
*** this signature added by listserv *** *** Visit http://www.listserv.dfn.de/archives/vswe-l.html *** *** for archive browsing and VSWE-L membership management *** |
|
|
In reply to this post by Leandro Caniglia
Hi everybody
Has someone already tried to read UTF-8 strings
?
My problem is we receive UTF-8 XML files, that are
then put in a MySQL database, in UTF8 format.
When I read them, all the accents are screwed
up. I looked on the net but I only find code for Squeak or other, and I am
unable to make it works.
Thanks
|
|
|
Hi:
Download http://www.infoil.com.ar/seaside/downloads/Seaside%203.0%20rc3%202010-NOV-06.zip Inside the zip file there is a folder "US" containing Dolphin Smalltalk source code to handle UTF-8 strings. Maybe you can port the code to VS. The implementation is based on standard windows api and standard Smalltalk so I guess is an easy job :) Best Regards Sebastian Calvo 2012/3/20 Chantal Thibodeau <[hidden email]>: > Hi everybody > > Has someone already tried to read UTF-8 strings ? > > My problem is we receive UTF-8 XML files, that are then put in a MySQL > database, in UTF8 format. > > When I read them, all the accents are screwed up. I looked on the net but I > only find code for Squeak or other, and I am unable to make it works. > > Thanks > > > *** this signature added by listserv *** *** Visit > http://www.listserv.dfn.de/archives/vswe-l.html *** *** for archive browsing > and VSWE-L membership management *** *** this signature added by listserv *** *** Visit http://www.listserv.dfn.de/archives/vswe-l.html *** *** for archive browsing and VSWE-L membership management *** |
|
|
In reply to this post by Chantal Thibodeau
Use MultiByteToWideChar with CP_UTF8 to convert the UTF-8 string to Unicode and then use WideCharToMultiByte to convert the unicode string the the codepage
of your SQL server (or leave it in unicode if you use the unicode fields in the database) From: Using Visual Smalltalk for Windows/Enterprise [mailto:[hidden email]]
On Behalf Of Chantal Thibodeau Hi everybody Has someone already tried to read UTF-8 strings ? My problem is we receive UTF-8 XML files, that are then put in a MySQL database, in UTF8 format. When I read them, all the accents are screwed up. I looked on the net but I only find code for Squeak or other, and I am unable to make it works. Thanks *** this signature added by listserv *** *** Visit
http://www.listserv.dfn.de/archives/vswe-l.html *** *** for archive browsing and VSWE-L membership management ***
|
|
|
In reply to this post by Chantal Thibodeau
If the accents are your only problem, it
sounds as if something is translating the text for you. Make sure
you have configured MySQL to accept and hold unicode data. If not,
the database itself will convert the text to latin and the accents are
lost. This link might help: http://dev.hubspot.com/bid/7049/MySQL-and-Unicode-Three-Gotchas
Also, since Windows 2000, Windows APIs use UTF16 (previously used UCS2), so as Todor pointed out you may need to convert your UTF8 to UTF16 if you want to call functions that expect Unicode parameters. Jon From: Chantal Thibodeau <[hidden email]> To: [hidden email], Date: 03/20/2012 03:54 PM Subject: UTF-8 Sent by: Using Visual Smalltalk for Windows/Enterprise <[hidden email]> Hi everybody Has someone already tried to read UTF-8 strings ? My problem is we receive UTF-8 XML files, that are then put in a MySQL database, in UTF8 format. When I read them, all the accents are screwed up. I looked on the net but I only find code for Squeak or other, and I am unable to make it works. Thanks *** this signature added by listserv *** *** Visit http://www.listserv.dfn.de/archives/vswe-l.html *** *** for archive browsing and VSWE-L membership management *** *** this signature added by listserv *** *** Visit http://www.listserv.dfn.de/archives/vswe-l.html *** *** for archive browsing and VSWE-L membership management *** |
| Free forum by Nabble | Edit this page |