Reading and writing C structs

An instrument I am working with creates a complicated binary file
(memory dumps of C structs). I would like to try reading the file into
Squeak for a cool new project. What's the best way to set up and read
C-based structs and then, once I'm done, to write them back out again
(including endianness, padding bytes and all the other nonsense)? I
could use pointers to related classes as well if they are
needed/useful for working with primitive types like this. An example
would be great.

I found that there are some interesting methods in PositionableStream
for reading and writing integer types from a binary stream. Is this
where I should start? What about floats and doubles? This seems a bit
low-level, but I could work with it. I'm a so-so C programmer with
experience in Python, but still learning Smalltalk.

David
_______________________________________________
Beginners mailing list
[hidden email]
http://lists.squeakfoundation.org/mailman/listinfo/beginners

On Fri, May 23, 2008 at 02:10:58PM -0700, David Finlayson wrote:
David,

Look at the Number class hierarchy (browse hierarchy) and look at the
"instance creation" category on the class side of each of these classes.
That will give you some good tips. If you are dealing with simple data
structures on a single machine architecture, this will not be too
difficult.

However:

The binary image of C data structures is inherently non-portable. The
compiler may pad data to different positions, the byte ordering may
vary on different hardware, and even simple data types can have different
sizes and byte orderings from one operating system to another. If you
are dealing with floating point data types, it's even trickier.

I don't know what instrument you are dealing with, but if there is
a way to get it to produce data in a more platform-independent manner,
that may save you some aggrivation in the long run.

Dave

_______________________________________________
Beginners mailing list
[hidden email]
http://lists.squeakfoundation.org/mailman/listinfo/beginners

On Fri, May 23, 2008 at 6:17 PM, David T. Lewis <[hidden email]> wrote:
>
> Look at the Number class hierarchy (browse hierarchy) and look at the
> "instance creation" category on the class side of each of these classes.
> That will give you some good tips. If you are dealing with simple data
> structures on a single machine architecture, this will not be too
> difficult.

I'll give it a look.

> The binary image of C data structures is inherently non-portable. The
> compiler may pad data to different positions, the byte ordering may
> vary on different hardware, and even simple data types can have different
> sizes and byte orderings from one operating system to another. If you
> are dealing with floating point data types, it's even trickier.
>

Yep, I know. And Smalltalk has reputation for not playing well with
others so there may be some friction here.

> I don't know what instrument you are dealing with, but if there is
> a way to get it to produce data in a more platform-independent manner,
> that may save you some aggrivation in the long run.
>

I work with off-the-shelf sonar systems and lots of ancillary
equipment such as sound velocimeters, RTK GPS, etc. I develop new
post-processing procedures, filters, visualization etc. and usually
code them up as Unix CLI programs in C. I don't have much control over
the design of the systems or the file formats they use. I've noticed
with multi-core machines that the real bottleneck now is I/O not CPU
time. So I thought, why not try a higher-level language than C and
boost my productivity? I am a scientist not a proffesional programmer,
but I've been coding all my life and coding is part of my job. That's
why I am looking into Smalltalk. Can't get much higher level than
this. Besides, I've always wanted to learn Smalltalk and/or Lisp.
_______________________________________________
Beginners mailing list
[hidden email]
http://lists.squeakfoundation.org/mailman/listinfo/beginners

On Fri, May 23, 2008 at 10:33:32PM -0700, David Finlayson wrote:
>
> I am a scientist not a proffesional programmer,
> but I've been coding all my life and coding is part of my job. That's
> why I am looking into Smalltalk. Can't get much higher level than
> this. Besides, I've always wanted to learn Smalltalk and/or Lisp.

Those sound like good enough reasons to me :)

In addition to the Simon Lewis book that you mentioned, I have found
"Smalltalk Best Practice Patterns" by Kent Beck to be something you
will want to read sooner rather than later. It's really worthwhile
and admirably brief (notwithstanding the title, there is none of the
nonsense that has been published as "patterns" elsewhere).

There is a good collection of free books to browse at
http://stephane.ducasse.free.fr/FreeBooks.html. "Smalltalk 80: The
Language" is still available in paperback, and is still worthwhile
today. "Squeak by Example" (http://squeakbyexample.org) is very good,
and covers contemporary Squeak.

Last but not least, be sure to read the "Back to the Future" paper
(http://users.ipa.net/~dwighth/squeak/oopsla_squeak.html) for an
appreciation of Squeak as both a high level and down-to-the-metal
environment.

Dave

_______________________________________________
Beginners mailing list
[hidden email]
http://lists.squeakfoundation.org/mailman/listinfo/beginners

>>>>> "David" == David T Lewis <[hidden email]> writes:

David> In addition to the Simon Lewis book that you mentioned, I have found
David> "Smalltalk Best Practice Patterns" by Kent Beck to be something you
David> will want to read sooner rather than later. It's really worthwhile
David> and admirably brief (notwithstanding the title, there is none of the
David> nonsense that has been published as "patterns" elsewhere).

I'll second, third, and fourth this.  Heck, reading that book improved my
*Perl* OO code.  I even wrote an entire half-day Perl course based on the
patterns in Beck's book.

--
Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095
<[hidden email]> <URL:http://www.stonehenge.com/merlyn/>
Smalltalk/Perl/Unix consulting, Technical writing, Comedy, etc. etc.
See http://methodsandmessages.vox.com/ for Smalltalk and Seaside discussion
_______________________________________________
Beginners mailing list
[hidden email]
http://lists.squeakfoundation.org/mailman/listinfo/beginners

OK, I've got an object that can read int and uint for 8, 16, 24 and 32
bit numbers from a binary file stream. But I cant figure out how to
read a float or double. What I want is something like this (these
don't work):

double
        | n |
        n := Float from: (aFileStream next: 32).
        ^n

float
        | n |
        n := Float from: (aFileStream next: 16).
        ^n

Any tips?

Thanks

david
_______________________________________________
Beginners mailing list
[hidden email]
http://lists.squeakfoundation.org/mailman/listinfo/beginners

Am 01.07.2008 um 07:27 schrieb David Finlayson:


Well, for one, C floats are 4 bytes and doubles are 8 bytes as defined  
by IEEE, so your code couldn't possibly work.

Secondly, Squeak Floats are represented as C doubles (inspect a Float,  
you will see two 32-bit words = 8 bytes), only in FloatArrays we have  
single-precision (32 bits = 4 bytes) floats that are converted to/from  
doubles on the fly when you access them.

You can see how to read/write Floats (doubles) in the readFloat:/
writeFloat: methods of DataStream. And to convert a Float into the bit  
pattern for singles there is Float>>asIEEE32BitWord and Float  
class>>fromIEEE32Bit:.

If speed was a concern you could use a FloatArray and set/get the 32  
bit word using #basicAt:/#basicAt:put: for about a 3x speed-up.

- Bert -


_______________________________________________
Beginners mailing list
[hidden email]
http://lists.squeakfoundation.org/mailman/listinfo/beginners

Thanks. The following methods work, can they be improved/simplified?

double
        " returns a double "
        | n a b |
        a := self uint32.
        b := self uint32.
        n := Float new: 2.
        n at: 1 put: b.
        n at: 2 put: a.
        ^n
float
        " returns a float "
        ^Float fromIEEE32Bit: self uint32.

uint32
        " Answer the next unsigned, 32-bit integer from this little endian
(binary) stream "
        ^ aFileStream nextLittleEndianNumber: 4
_______________________________________________
Beginners mailing list
[hidden email]
http://lists.squeakfoundation.org/mailman/listinfo/beginners

Yes, if you can load the contents on a ByteArray you can use the method:

array floatAt: position

It works just fine to read contents written from a C program, and I think it is a bit simpler.

Cheers.


r.





_______________________________________________
Beginners mailing list
[hidden email]
http://lists.squeakfoundation.org/mailman/listinfo/beginners

I need to read very large files so a single ByteArray holding all the
data wouldn't work (not until I get a 64-bit version of Squeak).
However, I could read in one or more structure at-a-time as a
ByteArray and then parse out the data I need, then read the next
sequence. This is kind-of like a buffered reading approach.

This raises a question about FileStreams, are they buffered?

David
_______________________________________________
Beginners mailing list
[hidden email]
http://lists.squeakfoundation.org/mailman/listinfo/beginners