Automation of MS Office from Pharo

The install instructions are not the same for Pharo 7, or pharo 8/9

If you are using pharo 7, you have to install a different branch of the project. Check in the documentation of pharo-com(the read me in the same repo) 

 

On Sun, Apr 5, 2020, 21:41 PBKResearch <[hidden email]> wrote:

Hello Pablo

Many thanks for the suggestion. I have tried to use Pharo-Com to connect to Outlook, and have now got myself into a complete tangle. I have been able to connect to my already-running copy of Outlook, but when I tried to enter propertyNamed: 'Folders', an error message appeared with the text: "Instance of FFIExternalObjectType did not understand emitArgument:context:inCallout:".
I could not understand this, so I closed the debugger, assuming the effects would be unwound.

I found that, whatever I tried after that, the same message appeared. In desperation I closed down Pharo, without saving the image, and started again. Whatever I tried in using Pharo-Com, the exact same message appeared. Even when I tried running one of your tests, the same message.

It seems clear that something in my system, not part of Pharo, has been changed in a way which persists from one run of Pharo to another. I have no idea what it can be, but I see that some parts of Pharo-Com address the registry, so I wonder if I have corrupted that in some way.

For the time being I have stopped work on Pharo-Com. I hope that a night's sleep will bring inspiration. In case it means anything to you, I attach a screenshot of the error stack when I try to run the testWord case in PharoCOM-Tests.

Thanks again

Peter Kenny

-----Original Message-----
From: Pharo-users <[hidden email]> On Behalf Of [hidden email]
Sent: 05 April 2020 16:23
To: Any question about pharo is welcome <[hidden email]>
Subject: Re: [Pharo-users] Automation of MS Office from Pharo

Hello,
   One of the alternatives to integrate with Office is the use of the COM interface. There is an implementation in https://github.com/tesonep/pharo-com. In the test package, there are examples to integrate with Word, Excell, Internet Explorer, and Access. Integration with Outlook can be done in the same fashion. It is good that the API of all the Office products is heavily documented.

You can get more information about how to use Outlook here:
https://docs.microsoft.com/en-us/office/vba/api/overview/outlook/object-model

If you have any questions, please let me know.
Cheers,
Pablo.

On Sun, Apr 5, 2020 at 2:44 PM PBKResearch <[hidden email]> wrote:

>
> Hello All
>
>
>
> I am using Pharo (currently P7, moving to P8) under Windows 10. I have MS Office 365 on my system, and use it regularly for non-Smalltalk tasks. It would be very convenient if I could issue commands to MS Office components from within Pharo. My current interest is to manage my incoming e-mails in Outlook, but I can see many other applications. I have a recollection of seeing a reference to automation of Excel to transfer data both ways, but I cannot recall where. I have searched the catalog, but could not see anything that looked relevant. I have also read the uFFI booklet, but again no luck. Could anyone point me to any information or examples of such control, please? It would not have to be Outlook, though that is my main interest; given examples for other Office components, I should be able to follow the pattern.
>
>
>
> TIA
>
>
>
> Peter Kenny

--
Pablo Tesone.
[hidden email]

Hello Peter,

If you look at the code in the Word test you will notice that the test firstly

- creates a new Word instance, 

- makes it visible to the end user, 

- then adds an empty document to the documents collection with the text "Hello from Pharo!"

- then it tests whether it can receive the same text back from Word.

After that, the test

- adds a new empty document to the documents collection with the text "Hello from Pharo! Some additional text. ", this time as an array of two texts

- it activates this second document (this imitates the end user's window activation on the desktop)

- then it tests whether it can receive the same text back from Word.

If you look at the Task Manager, you'll notice that you have one Word process with two open documents:

Namely, for each document Word creates a new, separate window - the documents are not displayed in one "Word application window", but separately - that's a normal behavior for some versions of MS Office, and it happens also if you open several documents directly in Word. So, there is just one Word instance.

'finalize' clears the references to the Word instance, it doesn't close the program by itself. If you want to do that, you can send Quit message to Word before you destroy the reference (https://docs.microsoft.com/en-us/office/vba/api/word.application.quit(method)). 

Similar behaviour is with Outlook, here's one example: https://www.excelcommand.com/excel-help/excel-how-to.php?i=124116

The calling among COM objects is asynchronous, and it's usually wise to wrap it in error handling structures.

Please tell us how it goes.

Best wishes,

Tomaz

------ Original Message ------

From: "PBKResearch" <[hidden email]>

To: "Any question about pharo is welcome" <[hidden email]>

Sent: 5.4.2020 23:18:02

Subject: Re: [Pharo-users] Automation of MS Office from Pharo

Pablo - a final update before I close for the night. The Word test on the pharo 8 version comes up green. The strange error message is nowhere to be seen in any Pharo 8 runs. The result is not what I expected; I finish up with two Word documents open, one with the first message, the other with the two messages. I thought the 'finalize' command would close it down.

 

Anyway, it looks as if I need to switch to P8 to use Pharo-Com. I shall continue testing tomorrow on P8.

 

Sorry for the late-night hassle.

 

Peter

 

Hi Peter,

Hello Tomaž

 

Many thanks for your patient explanation. I should have been able to work out for myself how the Word test works, and indeed I realized some of it when I came to shut down the Word instance and its two documents. But it was late at night, and I should have packed up before then.

 

My last test last night showed that basically I have cracked it for my job of automating Outlook. I have been able to connect to my running instance of Outlook, open my application and interrogate the names of my top-level folders. From now on it should be just a matter of understanding the MS documentation of the Outlook model.

 

However, all this is with Pharo-Com installed in a new Pharo 8 image. I have no idea what went wrong with my first effort on Pharo 7. But I shan’t worry about that – I shall gradually move all my bits and pieces to P8. I shall try to work it out myself from here, but I shall come back if I get stuck.

 

Thanks again

 

Peter Kenny

 

From: Pharo-users <[hidden email]> On Behalf Of Tomaž Turk
Sent: 06 April 2020 08:04
To: Any question about pharo is welcome <[hidden email]>
Subject: Re: [Pharo-users] Automation of MS Office from Pharo

 

Hello Peter,

 

If you look at the code in the Word test you will notice that the test firstly

- creates a new Word instance, 

- makes it visible to the end user, 

- then adds an empty document to the documents collection with the text "Hello from Pharo!"

- then it tests whether it can receive the same text back from Word.

 

After that, the test

- adds a new empty document to the documents collection with the text "Hello from Pharo! Some additional text. ", this time as an array of two texts

- it activates this second document (this imitates the end user's window activation on the desktop)

- then it tests whether it can receive the same text back from Word.

 

If you look at the Task Manager, you'll notice that you have one Word process with two open documents:

 

 

Namely, for each document Word creates a new, separate window - the documents are not displayed in one "Word application window", but separately - that's a normal behavior for some versions of MS Office, and it happens also if you open several documents directly in Word. So, there is just one Word instance.

 

'finalize' clears the references to the Word instance, it doesn't close the program by itself. If you want to do that, you can send Quit message to Word before you destroy the reference (https://docs.microsoft.com/en-us/office/vba/api/word.application.quit(method)). 

 

Similar behaviour is with Outlook, here's one example: https://www.excelcommand.com/excel-help/excel-how-to.php?i=124116

 

The calling among COM objects is asynchronous, and it's usually wise to wrap it in error handling structures.

 

Please tell us how it goes.

 

Best wishes,

Tomaz

 

 

 

 

 

 

 

 

------ Original Message ------

From: "PBKResearch" <[hidden email]>

To: "Any question about pharo is welcome" <[hidden email]>

Sent: 5.4.2020 23:18:02

Subject: Re: [Pharo-users] Automation of MS Office from Pharo

 

Pablo - a final update before I close for the night. The Word test on the pharo 8 version comes up green. The strange error message is nowhere to be seen in any Pharo 8 runs. The result is not what I expected; I finish up with two Word documents open, one with the first message, the other with the two messages. I thought the 'finalize' command would close it down.

 

Anyway, it looks as if I need to switch to P8 to use Pharo-Com. I shall continue testing tomorrow on P8.

 

Sorry for the late-night hassle.

 

Peter

 

Pablo Tesone.
[hidden email]

Dear Peter,

I'm glad that you find the package useful! Can you please explain a little bit more about your code, because it's not obvious from your description at which stage the problem occurs. Could you trace it with the debugger, and/or possibly post here the piece of code that fails? 

Thanks and best wishes,

Tomaz

------ Original Message ------

From: "PBKResearch" <[hidden email]>

To: "Any question about pharo is welcome" <[hidden email]>

Sent: 6. 04. 2020 19:15:16

Subject: Re: [Pharo-users] Automation of MS Office from Pharo

Hello Pablo (and Tomaž)

 

You asked for feedback; after a day of experiments, I have good news and bad news.

 

The good news is that I have been able to navigate through my Outlook Data File, find my way to an e-mail of interest and examine the fields of interest. This means I am 90% of the way to my objective. All I want is to get the HTML code from the message body and pass it to XMLHTMLParser for further analysis. I found it not too difficult to work this out; the fact that a COMDispatchInstance can display all its internals in a playground page meant that I could work it out interactively as a playground exercise. I am still discovering things about the interface; I have only just found out that I can select an item from a list by quoting its name as an argument, rather than its number in the list. I am sure my clumsy code can be tidied up still.

 

I can get to the 100% point in the job by exporting the contents of the e-mail to disk in HTML format, using the SaveAs procedure of the MailItem object. I can then read it back from disk into the parser. But this seems a bit clunky. The content of an HTML mail item is available in the HTMLBody property of the MailItem, so I should be able to pass it programmatically to the parser without going near the disk. This is where I run into trouble.

 

In most of the cases I tried, this worked fine; I could display the HTML body text as a WideString in the playground, assign it to a variable and do whatever I wanted. However, for just one e-mail I tried, as soon as I selected ‘Do it and go’ in the playground, a message came up that the VM had crashed; the application closed, leaving only a crash dump. I cannot find anything unusual about the message that failed in this way, except that when I save it, it produces a larger file than any of the others (just over 1MB, against up to 300KB for the ones that worked). Could there be a limit on the size of some internal buffer?

 

I thought it worth while mentioning this now, because crashing the VM is generally undesirable. I can solve my problems safely using the SaveAs route, so it’s not a problem for me.

 

Hope this helps; if you want more details, let me know. Overall I am very happy with this library.

 

Peter Kenny

 

From: Pharo-users <[hidden email]> On Behalf Of [hidden email]
Sent: 06 April 2020 11:24
To: Any question about pharo is welcome <[hidden email]>
Subject: Re: [Pharo-users] Automation of MS Office from Pharo

 

Hi Peter,

    First, thanks to try to use Pharo-COM, that is great, I love to have users for it and find it is useful. 

Secondly, the Pharo 7 problem is an error I have introduced. It is clear that some changes I have done to support the new version of UFFI (the framework in Pharo to handle FFI calls) have broken the Pharo 7 version, so I will fix it to maintain compatibility.

 

It is great that you were able to make it work!

 

Cheers,

 

 

On Mon, Apr 6, 2020 at 12:18 PM PBKResearch <[hidden email]> wrote:

Hello Tomaž

 

Many thanks for your patient explanation. I should have been able to work out for myself how the Word test works, and indeed I realized some of it when I came to shut down the Word instance and its two documents. But it was late at night, and I should have packed up before then.

 

My last test last night showed that basically I have cracked it for my job of automating Outlook. I have been able to connect to my running instance of Outlook, open my application and interrogate the names of my top-level folders. From now on it should be just a matter of understanding the MS documentation of the Outlook model.

 

However, all this is with Pharo-Com installed in a new Pharo 8 image. I have no idea what went wrong with my first effort on Pharo 7. But I shan’t worry about that – I shall gradually move all my bits and pieces to P8. I shall try to work it out myself from here, but I shall come back if I get stuck.

 

Thanks again

 

Peter Kenny

 

From: Pharo-users <[hidden email]> On Behalf Of Tomaž Turk
Sent: 06 April 2020 08:04
To: Any question about pharo is welcome <[hidden email]>
Subject: Re: [Pharo-users] Automation of MS Office from Pharo

 

Hello Peter,

 

If you look at the code in the Word test you will notice that the test firstly

- creates a new Word instance, 

- makes it visible to the end user, 

- then adds an empty document to the documents collection with the text "Hello from Pharo!"

- then it tests whether it can receive the same text back from Word.

 

After that, the test

- adds a new empty document to the documents collection with the text "Hello from Pharo! Some additional text. ", this time as an array of two texts

- it activates this second document (this imitates the end user's window activation on the desktop)

- then it tests whether it can receive the same text back from Word.

 

If you look at the Task Manager, you'll notice that you have one Word process with two open documents:

 

 

Namely, for each document Word creates a new, separate window - the documents are not displayed in one "Word application window", but separately - that's a normal behavior for some versions of MS Office, and it happens also if you open several documents directly in Word. So, there is just one Word instance.

 

'finalize' clears the references to the Word instance, it doesn't close the program by itself. If you want to do that, you can send Quit message to Word before you destroy the reference (https://docs.microsoft.com/en-us/office/vba/api/word.application.quit(method)). 

 

Similar behaviour is with Outlook, here's one example: https://www.excelcommand.com/excel-help/excel-how-to.php?i=124116

 

The calling among COM objects is asynchronous, and it's usually wise to wrap it in error handling structures.

 

Please tell us how it goes.

 

Best wishes,

Tomaz

 

 

 

 

 

 

 

 

------ Original Message ------

From: "PBKResearch" <[hidden email]>

To: "Any question about pharo is welcome" <[hidden email]>

Sent: 5.4.2020 23:18:02

Subject: Re: [Pharo-users] Automation of MS Office from Pharo

 

Pablo - a final update before I close for the night. The Word test on the pharo 8 version comes up green. The strange error message is nowhere to be seen in any Pharo 8 runs. The result is not what I expected; I finish up with two Word documents open, one with the first message, the other with the two messages. I thought the 'finalize' command would close it down.

 

Anyway, it looks as if I need to switch to P8 to use Pharo-Com. I shall continue testing tomorrow on P8.

 

Sorry for the late-night hassle.

 

Peter

 

 

--

Pablo Tesone.
[hidden email]

Dear Peter,

I'm glad that you find the package useful! Can you please explain a little bit more about your code, because it's not obvious from your description at which stage the problem occurs. Could you trace it with the debugger, and/or possibly post here the piece of code that fails? 

Thanks and best wishes,

Tomaz

------ Original Message ------

From: "PBKResearch" <[hidden email]>

To: "Any question about pharo is welcome" <[hidden email]>

Sent: 6. 04. 2020 19:15:16

Subject: Re: [Pharo-users] Automation of MS Office from Pharo

Hello Pablo (and Tomaž)

 

You asked for feedback; after a day of experiments, I have good news and bad news.

 

The good news is that I have been able to navigate through my Outlook Data File, find my way to an e-mail of interest and examine the fields of interest. This means I am 90% of the way to my objective. All I want is to get the HTML code from the message body and pass it to XMLHTMLParser for further analysis. I found it not too difficult to work this out; the fact that a COMDispatchInstance can display all its internals in a playground page meant that I could work it out interactively as a playground exercise. I am still discovering things about the interface; I have only just found out that I can select an item from a list by quoting its name as an argument, rather than its number in the list. I am sure my clumsy code can be tidied up still.

 

I can get to the 100% point in the job by exporting the contents of the e-mail to disk in HTML format, using the SaveAs procedure of the MailItem object. I can then read it back from disk into the parser. But this seems a bit clunky. The content of an HTML mail item is available in the HTMLBody property of the MailItem, so I should be able to pass it programmatically to the parser without going near the disk. This is where I run into trouble.

 

In most of the cases I tried, this worked fine; I could display the HTML body text as a WideString in the playground, assign it to a variable and do whatever I wanted. However, for just one e-mail I tried, as soon as I selected ‘Do it and go’ in the playground, a message came up that the VM had crashed; the application closed, leaving only a crash dump. I cannot find anything unusual about the message that failed in this way, except that when I save it, it produces a larger file than any of the others (just over 1MB, against up to 300KB for the ones that worked). Could there be a limit on the size of some internal buffer?

 

I thought it worth while mentioning this now, because crashing the VM is generally undesirable. I can solve my problems safely using the SaveAs route, so it’s not a problem for me.

 

Hope this helps; if you want more details, let me know. Overall I am very happy with this library.

 

Peter Kenny

 

From: Pharo-users <[hidden email]> On Behalf Of [hidden email]
Sent: 06 April 2020 11:24
To: Any question about pharo is welcome <[hidden email]>
Subject: Re: [Pharo-users] Automation of MS Office from Pharo

 

Hi Peter,

    First, thanks to try to use Pharo-COM, that is great, I love to have users for it and find it is useful. 

Secondly, the Pharo 7 problem is an error I have introduced. It is clear that some changes I have done to support the new version of UFFI (the framework in Pharo to handle FFI calls) have broken the Pharo 7 version, so I will fix it to maintain compatibility.

 

It is great that you were able to make it work!

 

Cheers,

 

 

On Mon, Apr 6, 2020 at 12:18 PM PBKResearch <[hidden email]> wrote:

Hello Tomaž

 

Many thanks for your patient explanation. I should have been able to work out for myself how the Word test works, and indeed I realized some of it when I came to shut down the Word instance and its two documents. But it was late at night, and I should have packed up before then.

 

My last test last night showed that basically I have cracked it for my job of automating Outlook. I have been able to connect to my running instance of Outlook, open my application and interrogate the names of my top-level folders. From now on it should be just a matter of understanding the MS documentation of the Outlook model.

 

However, all this is with Pharo-Com installed in a new Pharo 8 image. I have no idea what went wrong with my first effort on Pharo 7. But I shan’t worry about that – I shall gradually move all my bits and pieces to P8. I shall try to work it out myself from here, but I shall come back if I get stuck.

 

Thanks again

 

Peter Kenny

 

From: Pharo-users <[hidden email]> On Behalf Of Tomaž Turk
Sent: 06 April 2020 08:04
To: Any question about pharo is welcome <[hidden email]>
Subject: Re: [Pharo-users] Automation of MS Office from Pharo

 

Hello Peter,

 

If you look at the code in the Word test you will notice that the test firstly

- creates a new Word instance, 

- makes it visible to the end user, 

- then adds an empty document to the documents collection with the text "Hello from Pharo!"

- then it tests whether it can receive the same text back from Word.

 

After that, the test

- adds a new empty document to the documents collection with the text "Hello from Pharo! Some additional text. ", this time as an array of two texts

- it activates this second document (this imitates the end user's window activation on the desktop)

- then it tests whether it can receive the same text back from Word.

 

If you look at the Task Manager, you'll notice that you have one Word process with two open documents:

 

 

Namely, for each document Word creates a new, separate window - the documents are not displayed in one "Word application window", but separately - that's a normal behavior for some versions of MS Office, and it happens also if you open several documents directly in Word. So, there is just one Word instance.

 

'finalize' clears the references to the Word instance, it doesn't close the program by itself. If you want to do that, you can send Quit message to Word before you destroy the reference (https://docs.microsoft.com/en-us/office/vba/api/word.application.quit(method)). 

 

Similar behaviour is with Outlook, here's one example: https://www.excelcommand.com/excel-help/excel-how-to.php?i=124116

 

The calling among COM objects is asynchronous, and it's usually wise to wrap it in error handling structures.

 

Please tell us how it goes.

 

Best wishes,

Tomaz

 

 

 

 

 

 

 

 

------ Original Message ------

From: "PBKResearch" <[hidden email]>

To: "Any question about pharo is welcome" <[hidden email]>

Sent: 5.4.2020 23:18:02

Subject: Re: [Pharo-users] Automation of MS Office from Pharo

 

Pablo - a final update before I close for the night. The Word test on the pharo 8 version comes up green. The strange error message is nowhere to be seen in any Pharo 8 runs. The result is not what I expected; I finish up with two Word documents open, one with the first message, the other with the two messages. I thought the 'finalize' command would close it down.

 

Anyway, it looks as if I need to switch to P8 to use Pharo-Com. I shall continue testing tomorrow on P8.

 

Sorry for the late-night hassle.

 

Peter

 

 

--

Pablo Tesone.
[hidden email]

Thanks, Peter. I checked out the Outlook COM interface, noticed that the MailItem.HTMLBody property returns a string, which is implemented as BSTR in COM (more on this here: https://bytecomb.com/vba-internals-string-variables-and-pointers-in-depth/).

I tested the maximum lengths of strings with Word and my images crash under these circumstances:

- Pharo 8.0 64 bit when returning ByteString is longer than 16.379 characters

- Pharo 8.0 32 bit when returning ByteString is longer than 260.080 characters (approximately).

64 bit image crashes without any warning, however the 32 bit one crashes with

I presume this is the same behaviour as you report, Peter. I'm attaching the crash.dmp from 32 bit image. I believe that the crash happens when calling COMDispatchInstance>>propertyNamed: with the lengthy string to be returned.

Pablo, I hope you have some time to check this, my knowledge ends somewhere here ... :-| 

Thanks and best wishes,

Tomaz

Thanks, Stephane, for the acknowledgement. Peter, as I understand, the limits in COM BSTR data type are defined by the header's length prefix (which is 4 bytes) and software implementatios - for instance, string data type in Visual Basic for Applications is described as "a variable-length string can contain up to approximately 2 billion (2^31) characters", which is in line with the BSTR header. I'm not sure if the OS architecture (32 and 64 bit) influences these values. 

Best wishes,

Tomaz