Zn and AWS - retrieving object versions?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Zn and AWS - retrieving object versions?

Tim Mackinnon
Hi everyone - I’m wondering if someone knows the trick to listing object versions in AWS S3?

I was previously using a non-Zn library (there are a few around - but they are quite old and I’m not sure how much they are maintained) - however I hadn’t realised that Zn actually supports S3 until I read it a bit carefully and realised that I needed to load an extra AWS group.

So - I have a bucket that is versioned, and I wanted to read the versions of an object - and hopefully be able to read the contents of an older version.

I can list the contents of a bucket, but when I try to read the versions of an object - I get a forbidden error - which when I dig deeper gives a stranger explanation?

I have confirmed that the AWS CLI is able to list versions in that bucket - so I’m a bit confused what the issue might be?

(client := ZnAWSS3Client new)
        accessKeyId: 'xxx';
        secretAccessKey: ‘yyyyy';
        checkIntegrity: true.

client buckets. “Works"
client keysIn: 'mtt-data’. “Works"

client at: 'mtt-data' -> 'sample.txt’. “Works"

client keysIn: 'mtt-data' query: (Dictionary with: 'versions'->nil). “Gives an error?”


HTTP/1.1 403 Forbidden

<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message><AWSAccessKeyId>xxx</AWSAccessKeyId><StringToSign>GET


Has anyone tried doing this - I’m sure there is something really simple that I am missing?

Tim
Reply | Threaded
Open this post in threaded view
|

Re: Zn and AWS - retrieving object versions?

Sven Van Caekenberghe-2
Hi Tim,

It has been a very long time since I last looked at this code. It is a good thing that at least part of it still works.

Just by reading the code and reasoning about your problem, I would suggest you try the following change:

In ZnAWSS3RequestSignatureTool>>#canonicalStringFor: replace the last

 request uri pathPrintString

by

 request uri pathQueryFragmentPrintString

I did not try this myself, so it might not solve your issue.

Please let me know if this works.

Sven

> On 17 Jun 2021, at 03:05, Tim Mackinnon <[hidden email]> wrote:
>
> Hi everyone - I’m wondering if someone knows the trick to listing object versions in AWS S3?
>
> I was previously using a non-Zn library (there are a few around - but they are quite old and I’m not sure how much they are maintained) - however I hadn’t realised that Zn actually supports S3 until I read it a bit carefully and realised that I needed to load an extra AWS group.
>
> So - I have a bucket that is versioned, and I wanted to read the versions of an object - and hopefully be able to read the contents of an older version.
>
> I can list the contents of a bucket, but when I try to read the versions of an object - I get a forbidden error - which when I dig deeper gives a stranger explanation?
>
> I have confirmed that the AWS CLI is able to list versions in that bucket - so I’m a bit confused what the issue might be?
>
> (client := ZnAWSS3Client new)
> accessKeyId: 'xxx';
> secretAccessKey: ‘yyyyy';
> checkIntegrity: true.
>
> client buckets. “Works"
> client keysIn: 'mtt-data’. “Works"
>
> client at: 'mtt-data' -> 'sample.txt’. “Works"
>
> client keysIn: 'mtt-data' query: (Dictionary with: 'versions'->nil). “Gives an error?”
>
>
> HTTP/1.1 403 Forbidden
>
> <?xml version="1.0" encoding="UTF-8"?>
> <Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message><AWSAccessKeyId>xxx</AWSAccessKeyId><StringToSign>GET
>
>
> Has anyone tried doing this - I’m sure there is something really simple that I am missing?
>
> Tim
Reply | Threaded
Open this post in threaded view
|

Re: Zn and AWS - retrieving object versions?

Tim Mackinnon
Hi Sven - thanks for taking a quick look, this change did stop me getting an immediate error - but I seem to get no results back.

Digging into it a bit more - I seem to be issuing the GET request as indicated by https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectVersions.html

e.g.

GET /?versions HTTP/1.1

Headers:
a ZnMultiValueDictionary('User-Agent'->'Zinc HTTP Components 1.0 (Pharo/8.0)’
'Accept'->'*/*' 'Host'->'mtt-data.s3.amazonaws.com’
'Date'->'Thu, 17 Jun 2021 16:08:44 GMT’
'Authorization'->'AWS AKIAIP2xxxxxxx:DqPPbDyyyyy’)

From the spec (which I’m no expert on) - the get command looks right (the other params are optional) and the Bucket name is prefixed onto s3.amazonaws.com in the header.

But is seems that I am getting a 200 - 403 response :
"a ZnResponse(200 OK application/xml 1899B)"
HTTP/1.1 403 Forbidden

(Aside - the code should possibly handle this  better - the spec seems to say that response content needs to be interpreted - but that’s something I can look to help with when we know the actual issue).

But I’m a bit mystified why the previous: client keysIn: 'mtt-data’
Invocation would work, but this seemingly extra step would be forbidden. My IAM role says it has full S3 access, so I’m not sure if this is still that we haven’t got the right fix or if its something environmental. I’m not sure what to do - if you have another idea, I’m happy to be a guniea pig to try it.

Tim



> On 17 Jun 2021, at 09:06, Sven Van Caekenberghe <[hidden email]> wrote:
>
> Hi Tim,
>
> It has been a very long time since I last looked at this code. It is a good thing that at least part of it still works.
>
> Just by reading the code and reasoning about your problem, I would suggest you try the following change:
>
> In ZnAWSS3RequestSignatureTool>>#canonicalStringFor: replace the last
>
> request uri pathPrintString
>
> by
>
> request uri pathQueryFragmentPrintString
>
> I did not try this myself, so it might not solve your issue.
>
> Please let me know if this works.
>
> Sven
>
>> On 17 Jun 2021, at 03:05, Tim Mackinnon <[hidden email]> wrote:
>>
>> Hi everyone - I’m wondering if someone knows the trick to listing object versions in AWS S3?
>>
>> I was previously using a non-Zn library (there are a few around - but they are quite old and I’m not sure how much they are maintained) - however I hadn’t realised that Zn actually supports S3 until I read it a bit carefully and realised that I needed to load an extra AWS group.
>>
>> So - I have a bucket that is versioned, and I wanted to read the versions of an object - and hopefully be able to read the contents of an older version.
>>
>> I can list the contents of a bucket, but when I try to read the versions of an object - I get a forbidden error - which when I dig deeper gives a stranger explanation?
>>
>> I have confirmed that the AWS CLI is able to list versions in that bucket - so I’m a bit confused what the issue might be?
>>
>> (client := ZnAWSS3Client new)
>> accessKeyId: 'xxx';
>> secretAccessKey: ‘yyyyy';
>> checkIntegrity: true.
>>
>> client buckets. “Works"
>> client keysIn: 'mtt-data’. “Works"
>>
>> client at: 'mtt-data' -> 'sample.txt’. “Works"
>>
>> client keysIn: 'mtt-data' query: (Dictionary with: 'versions'->nil). “Gives an error?”
>>
>>
>> HTTP/1.1 403 Forbidden
>>
>> <?xml version="1.0" encoding="UTF-8"?>
>> <Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message><AWSAccessKeyId>xxx</AWSAccessKeyId><StringToSign>GET
>>
>>
>> Has anyone tried doing this - I’m sure there is something really simple that I am missing?
>>
>> Tim
Reply | Threaded
Open this post in threaded view
|

Re: Zn and AWS - retrieving object versions?

Tim Mackinnon
Hi Sven - actually I think its not just the versions thing - I can’t get filtering to work on the standard query either:

client keysIn: ‘mtt-data' query: (Dictionary with: 'prefix'->'sample’).

Given a normal query works, there must be something extra that is being missed/added that impacts things.

Tim

> On 17 Jun 2021, at 17:19, Tim Mackinnon <[hidden email]> wrote:
>
> Hi Sven - thanks for taking a quick look, this change did stop me getting an immediate error - but I seem to get no results back.
>
> Digging into it a bit more - I seem to be issuing the GET request as indicated by https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectVersions.html
>
> e.g.
>
> GET /?versions HTTP/1.1
>
> Headers:
> a ZnMultiValueDictionary('User-Agent'->'Zinc HTTP Components 1.0 (Pharo/8.0)’
> 'Accept'->'*/*' 'Host'->'mtt-data.s3.amazonaws.com’
> 'Date'->'Thu, 17 Jun 2021 16:08:44 GMT’
> 'Authorization'->'AWS AKIAIP2xxxxxxx:DqPPbDyyyyy’)
>
> From the spec (which I’m no expert on) - the get command looks right (the other params are optional) and the Bucket name is prefixed onto s3.amazonaws.com in the header.
>
> But is seems that I am getting a 200 - 403 response :
> "a ZnResponse(200 OK application/xml 1899B)"
> HTTP/1.1 403 Forbidden
>
> (Aside - the code should possibly handle this  better - the spec seems to say that response content needs to be interpreted - but that’s something I can look to help with when we know the actual issue).
>
> But I’m a bit mystified why the previous: client keysIn: 'mtt-data’
> Invocation would work, but this seemingly extra step would be forbidden. My IAM role says it has full S3 access, so I’m not sure if this is still that we haven’t got the right fix or if its something environmental. I’m not sure what to do - if you have another idea, I’m happy to be a guniea pig to try it.
>
> Tim
>
>
>
>> On 17 Jun 2021, at 09:06, Sven Van Caekenberghe <[hidden email]> wrote:
>>
>> Hi Tim,
>>
>> It has been a very long time since I last looked at this code. It is a good thing that at least part of it still works.
>>
>> Just by reading the code and reasoning about your problem, I would suggest you try the following change:
>>
>> In ZnAWSS3RequestSignatureTool>>#canonicalStringFor: replace the last
>>
>> request uri pathPrintString
>>
>> by
>>
>> request uri pathQueryFragmentPrintString
>>
>> I did not try this myself, so it might not solve your issue.
>>
>> Please let me know if this works.
>>
>> Sven
>>
>>> On 17 Jun 2021, at 03:05, Tim Mackinnon <[hidden email]> wrote:
>>>
>>> Hi everyone - I’m wondering if someone knows the trick to listing object versions in AWS S3?
>>>
>>> I was previously using a non-Zn library (there are a few around - but they are quite old and I’m not sure how much they are maintained) - however I hadn’t realised that Zn actually supports S3 until I read it a bit carefully and realised that I needed to load an extra AWS group.
>>>
>>> So - I have a bucket that is versioned, and I wanted to read the versions of an object - and hopefully be able to read the contents of an older version.
>>>
>>> I can list the contents of a bucket, but when I try to read the versions of an object - I get a forbidden error - which when I dig deeper gives a stranger explanation?
>>>
>>> I have confirmed that the AWS CLI is able to list versions in that bucket - so I’m a bit confused what the issue might be?
>>>
>>> (client := ZnAWSS3Client new)
>>> accessKeyId: 'xxx';
>>> secretAccessKey: ‘yyyyy';
>>> checkIntegrity: true.
>>>
>>> client buckets. “Works"
>>> client keysIn: 'mtt-data’. “Works"
>>>
>>> client at: 'mtt-data' -> 'sample.txt’. “Works"
>>>
>>> client keysIn: 'mtt-data' query: (Dictionary with: 'versions'->nil). “Gives an error?”
>>>
>>>
>>> HTTP/1.1 403 Forbidden
>>>
>>> <?xml version="1.0" encoding="UTF-8"?>
>>> <Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message><AWSAccessKeyId>xxx</AWSAccessKeyId><StringToSign>GET
>>>
>>>
>>> Has anyone tried doing this - I’m sure there is something really simple that I am missing?
>>>
>>> Tim
Reply | Threaded
Open this post in threaded view
|

Re: Zn and AWS - retrieving object versions?

Sven Van Caekenberghe-2
Tim,

I think that from the moment there is a query part added to the REST URL something goes wrong.

Step 1 is to make sure the signing works. Now, ZnAWSS3RequestSignatureTool was written for an earlier version (I don't remember which one, probably V1). This is tricky to figure out.

I know for example that if there are multiple query arguments, they have to be sorted, which is currently not happening. But that won't be the only problem (encoding could be another). The challenge is to construct the correct 'canonical' string to sign.

First you got a clear 'signature wrong' error, that seems to be gone. But you still get an authentication error, I am not sure what is going on.

I am afraid you will have to try to debug this a bit yourself. It has been years since I looked at this code.

If you get this to work with another tool, maybe you can enable debug logging to see what is happening with the URL and headers.

Sven

PS: I think there is other AWS S3 for Pharo out there, but I can't remember.

> On 17 Jun 2021, at 18:32, Tim Mackinnon <[hidden email]> wrote:
>
> Hi Sven - actually I think its not just the versions thing - I can’t get filtering to work on the standard query either:
>
> client keysIn: ‘mtt-data' query: (Dictionary with: 'prefix'->'sample’).
>
> Given a normal query works, there must be something extra that is being missed/added that impacts things.
>
> Tim
>
>> On 17 Jun 2021, at 17:19, Tim Mackinnon <[hidden email]> wrote:
>>
>> Hi Sven - thanks for taking a quick look, this change did stop me getting an immediate error - but I seem to get no results back.
>>
>> Digging into it a bit more - I seem to be issuing the GET request as indicated by https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectVersions.html
>>
>> e.g.
>>
>> GET /?versions HTTP/1.1
>>
>> Headers:
>> a ZnMultiValueDictionary('User-Agent'->'Zinc HTTP Components 1.0 (Pharo/8.0)’
>> 'Accept'->'*/*' 'Host'->'mtt-data.s3.amazonaws.com’
>> 'Date'->'Thu, 17 Jun 2021 16:08:44 GMT’
>> 'Authorization'->'AWS AKIAIP2xxxxxxx:DqPPbDyyyyy’)
>>
>> From the spec (which I’m no expert on) - the get command looks right (the other params are optional) and the Bucket name is prefixed onto s3.amazonaws.com in the header.
>>
>> But is seems that I am getting a 200 - 403 response :
>> "a ZnResponse(200 OK application/xml 1899B)"
>> HTTP/1.1 403 Forbidden
>>
>> (Aside - the code should possibly handle this  better - the spec seems to say that response content needs to be interpreted - but that’s something I can look to help with when we know the actual issue).
>>
>> But I’m a bit mystified why the previous: client keysIn: 'mtt-data’
>> Invocation would work, but this seemingly extra step would be forbidden. My IAM role says it has full S3 access, so I’m not sure if this is still that we haven’t got the right fix or if its something environmental. I’m not sure what to do - if you have another idea, I’m happy to be a guniea pig to try it.
>>
>> Tim
>>
>>
>>
>>> On 17 Jun 2021, at 09:06, Sven Van Caekenberghe <[hidden email]> wrote:
>>>
>>> Hi Tim,
>>>
>>> It has been a very long time since I last looked at this code. It is a good thing that at least part of it still works.
>>>
>>> Just by reading the code and reasoning about your problem, I would suggest you try the following change:
>>>
>>> In ZnAWSS3RequestSignatureTool>>#canonicalStringFor: replace the last
>>>
>>> request uri pathPrintString
>>>
>>> by
>>>
>>> request uri pathQueryFragmentPrintString
>>>
>>> I did not try this myself, so it might not solve your issue.
>>>
>>> Please let me know if this works.
>>>
>>> Sven
>>>
>>>> On 17 Jun 2021, at 03:05, Tim Mackinnon <[hidden email]> wrote:
>>>>
>>>> Hi everyone - I’m wondering if someone knows the trick to listing object versions in AWS S3?
>>>>
>>>> I was previously using a non-Zn library (there are a few around - but they are quite old and I’m not sure how much they are maintained) - however I hadn’t realised that Zn actually supports S3 until I read it a bit carefully and realised that I needed to load an extra AWS group.
>>>>
>>>> So - I have a bucket that is versioned, and I wanted to read the versions of an object - and hopefully be able to read the contents of an older version.
>>>>
>>>> I can list the contents of a bucket, but when I try to read the versions of an object - I get a forbidden error - which when I dig deeper gives a stranger explanation?
>>>>
>>>> I have confirmed that the AWS CLI is able to list versions in that bucket - so I’m a bit confused what the issue might be?
>>>>
>>>> (client := ZnAWSS3Client new)
>>>> accessKeyId: 'xxx';
>>>> secretAccessKey: ‘yyyyy';
>>>> checkIntegrity: true.
>>>>
>>>> client buckets. “Works"
>>>> client keysIn: 'mtt-data’. “Works"
>>>>
>>>> client at: 'mtt-data' -> 'sample.txt’. “Works"
>>>>
>>>> client keysIn: 'mtt-data' query: (Dictionary with: 'versions'->nil). “Gives an error?”
>>>>
>>>>
>>>> HTTP/1.1 403 Forbidden
>>>>
>>>> <?xml version="1.0" encoding="UTF-8"?>
>>>> <Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message><AWSAccessKeyId>xxx</AWSAccessKeyId><StringToSign>GET
>>>>
>>>>
>>>> Has anyone tried doing this - I’m sure there is something really simple that I am missing?
>>>>
>>>> Tim
Reply | Threaded
Open this post in threaded view
|

Re: Zn and AWS - retrieving object versions?

Sven Van Caekenberghe-2
Tim,

> On 17 Jun 2021, at 19:05, Sven Van Caekenberghe <[hidden email]> wrote:
>
> PS: I think there is other AWS S3 for Pharo out there, but I can't remember.

I found two of them:

- https://github.com/newapplesho/aws-sdk-smalltalk

- https://github.com/jvdsandt/pharo-aws-toolbox

I am sure both of these are more up to date and get more AWS details right that my old minimal, proof of concept package.

Sven