Zn and AWS - retrieving object versions?

> On 17 Jun 2021, at 03:05, Tim Mackinnon <[hidden email]> wrote:
>
> Hi everyone - I’m wondering if someone knows the trick to listing object versions in AWS S3?
>
> I was previously using a non-Zn library (there are a few around - but they are quite old and I’m not sure how much they are maintained) - however I hadn’t realised that Zn actually supports S3 until I read it a bit carefully and realised that I needed to load an extra AWS group.
>
> So - I have a bucket that is versioned, and I wanted to read the versions of an object - and hopefully be able to read the contents of an older version.
>
> I can list the contents of a bucket, but when I try to read the versions of an object - I get a forbidden error - which when I dig deeper gives a stranger explanation?
>
> I have confirmed that the AWS CLI is able to list versions in that bucket - so I’m a bit confused what the issue might be?
>
> (client := ZnAWSS3Client new)
> accessKeyId: 'xxx';
> secretAccessKey: ‘yyyyy';
> checkIntegrity: true.
>
> client buckets. “Works"
> client keysIn: 'mtt-data’. “Works"
>
> client at: 'mtt-data' -> 'sample.txt’. “Works"
>
> client keysIn: 'mtt-data' query: (Dictionary with: 'versions'->nil). “Gives an error?”
>
>
> HTTP/1.1 403 Forbidden
>
> <?xml version="1.0" encoding="UTF-8"?>
> <Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message><AWSAccessKeyId>xxx</AWSAccessKeyId><StringToSign>GET
>
>
> Has anyone tried doing this - I’m sure there is something really simple that I am missing?
>
> Tim

> On 17 Jun 2021, at 09:06, Sven Van Caekenberghe <[hidden email]> wrote:
>
> Hi Tim,
>
> It has been a very long time since I last looked at this code. It is a good thing that at least part of it still works.
>
> Just by reading the code and reasoning about your problem, I would suggest you try the following change:
>
> In ZnAWSS3RequestSignatureTool>>#canonicalStringFor: replace the last
>
> request uri pathPrintString
>
> by
>
> request uri pathQueryFragmentPrintString
>
> I did not try this myself, so it might not solve your issue.
>
> Please let me know if this works.
>
> Sven
>
>> On 17 Jun 2021, at 03:05, Tim Mackinnon <[hidden email]> wrote:
>>
>> Hi everyone - I’m wondering if someone knows the trick to listing object versions in AWS S3?
>>
>> I was previously using a non-Zn library (there are a few around - but they are quite old and I’m not sure how much they are maintained) - however I hadn’t realised that Zn actually supports S3 until I read it a bit carefully and realised that I needed to load an extra AWS group.
>>
>> So - I have a bucket that is versioned, and I wanted to read the versions of an object - and hopefully be able to read the contents of an older version.
>>
>> I can list the contents of a bucket, but when I try to read the versions of an object - I get a forbidden error - which when I dig deeper gives a stranger explanation?
>>
>> I have confirmed that the AWS CLI is able to list versions in that bucket - so I’m a bit confused what the issue might be?
>>
>> (client := ZnAWSS3Client new)
>> accessKeyId: 'xxx';
>> secretAccessKey: ‘yyyyy';
>> checkIntegrity: true.
>>
>> client buckets. “Works"
>> client keysIn: 'mtt-data’. “Works"
>>
>> client at: 'mtt-data' -> 'sample.txt’. “Works"
>>
>> client keysIn: 'mtt-data' query: (Dictionary with: 'versions'->nil). “Gives an error?”
>>
>>
>> HTTP/1.1 403 Forbidden
>>
>> <?xml version="1.0" encoding="UTF-8"?>
>> <Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message><AWSAccessKeyId>xxx</AWSAccessKeyId><StringToSign>GET
>>
>>
>> Has anyone tried doing this - I’m sure there is something really simple that I am missing?
>>
>> Tim

> On 17 Jun 2021, at 17:19, Tim Mackinnon <[hidden email]> wrote:
>
> Hi Sven - thanks for taking a quick look, this change did stop me getting an immediate error - but I seem to get no results back.
>
> Digging into it a bit more - I seem to be issuing the GET request as indicated by https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectVersions.html
>
> e.g.
>
> GET /?versions HTTP/1.1
>
> Headers:
> a ZnMultiValueDictionary('User-Agent'->'Zinc HTTP Components 1.0 (Pharo/8.0)’
> 'Accept'->'*/*' 'Host'->'mtt-data.s3.amazonaws.com’
> 'Date'->'Thu, 17 Jun 2021 16:08:44 GMT’
> 'Authorization'->'AWS AKIAIP2xxxxxxx:DqPPbDyyyyy’)
>
> From the spec (which I’m no expert on) - the get command looks right (the other params are optional) and the Bucket name is prefixed onto s3.amazonaws.com in the header.
>
> But is seems that I am getting a 200 - 403 response :
> "a ZnResponse(200 OK application/xml 1899B)"
> HTTP/1.1 403 Forbidden
>
> (Aside - the code should possibly handle this  better - the spec seems to say that response content needs to be interpreted - but that’s something I can look to help with when we know the actual issue).
>
> But I’m a bit mystified why the previous: client keysIn: 'mtt-data’
> Invocation would work, but this seemingly extra step would be forbidden. My IAM role says it has full S3 access, so I’m not sure if this is still that we haven’t got the right fix or if its something environmental. I’m not sure what to do - if you have another idea, I’m happy to be a guniea pig to try it.
>
> Tim
>
>
>
>> On 17 Jun 2021, at 09:06, Sven Van Caekenberghe <[hidden email]> wrote:
>>
>> Hi Tim,
>>
>> It has been a very long time since I last looked at this code. It is a good thing that at least part of it still works.
>>
>> Just by reading the code and reasoning about your problem, I would suggest you try the following change:
>>
>> In ZnAWSS3RequestSignatureTool>>#canonicalStringFor: replace the last
>>
>> request uri pathPrintString
>>
>> by
>>
>> request uri pathQueryFragmentPrintString
>>
>> I did not try this myself, so it might not solve your issue.
>>
>> Please let me know if this works.
>>
>> Sven
>>
>>> On 17 Jun 2021, at 03:05, Tim Mackinnon <[hidden email]> wrote:
>>>
>>> Hi everyone - I’m wondering if someone knows the trick to listing object versions in AWS S3?
>>>
>>> I was previously using a non-Zn library (there are a few around - but they are quite old and I’m not sure how much they are maintained) - however I hadn’t realised that Zn actually supports S3 until I read it a bit carefully and realised that I needed to load an extra AWS group.
>>>
>>> So - I have a bucket that is versioned, and I wanted to read the versions of an object - and hopefully be able to read the contents of an older version.
>>>
>>> I can list the contents of a bucket, but when I try to read the versions of an object - I get a forbidden error - which when I dig deeper gives a stranger explanation?
>>>
>>> I have confirmed that the AWS CLI is able to list versions in that bucket - so I’m a bit confused what the issue might be?
>>>
>>> (client := ZnAWSS3Client new)
>>> accessKeyId: 'xxx';
>>> secretAccessKey: ‘yyyyy';
>>> checkIntegrity: true.
>>>
>>> client buckets. “Works"
>>> client keysIn: 'mtt-data’. “Works"
>>>
>>> client at: 'mtt-data' -> 'sample.txt’. “Works"
>>>
>>> client keysIn: 'mtt-data' query: (Dictionary with: 'versions'->nil). “Gives an error?”
>>>
>>>
>>> HTTP/1.1 403 Forbidden
>>>
>>> <?xml version="1.0" encoding="UTF-8"?>
>>> <Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message><AWSAccessKeyId>xxx</AWSAccessKeyId><StringToSign>GET
>>>
>>>
>>> Has anyone tried doing this - I’m sure there is something really simple that I am missing?
>>>
>>> Tim

> On 17 Jun 2021, at 18:32, Tim Mackinnon <[hidden email]> wrote:
>
> Hi Sven - actually I think its not just the versions thing - I can’t get filtering to work on the standard query either:
>
> client keysIn: ‘mtt-data' query: (Dictionary with: 'prefix'->'sample’).
>
> Given a normal query works, there must be something extra that is being missed/added that impacts things.
>
> Tim
>
>> On 17 Jun 2021, at 17:19, Tim Mackinnon <[hidden email]> wrote:
>>
>> Hi Sven - thanks for taking a quick look, this change did stop me getting an immediate error - but I seem to get no results back.
>>
>> Digging into it a bit more - I seem to be issuing the GET request as indicated by https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectVersions.html
>>
>> e.g.
>>
>> GET /?versions HTTP/1.1
>>
>> Headers:
>> a ZnMultiValueDictionary('User-Agent'->'Zinc HTTP Components 1.0 (Pharo/8.0)’
>> 'Accept'->'*/*' 'Host'->'mtt-data.s3.amazonaws.com’
>> 'Date'->'Thu, 17 Jun 2021 16:08:44 GMT’
>> 'Authorization'->'AWS AKIAIP2xxxxxxx:DqPPbDyyyyy’)
>>
>> From the spec (which I’m no expert on) - the get command looks right (the other params are optional) and the Bucket name is prefixed onto s3.amazonaws.com in the header.
>>
>> But is seems that I am getting a 200 - 403 response :
>> "a ZnResponse(200 OK application/xml 1899B)"
>> HTTP/1.1 403 Forbidden
>>
>> (Aside - the code should possibly handle this  better - the spec seems to say that response content needs to be interpreted - but that’s something I can look to help with when we know the actual issue).
>>
>> But I’m a bit mystified why the previous: client keysIn: 'mtt-data’
>> Invocation would work, but this seemingly extra step would be forbidden. My IAM role says it has full S3 access, so I’m not sure if this is still that we haven’t got the right fix or if its something environmental. I’m not sure what to do - if you have another idea, I’m happy to be a guniea pig to try it.
>>
>> Tim
>>
>>
>>
>>> On 17 Jun 2021, at 09:06, Sven Van Caekenberghe <[hidden email]> wrote:
>>>
>>> Hi Tim,
>>>
>>> It has been a very long time since I last looked at this code. It is a good thing that at least part of it still works.
>>>
>>> Just by reading the code and reasoning about your problem, I would suggest you try the following change:
>>>
>>> In ZnAWSS3RequestSignatureTool>>#canonicalStringFor: replace the last
>>>
>>> request uri pathPrintString
>>>
>>> by
>>>
>>> request uri pathQueryFragmentPrintString
>>>
>>> I did not try this myself, so it might not solve your issue.
>>>
>>> Please let me know if this works.
>>>
>>> Sven
>>>
>>>> On 17 Jun 2021, at 03:05, Tim Mackinnon <[hidden email]> wrote:
>>>>
>>>> Hi everyone - I’m wondering if someone knows the trick to listing object versions in AWS S3?
>>>>
>>>> I was previously using a non-Zn library (there are a few around - but they are quite old and I’m not sure how much they are maintained) - however I hadn’t realised that Zn actually supports S3 until I read it a bit carefully and realised that I needed to load an extra AWS group.
>>>>
>>>> So - I have a bucket that is versioned, and I wanted to read the versions of an object - and hopefully be able to read the contents of an older version.
>>>>
>>>> I can list the contents of a bucket, but when I try to read the versions of an object - I get a forbidden error - which when I dig deeper gives a stranger explanation?
>>>>
>>>> I have confirmed that the AWS CLI is able to list versions in that bucket - so I’m a bit confused what the issue might be?
>>>>
>>>> (client := ZnAWSS3Client new)
>>>> accessKeyId: 'xxx';
>>>> secretAccessKey: ‘yyyyy';
>>>> checkIntegrity: true.
>>>>
>>>> client buckets. “Works"
>>>> client keysIn: 'mtt-data’. “Works"
>>>>
>>>> client at: 'mtt-data' -> 'sample.txt’. “Works"
>>>>
>>>> client keysIn: 'mtt-data' query: (Dictionary with: 'versions'->nil). “Gives an error?”
>>>>
>>>>
>>>> HTTP/1.1 403 Forbidden
>>>>
>>>> <?xml version="1.0" encoding="UTF-8"?>
>>>> <Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message><AWSAccessKeyId>xxx</AWSAccessKeyId><StringToSign>GET
>>>>
>>>>
>>>> Has anyone tried doing this - I’m sure there is something really simple that I am missing?
>>>>
>>>> Tim