Need provision to pass Body as parameter while presigning requests using sign_v4_query_params

[aravindanck]

Team,

Java SDK has two classes for signing requests with AWS4 signing protocol
AWS4UnsignedPayloadSigner.java - https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-core/src/main/java/com/amazonaws/auth/AWS4UnsignedPayloadSigner.java
AWS4Signer.java - https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-core/src/main/java/com/amazonaws/auth/AWS4Signer.java

AWS4UnsignedPayloadSigner adds "UNSIGNED-PAYLOAD" as body if the request is sent over HTTPS whereas AWS4Signer doesn't add "UNSIGNED-PAYLOAD" as request body for pre-signing

Background:
I'm trying to authenticate with AWS MSK via AWS_MSK_IAM SASL mechanism in Elixir and it needs me to presign the request without adding "UNSIGNED-PAYLOAD" as part of its canonical request. I see that aws_signature adds "UNSIGNED-PAYLOAD" as part of pre-signing here

aws_signature/src/aws_signature.erl

Line 170 in 9dff1ba

Body = <<"UNSIGNED-PAYLOAD">>,

.

This causes issues when I try to pre-sign requests to authenticate with services like Amazon MSK via IAM. Rather than always sending "UNSIGNED-PAYLOAD" as body, is it possible to have a provision to send "UNSIGNED-PAYLOAD" only when Body parameter is nil, otherwise use what is sent as Body? This needs modifying the api's definition to add Body as an additional parameter. Let me know if there are better alternatives to tackle this.
https://hexdocs.pm/aws_signature/aws_signature.html#sign_v4_query_params/8

[jonatanklosko]

Hey @aravindanck, the sign_v4_query_params function is used when sending the signature over query params, in that case the payload is always UNSIGNED-PAYLOAD as in the specification. So you most likely want to use the sign_v4 function and send the signature in the headers. sign_v4 calculates the payload hash. Also, this corresponds to the Java code you referenced, where the sign method modifies the request headers. Let me know if that helps :)

[aravindanck]

@jonatanklosko I understand that aws_signature implements the spec where UNSIGNED-PAYLOAD is mentioned as one of the valid payload strings in case of pre-signed URLs.
But, looks like in certain cases, IAM only allow empty strings as part of the canonical request creation for pre-signing URLs and doesn't consider UNSIGNED-PAYLOAD as a valid payload. When I look at it, it also says we don't include payload hash in the Canonical Request. Not sure how the validation is happening at AWS end.

You don't include a payload hash in the Canonical Request, because when you create a presigned URL, you don't know the payload content because the URL is used to upload an arbitrary payload. Instead, you use a constant string UNSIGNED-PAYLOAD

Background:
I'm creating an Elixir version of the Java reference implementation for Amazon MSK's IAM based SASL authentication, where it states that the payload has to be an empty string "" and the signing type is over query params. Do you still think I can use sign_v4 for my purpose?

References:

1. Signing is via query params and not headers (i.e., pre-signing) https://github.com/aws/aws-msk-iam-auth#generating-the-authentication-payload
2. Payload passed is an empty string - https://github.com/aws/aws-msk-iam-auth#hashed-payload
3. Reference implementation uses AWS4Signer for pre-signing - https://github.com/aws/aws-msk-iam-auth/blob/cde87031656333785f6e1414722fe635d365d89d/src/main/java/software/amazon/msk/auth/iam/internals/AWS4SignedPayloadGenerator.java#L58-L61

[jonatanklosko]

@aravindanck thanks for the details, I see it now. The code you referenced in 3. uses presignRequest, which corresponds to sign_v4_query_params. Also, the spec I mentioned is a part of S3 docs, so it adds this assumption about unsigned payload, which may not be the case for other services. The Ruby implementation supports body and body_digest options to control this, I think we need to mirror that, I'll send a PR.

[aravindanck]

Great, thanks @jonatanklosko

[jonatanklosko]

@aravindanck I've just released 0.3.0 and you should be good to go, if you run into any issues let us know :)

[aravindanck]

Works as expected for me. Thanks @jonatanklosko

[collegeimprovements]

Hi @jonatanklosko,
We have had the following working code with version-0.2.0:

    opts = [ttl: 86_400]
    signed_url =
      :aws_signature.sign_v4_query_params(
        access_key,
        secret_key,
        region,
        service,
        datetime,
        method,
        url,
        opts
      )

This breaks for pre-signed urls for 0.3.0. Do we need to change something to be compatible with 0.3.0 ?
The error we get is:

<Error>
<script id="youtube-hd-fjdmkanbdloodhegphphhklnjfngoffa">var ythdlog = () => {};;var ythderror = () => {};</script>
<Code>SignatureDoesNotMatch</Code>
<Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message>
<AWSAccessKeyId>AKIA4KVFM...6VN56....OG</AWSAccessKeyId>
<StringToSign>AWS4-HMAC-SHA256 20220530T070125Z 20220530/us-east-2/s3/aws4_request 0956990770e19ccaa5ef4f610fb6b0355db55c1d45fcb975a1d52511f79395fb</StringToSign>
<SignatureProvided>2a032f4c484eafa403388764758a6d72fa6421564444fbf58d384c9e5a1d8154</SignatureProvided>
<StringToSignBytes>41 57 53 34 2d 48 4d 41 43 2d 53 48 41 32 35 36 0a 32 30 32 32 30 35 33 30 54 30 37 30 31 32 35 5a 0a 32 30 32 32 30 35 33 30 2f 75 73 2d 65 61 73 74 2d 32 2f 73 33 2f 61 77 73 34 5f 72 65 71 75 65 73 74 0a 30 39 35 36 39 39 30 37 37 30 65 31 39 63 63 61 61 35 65 66 34 66 36 31 30 66 62 36 62 30 33 35 35 64 62 35 35 63 31 64 34 35 66 63 62 39 37 35 61 31 64 35 32 35 31 31 66 37 39 33 39 35 66 62</StringToSignBytes>
<CanonicalRequest>GET /public/9660a06a-59d8-482d-821c-1dd9718ec6d8/534e8b21-7188-49ed-b020-65e137091bbb.jpeg X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIA4KVFMDO6VN564QOG%2F20220530%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Date=20220530T070125Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host host:cookstro.s3.us-east-2.amazonaws.com host UNSIGNED-PAYLOAD</CanonicalRequest>
<CanonicalRequestBytes>47 45 54 0a 2f 70 75 62 6c 69 63 2f 39 36 36 30 61 30 36 61 2d 35 39 64 38 2d 34 38 32 64 2d 38 32 31 63 2d 31 64 64 39 37 31 38 65 63 36 64 38 2f 35 33 34 65 38 62 32 31 2d 37 31 38 38 2d 34 39 65 64 2d 62 30 32 30 2d 36 35 65 31 33 37 30 39 31 62 62 62 2e 6a 70 65 67 0a 58 2d 41 6d 7a 2d 41 6c 67 6f 72 69 74 68 6d 3d 41 57 53 34 2d 48 4d 41 43 2d 53 48 41 32 35 36 26 58 2d 41 6d 7a 2d 43 72 65 64 65 6e 74 69 61 6c 3d 41 4b 49 41 34 4b 56 46 4d 44 4f 36 56 4e 35 36 34 51 4f 47 25 32 46 32 30 32 32 30 35 33 30 25 32 46 75 73 2d 65 61 73 74 2d 32 25 32 46 73 33 25 32 46 61 77 73 34 5f 72 65 71 75 65 73 74 26 58 2d 41 6d 7a 2d 44 61 74 65 3d 32 30 32 32 30 35 33 30 54 30 37 30 31 32 35 5a 26 58 2d 41 6d 7a 2d 45 78 70 69 72 65 73 3d 38 36 34 30 30 26 58 2d 41 6d 7a 2d 53 69 67 6e 65 64 48 65 61 64 65 72 73 3d 68 6f 73 74 0a 68 6f 73 74 3a 63 6f 6f 6b 73 74 72 6f 2e 73 33 2e 75 73 2d 65 61 73 74 2d 32 2e 61 6d 61 7a 6f 6e 61 77 73 2e 63 6f 6d 0a 0a 68 6f 73 74 0a 55 4e 53 49 47 4e 45 44 2d 50 41 59 4c 4f 41 44</CanonicalRequestBytes>
<RequestId>W8BFFQJPX6KFA8VQ</RequestId>
<HostId>ic//EUI+wtcZaWw5LP99znf08uuicOBqsjvbLRb5jvIAyWRqXLq+Ysl2yRcdW1lY/I/ZXa6y9JU=</HostId>
</Error>

[jonatanklosko]

Hey @collegeimprovements, you most likely need body_digest: "UNSIGNED-PAYLOAD", please see v0.3.0 release notes :)

[collegeimprovements]

That was it. Thanks a lot @jonatanklosko :)