feat(events): add APIs to list webhook events and webhook delivery attempts

[SanchithHegde]

Type of Change

• Bugfix
• New feature
• Enhancement
• Refactoring
• Dependency updates
• Documentation
• CI/CD

Description

This PR introduces two API endpoints, one to list (and filter) webhook events and one to list webhook delivery attempts:

1. Events List:

   • URL: /events/{merchant_id}
   • Authentication: Admin API authentication + JWT
   • The endpoint returns the list of events without the request and response information.
   • Optionally, filtering can be performed by the query parameters:
     • object_id: Primary object ID (such as payment ID, refund ID, mandate ID, dispute ID) used to filter events related to that object.
     • limit
     • offset
     • created_after, created_before: Specify date and time range to include events from.

2. Webhook Delivery Attempts List

   • URL: /events/{merchant_id}/{initial_event_id}/attempts
   • Authentication: Admin API authentication + JWT
   • The endpoint returns the delivery attempts for the specified initial event, and includes the request and response information (request and response bodies are strings).

This PR includes the following changes:

1. Queries on the events table now include the merchant_id field everywhere.
2. Introduces APIs to list and filter initial events, and to list delivery webhook delivery attempts for a specified event.
3. OpenAPI specification additions for the newly introduced endpoints.
4. Database migrations to add indexes on the columns on the events table that would be queried by the newly introduced endpoints.
5. Adds a delivery_attempt column to the events table to indicate the type of delivery attempt for the webhook.

In addition this PR has a few changes not directly related to the PR:

• Fixes a typo: rename EncryptionStratergy to EncryptionStrategy.
• Removes some unused and commented code in the ext_traits module.
• Replaces unnecessary attach_printable_lazy() calls in the generics module with attach_printable().
• Bumps the utoipa dependency from version 3.3.0 to 3.5.0 and enables the preserve_path_order feature flag.
• Moves around a couple of routes in the OpenAPI specification, and updates their tags to group them better.

The PR can easily be reviewed one commit at a time.

Known Issues

The above endpoints can be used to retrieve information on events created by the application versions newer than 2024.03.15.0, since events are now queried with the merchant ID, and the merchant ID field is populated in the database table starting version 2024.03.15.0. The application will throw an internal server error when older events are being retrieved.

Additional Changes

• This PR modifies the API contract
• This PR modifies the database schema
• This PR modifies application configuration/environment variables

The database migrations for this PR can be found in the up.sql and down.sql files.

Motivation and Context

These endpoints would better allow users to view the recently delivered webhooks and debug issues with webhook deliveries to their application servers.

How did you test it?

1. Ensuring that the delivery_attempt field is being populated correctly in the events table for initial attempt and automatic retries via the scheduler:

   This can be verified using a SQL query like so:

   SELECT event_id, is_webhook_notified, primary_object_id, created_at, idempotent_event_id, initial_attempt_id, delivery_attempt FROM events WHERE initial_attempt_id = 'evt_018e562fcd4571b0b831098bb9ebe230' ORDER BY created_at DESC;

   

2. Listing events related to a specific object (say payment intent, refund, dispute):

   curl -H 'api-key: {{admin_api_key}}' '{{baseUrl}}/events/{{merchant_id}}?object_id={{payment_id}}'

   You should see a response like so, with events sorted in descending order of creation:

   $ curl -H 'api-key: test_admin' 'http://localhost:8080/events/merchant_1710609960?object_id=pay_9uclJXQboJ0FZPm9Nq2K' 2>/dev/null | jq '.'
   [
     {
       "event_id": "evt_018e484cb6207265a1c715568f889590",
       "merchant_id": "merchant_1710609960",
       "profile_id": "pro_H7FbvX2pBuJLjlPhkpRa",
       "object_id": "pay_9uclJXQboJ0FZPm9Nq2K",
       "event_type": "payment_succeeded",
       "event_class": "payments",
       "is_delivery_successful": true,
       "initial_attempt_id": "evt_018e484cb6207265a1c715568f889590",
       "created": "2024-03-16T17:26:10.720Z"
     }
   ]

3. Listing and filtering events based on time range:

   curl -H 'api-key: {{admin_api_key}}' '{{baseUrl}}/events/{{merchant_id}}?created_after={{fromTimestamp}}&created_before={{toTimestamp}}&limit={{limit}}&offset={{offset}}'

   • All query parameters created_after, created_before, limit and offset are optional, and one or more of them may be specified together.
   • created_after, created_before, limit and offset cannot be specified if object_id is specified, the application throws a bad request error in this case.

   1. Specifying all query params:

      $ curl -H 'api-key: test_admin' 'http://localhost:8080/events/merchant_1710609960?created_after=2024-03-18T09:00Z&created_before=2024-03-18T11:00Z&limit=2&offset=2' 2>/dev/null | jq '.'
      [
        {
          "event_id": "evt_018e50da61a674d6bdbd7150775087f2",
          "merchant_id": "merchant_1710609960",
          "profile_id": "pro_H7FbvX2pBuJLjlPhkpRa",
          "object_id": "pay_WkrmqLjtHqyjWdCYA1Ze",
          "event_type": "payment_succeeded",
          "event_class": "payments",
          "is_delivery_successful": false,
          "initial_attempt_id": "evt_018e50da61a674d6bdbd7150775087f2",
          "created": "2024-03-18T09:17:52.934Z"
        },
        {
          "event_id": "evt_018e50da59e577c6a5e843fe08537a01",
          "merchant_id": "merchant_1710609960",
          "profile_id": "pro_H7FbvX2pBuJLjlPhkpRa",
          "object_id": "pay_ECPc1GBlY8uC4HushNUU",
          "event_type": "payment_succeeded",
          "event_class": "payments",
          "is_delivery_successful": false,
          "initial_attempt_id": "evt_018e50da59e577c6a5e843fe08537a01",
          "created": "2024-03-18T09:17:50.950Z"
        }
      ]

   2. Specifying a subset of the query params (only created_after):

      $ curl -H 'api-key: test_admin' 'http://localhost:8080/events/merchant_1710609960?created_after=2024-03-19T0910:00Z' 2>/dev/null | jq '.'
      [
        {
          "event_id": "evt_018e562fcd4571b0b831098bb9ebe230",
          "merchant_id": "merchant_1710609960",
          "profile_id": "pro_H7FbvX2pBuJLjlPhkpRa",
          "object_id": "pay_w1nwB6UxMcIgEc3UDPvx",
          "event_type": "payment_succeeded",
          "event_class": "payments",
          "is_delivery_successful": false,
          "initial_attempt_id": "evt_018e562fcd4571b0b831098bb9ebe230",
          "created": "2024-03-19T10:09:17.125Z"
        }
      ]

   3. Error when object_id is specified with any of the other query params:

      $ curl -H 'api-key: test_admin' 'http://localhost:8080/events/merchant_1710609960?created_after=2024-03-19T0910:00Z&object_id=123' 2>/dev/null | jq '.'
      {
        "error": {
          "type": "invalid_request",
          "message": "Either only `object_id` must be specified, or one or more of `created_after`, `created_before`, `limit` and `offset` must be specified",
          "code": "IR_16"
        }
      }

4. Listing delivery attempts for a specific initial event ID:

   curl -H 'api-key: {{admin_api_key}}' '{{baseUrl}}/events/{{merchant_id}}/{{event_id}}/attempts'

   $ curl -H 'api-key: test_admin' 'http://localhost:8080/events/merchant_1710609960/evt_018e562fcd4571b0b831098bb9ebe230/attempts'  2>/dev/null | jq '.'
   [
     {
       "event_id": "evt_018e563030bc75a7a682fc3281f1d13d",
       "merchant_id": "merchant_1710609960",
       "profile_id": "pro_H7FbvX2pBuJLjlPhkpRa",
       "object_id": "pay_w1nwB6UxMcIgEc3UDPvx",
       "event_type": "payment_succeeded",
       "event_class": "payments",
       "is_delivery_successful": false,
       "initial_attempt_id": "evt_018e562fcd4571b0b831098bb9ebe230",
       "created": "2024-03-19T10:09:42.592Z",
       "request": {
         "body": "{\"merchant_id\":\"merchant_1710609960\",\"event_id\":\"evt_018e562fcd4571b0b831098bb9ebe230\",\"event_type\":\"payment_succeeded\",\"content\":{\"type\":\"payment_details\",\"object\":{\"payment_id\":\"pay_w1nwB6UxMcIgEc3UDPvx\",\"merchant_id\":\"merchant_1710609960\",\"status\":\"succeeded\",\"amount\":6540,\"net_amount\":6540,\"amount_capturable\":0,\"amount_received\":6540,\"connector\":\"stripe\",\"client_secret\":\"pay_w1nwB6UxMcIgEc3UDPvx_secret_ys9LDfhjJE7wiTQkMQru\",\"created\":\"2024-03-19T10:09:15.060Z\",\"currency\":\"USD\",\"customer_id\":\"StripeCustomer\",\"description\":\"Its my first payment request\",\"refunds\":null,\"disputes\":null,\"mandate_id\":null,\"mandate_data\":null,\"setup_future_usage\":null,\"off_session\":null,\"capture_on\":null,\"capture_method\":\"automatic\",\"payment_method\":\"card\",\"payment_method_data\":{\"card\":{\"last4\":\"4242\",\"card_type\":null,\"card_network\":null,\"card_issuer\":null,\"card_issuing_country\":null,\"card_isin\":\"424242\",\"card_extended_bin\":\"42424242\",\"card_exp_month\":\"10\",\"card_exp_year\":\"25\",\"card_holder_name\":\"joseph Doe\"},\"billing\":null},\"payment_token\":null,\"shipping\":{\"address\":{\"city\":\"San Fransico\",\"country\":\"US\",\"line1\":\"1467\",\"line2\":\"Harrison Street\",\"line3\":\"Harrison Street\",\"zip\":\"94122\",\"state\":\"California\",\"first_name\":\"joseph\",\"last_name\":\"Doe\"},\"phone\":{\"number\":\"8056594427\",\"country_code\":\"+91\"},\"email\":null},\"billing\":{\"address\":{\"city\":\"San Fransico\",\"country\":\"US\",\"line1\":\"1467\",\"line2\":\"Harrison Street\",\"line3\":\"Harrison Street\",\"zip\":\"94122\",\"state\":\"California\",\"first_name\":\"joseph\",\"last_name\":\"Doe\"},\"phone\":{\"number\":\"8056594427\",\"country_code\":\"+91\"},\"email\":null},\"order_details\":null,\"email\":\"[email protected]\",\"name\":\"John Doe\",\"phone\":\"999999999\",\"return_url\":\"https://google.com/\",\"authentication_type\":\"no_three_ds\",\"statement_descriptor_name\":\"joseph\",\"statement_descriptor_suffix\":\"JS\",\"next_action\":null,\"cancellation_reason\":null,\"error_code\":null,\"error_message\":null,\"unified_code\":null,\"unified_message\":null,\"payment_experience\":null,\"payment_method_type\":\"credit\",\"connector_label\":null,\"business_country\":null,\"business_label\":\"default\",\"business_sub_label\":null,\"allowed_payment_method_types\":null,\"ephemeral_key\":{\"customer_id\":\"StripeCustomer\",\"created_at\":1710842955,\"expires\":1710846555,\"secret\":\"epk_ea8a42533ae3472fb32da95c3f616886\"},\"manual_retry_allowed\":false,\"connector_transaction_id\":\"pi_3OvzbcD5R7gDAGff0njPHLCh\",\"frm_message\":null,\"metadata\":{\"udf1\":\"value1\",\"login_date\":\"2019-09-10T10:11:12Z\",\"new_customer\":\"true\"},\"connector_metadata\":null,\"feature_metadata\":null,\"reference_id\":\"pi_3OvzbcD5R7gDAGff0njPHLCh\",\"payment_link\":null,\"profile_id\":\"pro_H7FbvX2pBuJLjlPhkpRa\",\"surcharge_details\":null,\"attempt_count\":1,\"merchant_decision\":null,\"merchant_connector_id\":\"mca_qeS5H2fLkr8GLzN9SNWd\",\"incremental_authorization_allowed\":null,\"authorization_count\":null,\"incremental_authorizations\":null,\"external_authentication_details\":null,\"external_3ds_authentication_attempted\":false,\"expires_on\":\"2024-03-19T10:24:15.060Z\",\"fingerprint\":null,\"payment_method_id\":null,\"payment_method_status\":null}},\"timestamp\":\"2024-03-19T10:09:17.125Z\"}",
         "headers": [
           [
             "content-type",
             "application/json"
           ],
           [
             "X-Webhook-Signature-512",
             "af25f54b7018cc35286451645a3ad0167e28a7fa32417c7411cd655c5db06a4bf1f9f7bd4330f2bed1ef54fedcd5b1fc35b09eb6e50f56b035c90cdf8818df30"
           ]
         ]
       },
       "response": {
         "body": "",
         "headers": [
           [
             "server",
             "mitmproxy 10.2.4"
           ],
           [
             "content-length",
             "0"
           ]
         ],
         "status_code": 404
       },
       "delivery_attempt": "automatic_retry"
     },
     {
       "event_id": "evt_018e562fcd4571b0b831098bb9ebe230",
       "merchant_id": "merchant_1710609960",
       "profile_id": "pro_H7FbvX2pBuJLjlPhkpRa",
       "object_id": "pay_w1nwB6UxMcIgEc3UDPvx",
       "event_type": "payment_succeeded",
       "event_class": "payments",
       "is_delivery_successful": false,
       "initial_attempt_id": "evt_018e562fcd4571b0b831098bb9ebe230",
       "created": "2024-03-19T10:09:17.125Z",
       "request": {
         "body": "{\"merchant_id\":\"merchant_1710609960\",\"event_id\":\"evt_018e562fcd4571b0b831098bb9ebe230\",\"event_type\":\"payment_succeeded\",\"content\":{\"type\":\"payment_details\",\"object\":{\"payment_id\":\"pay_w1nwB6UxMcIgEc3UDPvx\",\"merchant_id\":\"merchant_1710609960\",\"status\":\"succeeded\",\"amount\":6540,\"net_amount\":6540,\"amount_capturable\":0,\"amount_received\":6540,\"connector\":\"stripe\",\"client_secret\":\"pay_w1nwB6UxMcIgEc3UDPvx_secret_ys9LDfhjJE7wiTQkMQru\",\"created\":\"2024-03-19T10:09:15.060Z\",\"currency\":\"USD\",\"customer_id\":\"StripeCustomer\",\"description\":\"Its my first payment request\",\"refunds\":null,\"disputes\":null,\"mandate_id\":null,\"mandate_data\":null,\"setup_future_usage\":null,\"off_session\":null,\"capture_on\":null,\"capture_method\":\"automatic\",\"payment_method\":\"card\",\"payment_method_data\":{\"card\":{\"last4\":\"4242\",\"card_type\":null,\"card_network\":null,\"card_issuer\":null,\"card_issuing_country\":null,\"card_isin\":\"424242\",\"card_extended_bin\":\"42424242\",\"card_exp_month\":\"10\",\"card_exp_year\":\"25\",\"card_holder_name\":\"joseph Doe\"},\"billing\":null},\"payment_token\":null,\"shipping\":{\"address\":{\"city\":\"San Fransico\",\"country\":\"US\",\"line1\":\"1467\",\"line2\":\"Harrison Street\",\"line3\":\"Harrison Street\",\"zip\":\"94122\",\"state\":\"California\",\"first_name\":\"joseph\",\"last_name\":\"Doe\"},\"phone\":{\"number\":\"8056594427\",\"country_code\":\"+91\"},\"email\":null},\"billing\":{\"address\":{\"city\":\"San Fransico\",\"country\":\"US\",\"line1\":\"1467\",\"line2\":\"Harrison Street\",\"line3\":\"Harrison Street\",\"zip\":\"94122\",\"state\":\"California\",\"first_name\":\"joseph\",\"last_name\":\"Doe\"},\"phone\":{\"number\":\"8056594427\",\"country_code\":\"+91\"},\"email\":null},\"order_details\":null,\"email\":\"[email protected]\",\"name\":\"John Doe\",\"phone\":\"999999999\",\"return_url\":\"https://google.com/\",\"authentication_type\":\"no_three_ds\",\"statement_descriptor_name\":\"joseph\",\"statement_descriptor_suffix\":\"JS\",\"next_action\":null,\"cancellation_reason\":null,\"error_code\":null,\"error_message\":null,\"unified_code\":null,\"unified_message\":null,\"payment_experience\":null,\"payment_method_type\":\"credit\",\"connector_label\":null,\"business_country\":null,\"business_label\":\"default\",\"business_sub_label\":null,\"allowed_payment_method_types\":null,\"ephemeral_key\":{\"customer_id\":\"StripeCustomer\",\"created_at\":1710842955,\"expires\":1710846555,\"secret\":\"epk_ea8a42533ae3472fb32da95c3f616886\"},\"manual_retry_allowed\":false,\"connector_transaction_id\":\"pi_3OvzbcD5R7gDAGff0njPHLCh\",\"frm_message\":null,\"metadata\":{\"udf1\":\"value1\",\"login_date\":\"2019-09-10T10:11:12Z\",\"new_customer\":\"true\"},\"connector_metadata\":null,\"feature_metadata\":null,\"reference_id\":\"pi_3OvzbcD5R7gDAGff0njPHLCh\",\"payment_link\":null,\"profile_id\":\"pro_H7FbvX2pBuJLjlPhkpRa\",\"surcharge_details\":null,\"attempt_count\":1,\"merchant_decision\":null,\"merchant_connector_id\":\"mca_qeS5H2fLkr8GLzN9SNWd\",\"incremental_authorization_allowed\":null,\"authorization_count\":null,\"incremental_authorizations\":null,\"external_authentication_details\":null,\"external_3ds_authentication_attempted\":false,\"expires_on\":\"2024-03-19T10:24:15.060Z\",\"fingerprint\":null,\"payment_method_id\":null,\"payment_method_status\":null}},\"timestamp\":\"2024-03-19T10:09:17.125Z\"}",
         "headers": [
           [
             "content-type",
             "application/json"
           ],
           [
             "X-Webhook-Signature-512",
             "af25f54b7018cc35286451645a3ad0167e28a7fa32417c7411cd655c5db06a4bf1f9f7bd4330f2bed1ef54fedcd5b1fc35b09eb6e50f56b035c90cdf8818df30"
           ]
         ]
       },
       "response": {
         "body": "",
         "headers": [
           [
             "server",
             "mitmproxy 10.2.4"
           ],
           [
             "content-length",
             "0"
           ]
         ],
         "status_code": 404
       },
       "delivery_attempt": "initial_attempt"
     }
   ]

Checklist

• I formatted the code cargo +nightly fmt --all
• I addressed lints thrown by cargo clippy
• I reviewed the submitted code
• I added unit tests for my changes where possible
• I added a CHANGELOG entry if applicable

[hrithikesh026]

LGTM

[ThisIsMani]

Dashboard related changes looks fine.

[lsampras]

api event generation part looks good to me