feat(analytics): implement currency conversion to power multi-currency aggregation #6418

maverox · 2024-10-23T19:46:05Z

Type of Change

Description

implemented currency_conversion for calculating values of amount related metrics from their currency (i.e INR) to
USD and store it in a separate field variant I created <fieldname>_in_usd
Implemented a total_amount_in_usd variant for all the total fields related to amounts in query Metadata

Fixes #6416

Additional Changes

This PR modifies the API contract
This PR modifies the database schema
This PR modifies application configuration/environment variables

Motivation and Context

The total amount calculated for all payment analytics buckets related to amounts is incorrect because currency conversion is not being applied before summing the values. As a result, amounts in different currencies are being added together, leading to an inaccurate total.

Dependency

This functionality depends on currency_conversion crate directly to fetch exchange rates.

How did you test it?

API request for metrics

curl --location 'http://localhost:8080/analytics/v1/org/metrics/payments' \
--header 'Accept: */*' \
--header 'Accept-Language: en-GB,en-US;q=0.9,en;q=0.8' \
--header 'Connection: keep-alive' \
--header 'Content-Type: application/json' \
--header 'Origin: http://localhost:9000' \
--header 'QueryType: SingleStat' \
--header 'Referer: http://localhost:9000/' \
--header 'Sec-Fetch-Dest: empty' \
--header 'Sec-Fetch-Mode: cors' \
--header 'Sec-Fetch-Site: same-site' \
--header 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36' \
--header 'api-key: dev_r3kVZ4URj3fMiHhJJuhHOVqteaXwKyEMGCaFo6PiCS6S9vZsb0ErV4kCVqogM60H' \
--header 'authorization: Bearer eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJ1c2VyX2lkIjoiMDZlOGRmZWYtMzI0Zi00Yjc0LTg2MTItYzdkZjNhOGZmZTcyIiwibWVyY2hhbnRfaWQiOiJtZXJjaGFudF8xNzI3NDM0NTkzIiwicm9sZV9pZCI6Im9yZ19hZG1pbiIsImV4cCI6MTcyOTgzNzUwNywib3JnX2lkIjoib3JnXzl3Y0kxQ2hTOGVEdnRRQmRXak9IIiwicHJvZmlsZV9pZCI6InByb19FNW5lek43YjZUbVB1WlUzbEU1VSJ9.KlBRpu8DpGZ43u9hsB4xE2oSXNM21HC0RadRdfMe2VI' \
--header 'sec-ch-ua: "Chromium";v="130", "Google Chrome";v="130", "Not?A_Brand";v="99"' \
--header 'sec-ch-ua-mobile: ?0' \
--header 'sec-ch-ua-platform: "macOS"' \
--data '[
    {
        "timeRange": {
            "startTime": "2024-10-13T18:30:00Z",
            "endTime": "2024-10-21T12:54:00Z"
        },
        "mode": "ORDER",
        "source": "BATCH",
        "metrics": [
            
            
            
            
            "payment_processed_amount"
        ],
        "delta": true
    }
]'

API response

{
    "queryData": [
        {
            "payment_success_rate": null,
            "payment_count": null,
            "payment_success_count": null,
            "payment_processed_amount": 150000,
            "payment_processed_amount_usd": 1782,
            "payment_processed_count": null,
            "payment_processed_amount_without_smart_retries": 0,
            "payment_processed_amount_without_smart_retries_usd": 0,
            "payment_processed_count_without_smart_retries": null,
            "avg_ticket_size": null,
            "payment_error_message": null,
            "retries_count": null,
            "retries_amount_processed": 0,
            "connector_success_rate": null,
            "payments_success_rate_distribution": null,
            "payments_success_rate_distribution_without_smart_retries": null,
            "payments_failure_rate_distribution": null,
            "payments_failure_rate_distribution_without_smart_retries": null,
            "failure_reason_count": 0,
            "failure_reason_count_without_smart_retries": 0,
            "currency": "INR",
            "status": null,
            "connector": null,
            "authentication_type": null,
            "payment_method": null,
            "payment_method_type": null,
            "client_source": null,
            "client_version": null,
            "profile_id": null,
            "card_network": null,
            "merchant_id": null,
            "card_last_4": null,
            "card_issuer": null,
            "error_reason": null,
            "time_range": {
                "start_time": "2024-10-10T18:30:00.000Z",
                "end_time": "2024-10-18T08:24:00.000Z"
            },
            "time_bucket": "2024-10-10 18:30:00"
        },
        {
            "payment_success_rate": null,
            "payment_count": null,
            "payment_success_count": null,
            "payment_processed_amount": 1794006540,
            "payment_processed_amount_usd": 1794006540,
            "payment_processed_count": null,
            "payment_processed_amount_without_smart_retries": 0,
            "payment_processed_amount_without_smart_retries_usd": 0,
            "payment_processed_count_without_smart_retries": null,
            "avg_ticket_size": null,
            "payment_error_message": null,
            "retries_count": null,
            "retries_amount_processed": 0,
            "connector_success_rate": null,
            "payments_success_rate_distribution": null,
            "payments_success_rate_distribution_without_smart_retries": null,
            "payments_failure_rate_distribution": null,
            "payments_failure_rate_distribution_without_smart_retries": null,
            "failure_reason_count": 0,
            "failure_reason_count_without_smart_retries": 0,
            "currency": "USD",
            "status": null,
            "connector": null,
            "authentication_type": null,
            "payment_method": null,
            "payment_method_type": null,
            "client_source": null,
            "client_version": null,
            "profile_id": null,
            "card_network": null,
            "merchant_id": null,
            "card_last_4": null,
            "card_issuer": null,
            "error_reason": null,
            "time_range": {
                "start_time": "2024-10-10T18:30:00.000Z",
                "end_time": "2024-10-18T08:24:00.000Z"
            },
            "time_bucket": "2024-10-10 18:30:00"
        },
        {
            "payment_success_rate": null,
            "payment_count": null,
            "payment_success_count": null,
            "payment_processed_amount": 6540,
            "payment_processed_amount_usd": 7118,
            "payment_processed_count": null,
            "payment_processed_amount_without_smart_retries": 0,
            "payment_processed_amount_without_smart_retries_usd": 0,
            "payment_processed_count_without_smart_retries": null,
            "avg_ticket_size": null,
            "payment_error_message": null,
            "retries_count": null,
            "retries_amount_processed": 0,
            "connector_success_rate": null,
            "payments_success_rate_distribution": null,
            "payments_success_rate_distribution_without_smart_retries": null,
            "payments_failure_rate_distribution": null,
            "payments_failure_rate_distribution_without_smart_retries": null,
            "failure_reason_count": 0,
            "failure_reason_count_without_smart_retries": 0,
            "currency": "EUR",
            "status": null,
            "connector": null,
            "authentication_type": null,
            "payment_method": null,
            "payment_method_type": null,
            "client_source": null,
            "client_version": null,
            "profile_id": null,
            "card_network": null,
            "merchant_id": null,
            "card_last_4": null,
            "card_issuer": null,
            "error_reason": null,
            "time_range": {
                "start_time": "2024-10-10T18:30:00.000Z",
                "end_time": "2024-10-18T08:24:00.000Z"
            },
            "time_bucket": "2024-10-10 18:30:00"
        }
    ],
    "metaData": [
        {
            "total_payment_processed_amount": 1794163080,
            "total_payment_processed_amount_usd": 1794015440,
            "total_payment_processed_amount_without_smart_retries": 0,
            "total_payment_processed_amount_without_smart_retries_usd": 0,
            "total_payment_processed_count": 0,
            "total_payment_processed_count_without_smart_retries": 0,
            "total_failure_reasons_count": 0,
            "total_failure_reasons_count_without_smart_retries": 0
        }
    ]
}

Testing for various other scenarios

by tweaking `development.toml` we can create these scenarios

[forex_api]
call_delay = 30
local_fetch_retry_count = 5
local_fetch_retry_delay = 1000
api_timeout = 20000
api_key = "YOUR PRIMARY API KEY HER" # set this value to anything else to invalidate it
fallback_api_key = "YOUR FALLBACK API KEY HER" # set this value to anything else to invalidate it
redis_lock_timeout = 2600

by setting call_delay to 30s we can check for the fetch cycle and redis updates.

When primary api isn't working (invalidating the primary api key)

API response : 200

when both keys are invalidated

API response : 500

Checklist

I formatted the code cargo +nightly fmt --all
I addressed lints thrown by cargo clippy
I reviewed the submitted code
I added unit tests for my changes where possible

- create get_conversion_rates() function - change get_metrics() to accept ExchangeRates hashmap

…lyticsMetadata and PaymentAnalyticsBucketValue

…ep and rust_decimal as 3rd party dep

…ts and payment_intents amount metrics

…tors for both payments and payment_intents

…or payments and payments_intents

semanticdiff-com · 2024-10-23T19:46:09Z

Review changes with

Changed Files

File	Status
crates/router/src/analytics.rs	70% smaller
crates/analytics/src/payment_intents/accumulator.rs	51% smaller
crates/analytics/src/payments/accumulator.rs	47% smaller
crates/router/src/utils/currency.rs	34% smaller
crates/analytics/src/payment_intents/core.rs	2% smaller
crates/analytics/src/payments/core.rs	1% smaller
Cargo.lock	Unsupported file format
crates/analytics/Cargo.toml	Unsupported file format
crates/analytics/src/errors.rs	0% smaller
crates/analytics/src/payment_intents/metrics/sessionized_metrics/payment_processed_amount.rs	0% smaller
crates/analytics/src/payment_intents/metrics/sessionized_metrics/smart_retried_amount.rs	0% smaller
crates/analytics/src/payment_intents/metrics/smart_retried_amount.rs	0% smaller
crates/analytics/src/payments/metrics/payment_processed_amount.rs	0% smaller
crates/analytics/src/payments/metrics/sessionized_metrics/payment_processed_amount.rs	0% smaller
crates/api_models/src/analytics.rs	0% smaller
crates/api_models/src/analytics/payment_intents.rs	0% smaller
crates/api_models/src/analytics/payments.rs	0% smaller
crates/router/src/core/currency.rs	0% smaller

crates/analytics/Cargo.toml

crates/analytics/src/payment_intents/core.rs

crates/analytics/src/payments/accumulator.rs

crates/analytics/src/payments/core.rs

…n calls

- fallback api doesnt return a USD factor of 1 inside the rates field like the primary api's response, which was causing different response when using fall_back api for fetching exchange rates.

crates/analytics/src/payment_intents/core.rs

crates/analytics/src/payment_intents/accumulator.rs

…y aggregation (#6418) Co-authored-by: hyperswitch-bot[bot] <148525504+hyperswitch-bot[bot]@users.noreply.github.com>

maverox added 7 commits October 24, 2024 00:11

feat(analytics): implement currency conversion for amount related metric

3613f60

- create get_conversion_rates() function - change get_metrics() to accept ExchangeRates hashmap

feat(analytics): add in_usd variants of amount fields in PaymentAna…

cd09f40

…lyticsMetadata and PaymentAnalyticsBucketValue

chore(analytics): add currency_conversion, euclid_wasm as 1st party d…

46d7893

…ep and rust_decimal as 3rd party dep

feat(analytics): add currency as select and group by clause in paymen…

06eeb9e

…ts and payment_intents amount metrics

feat(analytics): add the fields and query metadata fields in accumula…

ea3220a

…tors for both payments and payment_intents

feat(analytics): implement amount conversion and accumulation logic f…

c319fb5

…or payments and payments_intents

chore(analytics): commit lock file for analytics crate

b3dca6d

maverox self-assigned this Oct 23, 2024

maverox requested review from a team as code owners October 23, 2024 19:46

maverox added C-feature Category: Feature request or enhancement A-Analytics labels Oct 23, 2024

chore: run formatter

7b80f62

maverox added the Rust Pull requests that update Rust code label Oct 23, 2024

tsdk02 previously approved these changes Oct 25, 2024

View reviewed changes

SanchithHegde reviewed Oct 28, 2024

View reviewed changes

tsdk02 self-requested a review October 28, 2024 11:12

maverox added 3 commits October 29, 2024 01:24

feat(analytics): add error handling/logging during currency conversio…

218e242

…n calls

fix(currency_conversion): add USD factor in response of fall_back api

cf94a25

- fallback api doesnt return a USD factor of 1 inside the rates field like the primary api's response, which was causing different response when using fall_back api for fetching exchange rates.

build(deps): remove unused dep euclid_wasm

31adef5

maverox dismissed tsdk02’s stale review via 31adef5 October 28, 2024 20:00

SanchithHegde reviewed Oct 29, 2024

View reviewed changes

crates/analytics/src/payment_intents/core.rs Show resolved Hide resolved

feat(analytics): improve error logging

7ce8ed5

SanchithHegde approved these changes Nov 3, 2024

View reviewed changes

tsdk02 approved these changes Nov 4, 2024

View reviewed changes

Chethan-rao approved these changes Nov 4, 2024

View reviewed changes

crates/analytics/src/payment_intents/accumulator.rs Show resolved Hide resolved

Gnanasundari24 added this pull request to the merge queue Nov 5, 2024

github-merge-queue bot removed this pull request from the merge queue due to failed status checks Nov 5, 2024

Gnanasundari24 added this pull request to the merge queue Nov 6, 2024

Merged via the queue into main with commit 01c5216 Nov 6, 2024
24 of 25 checks passed

Gnanasundari24 deleted the currency-conversion-analytics branch November 6, 2024 09:51

maverox mentioned this pull request Nov 8, 2024

docs(analytics): add setup instructions for currency_conversion service #6516

Merged

14 tasks

bsayak03 pushed a commit that referenced this pull request Nov 26, 2024

feat(analytics): implement currency conversion to power multi-currenc…

9d32765

…y aggregation (#6418) Co-authored-by: hyperswitch-bot[bot] <148525504+hyperswitch-bot[bot]@users.noreply.github.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(analytics): implement currency conversion to power multi-currency aggregation #6418

feat(analytics): implement currency conversion to power multi-currency aggregation #6418

maverox commented Oct 23, 2024 •

edited

Loading

semanticdiff-com bot commented Oct 23, 2024 •

edited

Loading

feat(analytics): implement currency conversion to power multi-currency aggregation #6418

feat(analytics): implement currency conversion to power multi-currency aggregation #6418

Conversation

maverox commented Oct 23, 2024 • edited Loading

Type of Change

Description

Fixes #6416

Additional Changes

Motivation and Context

Dependency

How did you test it?

API request for metrics

API response

Testing for various other scenarios

by tweaking development.toml we can create these scenarios

API response : 200

API response : 500

Checklist

semanticdiff-com bot commented Oct 23, 2024 • edited Loading

maverox commented Oct 23, 2024 •

edited

Loading

by tweaking `development.toml` we can create these scenarios

semanticdiff-com bot commented Oct 23, 2024 •

edited

Loading