Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reduce memory usage of Analyzer test suite #1429

Merged
merged 8 commits into from
Aug 28, 2024

Conversation

hhobson
Copy link
Contributor

@hhobson hhobson commented Aug 4, 2024

Change Description

Reduce memory usage of Presidio Analyzer test suite (excluding Azure AI language recognizer tests) down to ~6GB from ~8GB. When run locally the tests runtime went from ~60 seconds to ~25 seconds.

Memray memory usgae graphs

Before

Test suite memory usage before

After

Test suite memory usage after

Changes

  • Removed duplicate nlp.load call on each iteration of Stanza and transformers recognizer tests.
  • Paramatized test_when_analyze_with_unsupported_language_must_match tests to avoid calling AnalyzerEngine five times in single test
  • Mocked nlp engine in test_when_analyze_with_unsupported_language_must_match as test doesn't require nlp model to be loaded
  • Removed unused fixtures - pytest-unused-fixtures plugin used to detect them

Issue reference

This PR fixes issue #1427

Checklist

  • I have reviewed the contribution guidelines
  • I have signed the CLA (if required)
  • My code includes unit tests
  • All unit tests and lint checks pass locally
  • My PR contains documentation updates / additions if required

@hhobson
Copy link
Contributor Author

hhobson commented Aug 4, 2024

@microsoft-github-policy-service agree

@omri374
Copy link
Contributor

omri374 commented Aug 4, 2024

Thanks!

@omri374
Copy link
Contributor

omri374 commented Aug 4, 2024

/azp run

Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@omri374
Copy link
Contributor

omri374 commented Aug 4, 2024

/azp run

Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@hhobson hhobson requested a review from omri374 August 27, 2024 17:07
@omri374
Copy link
Contributor

omri374 commented Aug 28, 2024

/azp run

Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@omri374 omri374 merged commit 9696b9e into microsoft:main Aug 28, 2024
32 checks passed
@hhobson hhobson deleted the chore/improve-analyzer-tests branch August 31, 2024 21:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants