[feature] Optionally Include settings source in ValidationError

[AlbertDeFusco]

When working with settings configured for multiple sources it can be hard to pin down the exact source of a validation error. See #538

In this PR I've added a configuration parameter validate_each_source=True that performs the validation while constructing the merged state of all configured sources. All validation errors are collected and raised after all sources have been parsed. The settings source is prepended to the loc field. Further, type=missing line errors are checked using the fully merged state and added to the raised ValidationError exception.

Here is an example using the Init and Env sources.

import os
from pydantic_settings import BaseSettings

class MySettings(BaseSettings, validate_each_source=True, env_prefix='settings_'):
    foo: str
    default: int = 0
    extra: str | None = None

os.environ['settings_default'] = 'a'
settings = MySettings(extra=0)

The following error message is reported

Traceback (most recent call last):
  File "/Users/adefusco/Development/pydantic/pydantic-settings/testy.py", line 10, in <module>
    settings = MySettings(extra=0)
               ^^^^^^^^^^^^^^^^^^^
  File "/Users/adefusco/Development/pydantic/pydantic-settings/pydantic_settings/main.py", line 182, in __init__
    **self._settings_build_values(
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/adefusco/Development/pydantic/pydantic-settings/pydantic_settings/main.py", line 451, in _settings_build_values
    raise ValidationError.from_exception_data(
pydantic_core._pydantic_core.ValidationError: 3 validation errors for MySettings
InitSettingsSource.extra
  Input should be a valid string [type=string_type, input_value=0, input_type=int]
    For further information visit https://errors.pydantic.dev/2.10/v/string_type
EnvSettingsSource.default
  Input should be a valid integer, unable to parse string as an integer [type=int_parsing, input_value='a', input_type=str]
    For further information visit https://errors.pydantic.dev/2.10/v/int_parsing
foo
  Field required [type=missing, input_value={'default': 'a', 'extra': 0}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.10/v/missing

[hramezani]

Thanks @AlbertDeFusco for this PR.

‌In this PR I've added a configuration parameter validate_each_source=True that performs the validation while constructing the merged state of all configured sources.

I think we decided to have a flag to print out collected values for each source.

[AlbertDeFusco]

I can add that. Maybe an env var like PYDANTIC_SETTINGS_DEBUG_SOURCE_INPUTS=1 to log the parsed source states?

[hramezani]

I can add that. Maybe an env var like PYDANTIC_SETTINGS_DEBUG_SOURCE_INPUTS=1 to log the parsed source states?

Looks good. please make another PR for it.

BTW, I am not sure we want to add the validate_each_source configuration. I can't see any value in validating every source data separately.

[AlbertDeFusco]

I can agree that validating each source is excessive. I'll work on some alternate implementations to provide source information in the ValidationError

[hramezani]

I can agree that validating each source is excessive. I'll work on some alternate implementations to provide source information in the ValidationError

JFYI, At the point of validation, we don't know which data comes from which sources because pydantic-settings merges values from all the sources.