Provide machine-friendly, streaming output format

[davidmatson]

Checklist

• Feature request has a meaningful title
• I have searched the existing issues. See all issues
• I have tested using the latest version of Pester. See Installation and update guide.

Summary of the feature request

Our ES testing infrastructure has its own reporting mechanism, and we've previously integrated with many other test frameworks, including GoogleTest, Catch2, Boost.Test, VSTest / xUnit.Net / NUnit / MSTest. Most of these frameworks provide a built-in output format that is streamed (provides the data as the tests are running) as well as programmatically-processible (XML written to stdout for Catch2 and Boost.Test, URL-like lines written to a socket for GoogleTest). Could Pester provide such an output format as well?

This would provide a mechanism that could be used to adapt Pester test results to any other format, while the Pester tests are running. Runners such as VS Code might also make use of this format as a stable way of presenting results, rather than needing the currently-non-public add-in mechanism.

How should it work?

Invoke-Pester (or the configuration object) provides a way to specify that the output mode is machine-friendly (rather than the default of human-friendly). Perhaps a new Output.Mode on PesterConfiguration, with values of 'Human' (default) and 'Machine' (new), with machine output looking something like this (inspired by the GoogleTest machine-friendly output format):

pester_streaming_protocol_version=1.0
event=PesterStart&version=6.0.0
event=DiscoveryStart
event=Discovered&path=C:\t\Planets\Get-Planet.Tests.ps1
event=DiscoveryEnd
event=RunStart&path=C:\t\Planets\Get-Planet.Tests.ps1
event=DescribeStart&name=Get-Planet
event=ItStart&name=Given%20no%20parameters,%20it%20lists%20all%208%20planets
event=ItEnd&result=fail&duration=40ms&message=The%20term...&file=C:\t\Planets\Get-Planet.Tests.ps1&line=7
event=DescribeEnd
event=RunEnd

[davidmatson]

Other options that could work for the same scenario:

XML to stdout:
Parsing streaming XML (no DOM, as the file isn't fully written yet) is harder than the format above, but Catch2 and Boost.Test use this mechanism, and it could work.

Add-in:
This requires getting the caller involved more in the execution environment than passing a switch and consuming a programmatic format from stdout. We had to use this approach for VSTest, and it was much less flexible for execution environment integration.

[davidmatson]

Related to #550 and #447 - this might work for both of those cases as well as others.

[davidmatson]

@nohwnd - thoughts on this feature? We'd be interested in working on an implementation draft, if there's agreement that this would be a good idea.

[nohwnd]

I can't promise anything right now, I work on v6 when I can, and my time is limited.

This does not look like a feature that would be usable to everyone. Do you see some uses for this that would allow us to support some new exciting thing, rather than just your own reporting mechanism? e.g better ide integration in vscode.

[fflaten]

Also related to #2021 so you could add or replace with custom output.

[davidmatson]

@nohwnd - yes, I'd imagine VSCode integration could use this rather than the currently-unsupported hook that it's currently using to show progress in the IDE as tests are executing. Thoughts?

[cdhunt]

@nohwnd Could you emit an Event at the end of Invoke-Block that has $block as the data? I'm looking to try and do something similar to this xUnit Otel instrumentation. That would allow a user access to subscribe to the raw object and should provide some useful functionality for IDEs - though I don't know how the VSCode Pester integrations work currently.

[cdhunt]

I did a crude POC.

Test

Describe "Pester Otel integration" {
    It "fake test" {
        $true | Should -BeTrue
        New-Event -SourceIdentifier Pester -Sender $____Pester.CurrentBlock.Name -MessageData $____Pester.CurrentBlock
    }
}

Data

$Events = Get-Event -SourceIdentifier Pester
$Events[-1].MessageData | select Name, Passed

Name                    Passed
----                    ------
Pester Otel integration   True

[nohwnd]

The current vscode integration that Justin did registers a plugin into the standard plugin subsystem we have. Can’t share details now, but will try to do some coding / maintenance on pester in the evening, in an hour or so.

…

-j

________________________________ Od: Chris Hunt ***@***.***> Odesláno: Wednesday, January 8, 2025 7:01:16 PM Komu: pester/Pester ***@***.***> Kopie: Comment ***@***.***>; Subscribed ***@***.***> Předmět: Re: [pester/Pester] Provide machine-friendly, streaming output format (Issue #2581) I did a crude POC. Test Describe "Pester Otel integration" { It "fake test" { $true | Should -BeTrue New-Event -SourceIdentifier Pester -Sender $____Pester.CurrentBlock.Name -MessageData $____Pester.CurrentBlock } } Data $Events = Get-Event -SourceIdentifier Pester $Events[-1].MessageData | select Name, Passed Name Passed ---- ------ Pester Otel integration True — Reply to this email directly, view it on GitHub<#2581 (comment)> or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABLYLYOF3VAYYNJT6NXQTOT2JVRW3BFKMF2HI4TJMJ2XIZLTSSBKK5TBNR2WLJDUOJ2WLJDOMFWWLO3UNBZGKYLEL5YGC4TUNFRWS4DBNZ2F6YLDORUXM2LUPGBKK5TBNR2WLJDUOJ2WLJDOMFWWLLTXMF2GG2C7MFRXI2LWNF2HTAVFOZQWY5LFUVUXG43VMWSG4YLNMWVXI2DSMVQWIX3UPFYGLAVFOZQWY5LFVEZDAMJYGYZTKOJXURXGC3LFVFUGC427NRQWEZLMVRZXKYTKMVRXIX3UPFYGLLCJONZXKZKDN5WW2ZLOOSTHI33QNFRXHE4CUR2HS4DFVJZGK4DPONUXI33SPGSXMYLMOVS2OMJSGEZTINZUQKSHI6LQMWSWS43TOVS2K5TBNR2WLKRSGY3TOMRRGAYDKMECUR2HS4DFUVWGCYTFNSSXMYLMOVS2SMRQGE4DMMZVHE32O5DSNFTWOZLSUZRXEZLBORSQ>. You are receiving this email because you commented on the thread. Triage notifications on the go with GitHub Mobile for iOS<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[cdhunt]

There's a plugin subsystem?

[nohwnd]

Yes internaly most of pester functionality is implemeted as plugins that subscribe to “events” coming from the main engine. So we don’t mix up the order of executing them. Can handle failures in consistent way and handle scoping / isolation correctly. New-pesterPlugin is imho how the function that produces plugin object is called, if you find that it should be more obvios what is going on.

…

-j

________________________________ Od: Chris Hunt ***@***.***> Odesláno: Wednesday, January 8, 2025 7:34:23 PM Komu: pester/Pester ***@***.***> Kopie: Comment ***@***.***>; Subscribed ***@***.***> Předmět: Re: [pester/Pester] Provide machine-friendly, streaming output format (Issue #2581) There's a plugin subsystem? — Reply to this email directly, view it on GitHub<#2581 (comment)> or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABLYLYJFOYL547MLN6SHKCL2JVVS7BFKMF2HI4TJMJ2XIZLTSSBKK5TBNR2WLJDUOJ2WLJDOMFWWLO3UNBZGKYLEL5YGC4TUNFRWS4DBNZ2F6YLDORUXM2LUPGBKK5TBNR2WLJDUOJ2WLJDOMFWWLLTXMF2GG2C7MFRXI2LWNF2HTAVFOZQWY5LFUVUXG43VMWSG4YLNMWVXI2DSMVQWIX3UPFYGLAVFOZQWY5LFVEZDAMJYGYZTKOJXURXGC3LFVFUGC427NRQWEZLMVRZXKYTKMVRXIX3UPFYGLLCJONZXKZKDN5WW2ZLOOSTHI33QNFRXHE4CUR2HS4DFVJZGK4DPONUXI33SPGSXMYLMOVS2OMJSGEZTINZUQKSHI6LQMWSWS43TOVS2K5TBNR2WLKRSGY3TOMRRGAYDKMECUR2HS4DFUVWGCYTFNSSXMYLMOVS2SMRQGE4DMMZVHE32O5DSNFTWOZLSUZRXEZLBORSQ>. You are receiving this email because you commented on the thread. Triage notifications on the go with GitHub Mobile for iOS<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[nohwnd]

Register callbacks, rather that subscribe to events.

…

-j

________________________________ Od: Jakub Jareš ***@***.***> Odesláno: Wednesday, January 8, 2025 7:37:26 PM Komu: ***@***.*** ***@***.***>; pester/Pester ***@***.***> Kopie: Comment ***@***.***>; Subscribed ***@***.***> Předmět: Re: [pester/Pester] Provide machine-friendly, streaming output format (Issue #2581) Yes internaly most of pester functionality is implemeted as plugins that subscribe to “events” coming from the main engine. So we don’t mix up the order of executing them. Can handle failures in consistent way and handle scoping / isolation correctly. New-pesterPlugin is imho how the function that produces plugin object is called, if you find that it should be more obvios what is going on.

-j

________________________________ Od: Chris Hunt ***@***.***> Odesláno: Wednesday, January 8, 2025 7:34:23 PM Komu: pester/Pester ***@***.***> Kopie: Comment ***@***.***>; Subscribed ***@***.***> Předmět: Re: [pester/Pester] Provide machine-friendly, streaming output format (Issue #2581) There's a plugin subsystem? — Reply to this email directly, view it on GitHub<#2581 (comment)> or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABLYLYJFOYL547MLN6SHKCL2JVVS7BFKMF2HI4TJMJ2XIZLTSSBKK5TBNR2WLJDUOJ2WLJDOMFWWLO3UNBZGKYLEL5YGC4TUNFRWS4DBNZ2F6YLDORUXM2LUPGBKK5TBNR2WLJDUOJ2WLJDOMFWWLLTXMF2GG2C7MFRXI2LWNF2HTAVFOZQWY5LFUVUXG43VMWSG4YLNMWVXI2DSMVQWIX3UPFYGLAVFOZQWY5LFVEZDAMJYGYZTKOJXURXGC3LFVFUGC427NRQWEZLMVRZXKYTKMVRXIX3UPFYGLLCJONZXKZKDN5WW2ZLOOSTHI33QNFRXHE4CUR2HS4DFVJZGK4DPONUXI33SPGSXMYLMOVS2OMJSGEZTINZUQKSHI6LQMWSWS43TOVS2K5TBNR2WLKRSGY3TOMRRGAYDKMECUR2HS4DFUVWGCYTFNSSXMYLMOVS2SMRQGE4DMMZVHE32O5DSNFTWOZLSUZRXEZLBORSQ>. You are receiving this email because you commented on the thread. Triage notifications on the go with GitHub Mobile for iOS<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[cdhunt]

That would be simple enough to adapt for my purposes.

[nohwnd]

This is the change we did for justin to allow adding external plugins: #2020

…

-j

________________________________ Od: Chris Hunt ***@***.***> Odesláno: Wednesday, January 8, 2025 7:56:10 PM Komu: pester/Pester ***@***.***> Kopie: Comment ***@***.***>; Subscribed ***@***.***> Předmět: Re: [pester/Pester] Provide machine-friendly, streaming output format (Issue #2581) That would be simple enough to adapt for my purposes. — Reply to this email directly, view it on GitHub<#2581 (comment)> or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABLYLYNXPE6QKTUNJASBWHD2JVYEVBFKMF2HI4TJMJ2XIZLTSSBKK5TBNR2WLJDUOJ2WLJDOMFWWLO3UNBZGKYLEL5YGC4TUNFRWS4DBNZ2F6YLDORUXM2LUPGBKK5TBNR2WLJDUOJ2WLJDOMFWWLLTXMF2GG2C7MFRXI2LWNF2HTAVFOZQWY5LFUVUXG43VMWSG4YLNMWVXI2DSMVQWIX3UPFYGLAVFOZQWY5LFVEZDAMJYGYZTKOJXURXGC3LFVFUGC427NRQWEZLMVRZXKYTKMVRXIX3UPFYGLLCJONZXKZKDN5WW2ZLOOSTHI33QNFRXHE4CUR2HS4DFVJZGK4DPONUXI33SPGSXMYLMOVS2OMJSGEZTINZUQKSHI6LQMWSWS43TOVS2K5TBNR2WLKRSGY3TOMRRGAYDKMECUR2HS4DFUVWGCYTFNSSXMYLMOVS2SMRQGE4DMMZVHE32O5DSNFTWOZLSUZRXEZLBORSQ>. You are receiving this email because you commented on the thread. Triage notifications on the go with GitHub Mobile for iOS<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.