feature/monitor-auto-restart

[bgunnar5]

This PR adds in the monitor auto-restart functionality that we've been discussing @lucpeterson @ryannova @rydtran . It lays the foundation for future changes I'd like to make in Merlin involving database integration and a more OOP approach to handling backend/broker connections.

No tests have been written yet as an alpha was requested ASAP. I can add documentation and tests in the days to come but for now it's probably important to get Bogdan and Ryan Tran using this and providing feedback. If we don't want to create a full alpha release just yet, maybe we can have them just use this branch for now until I get all tests and documentation finished. I'll leave that up for discussion.

[bgunnar5]

Tests are failing due to Python 3.7 and the linter now using Python 3.13. Maestro is dropping support for 3.7 so I think it's time Merlin does as well. I will work on this more tomorrow.

[ryanadtran]

I will go build your branch locally and test it.

[bgunnar5]

@ryanadtran Ok, I'm in the process of writing some additional documentation too. For now, just know that you may want to run merlin database delete all-runs prior to submitting a new run. This just clears the database of existing runs so the monitor will start fresh each time.

[bgunnar5]

Updated docs for this PR can be found here: https://merlin--494.org.readthedocs.build/en/494/. The new monitor stuff (including a flow chart) is at: https://merlin--494.org.readthedocs.build/en/494/user_guide/monitoring/monitor_for_allocation/. Additionally, documentation on the new merlin database command can be found here: https://merlin--494.org.readthedocs.build/en/494/user_guide/monitoring/database/.

I still need to add a reference guide for the database architecture and update the "What is a results backend?" section of the docs but I don't have the brain power to do any more documentation tonight. Using this as more of a note to myself than anything.