Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Call sync_from_system before_sync_workers at startup #23309

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

agrare
Copy link
Member

@agrare agrare commented Jan 15, 2025

#23112 allowed non-rails workers to be marked as started even if they didn't go through run_single_worker with database access, specifically the opentofu-worker.

The issue here is that the threads which keep the list of current deployments and pods up to date are not started by MiqServer#start_workers, rather only the initial startup has completed and MiqServer has moved to the MiqServer#monitor_workers loop. The result of this was everything worked if the server was started with the embedded_terraform role disabled and the role was enabled later, but if the role was enabled at startup the wait_for_started_workers would never complete and timeout.

Fixes ManageIQ/manageiq-providers-embedded_terraform#81

@agrare
Copy link
Member Author

agrare commented Jan 15, 2025

WIP pending a live test to confirm the fix.
Also want to see if there is any reason to not just call sync_from_system at the start of sync_workers instead of having that requirement of every caller.

@agrare agrare force-pushed the fix_pod_watches_not_running_for_start_workers branch from aea5963 to 9fb477a Compare January 17, 2025 16:28
@agrare agrare added the wip label Jan 22, 2025
@agrare
Copy link
Member Author

agrare commented Jan 23, 2025

On master:

{"@timestamp":"2025-01-22T20:18:39.220783","hostname":"orchestrator-6655869579-b5xmt","pid":7,"tid":"96a0","service":"evm","level":"info","message":"MIQ(AutomationWorker#start) Worker started: ID [], PID [], GUID [bc53ddb2-51d2-484f-8ab6-8325db70e469]"}
{"@timestamp":"2025-01-22T20:18:41.545744","hostname":"orchestrator-6655869579-b5xmt","pid":7,"tid":"96a0","service":"evm","level":"info","message":"MIQ(MiqServer::WorkerManagement::Kubernetes#wait_for_started_workers) Waiting for the following workers to start: OpentofuWorker (1)"}
{"@timestamp":"2025-01-22T20:18:51.577956","hostname":"orchestrator-6655869579-b5xmt","pid":7,"tid":"96a0","service":"evm","level":"info","message":"MIQ(MiqServer::WorkerManagement::Kubernetes#wait_for_started_workers) Waiting for the following workers to start: MiqRemoteConsoleWorker (1), OpentofuWorker (1)"}
{"@timestamp":"2025-01-22T20:19:01.583472","hostname":"orchestrator-6655869579-b5xmt","pid":7,"tid":"96a0","service":"evm","level":"info","message":"MIQ(MiqServer::WorkerManagement::Kubernetes#wait_for_started_workers) Waiting for the following workers to start: MiqGenericWorker (1), OpentofuWorker (1)"}
{"@timestamp":"2025-01-22T20:19:11.589923","hostname":"orchestrator-6655869579-b5xmt","pid":7,"tid":"96a0","service":"evm","level":"info","message":"MIQ(MiqServer::WorkerManagement::Kubernetes#wait_for_started_workers) Waiting for the following workers to start: OpentofuWorker (1)"}

With this PR:

{"@timestamp":"2025-01-23T15:28:19.021108","hostname":"orchestrator-5cbd76459f-878rr","pid":7,"tid":"96a0","service":"evm","level":"info","message":"MIQ(AutomationWorker#start) Worker started: ID [], PID [], GUID [7821cbd4-0f10-43a8-9c3b-085721399600]"}
{"@timestamp":"2025-01-23T15:28:19.421110","hostname":"orchestrator-5cbd76459f-878rr","pid":7,"tid":"96a0","service":"evm","level":"info","message":"MIQ(MiqServer::WorkerManagement::Kubernetes#wait_for_started_workers) Waiting for the following workers to start: OpentofuWorker (1)"}
{"@timestamp":"2025-01-23T15:29:29.497771","hostname":"orchestrator-5cbd76459f-878rr","pid":7,"tid":"96a0","service":"evm","level":"info","message":"MIQ(MiqServer::WorkerManagement::Kubernetes#wait_for_started_workers) All workers have been started"}
{"@timestamp":"2025-01-23T15:29:29.503273","hostname":"orchestrator-5cbd76459f-878rr","pid":7,"tid":"96a0","service":"evm","level":"info","message":"MIQ(EvmServer#start_server) Server starting...Complete"}
** Server starting...Complete

@agrare agrare changed the title [WIP] Call sync_from_system before_sync_workers at startup Call sync_from_system before_sync_workers at startup Jan 23, 2025
@agrare agrare removed the wip label Jan 23, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

OpentofuWorker preventing system startup on podified if role enabled at boot
1 participant