Fixes #39359 - Reuse persistent Foreman connections and retry stale POST /register

[pablomh]

Summary

• Replace per-instance Net::HTTP with two class-level persistent connections shared across all ForemanRequest consumers:
  • foreman_http — shared connection for POST /register, templates, and other Foreman-bound traffic
  • registration_get_http — dedicated connection for GET /register cache-miss fetches
• Add configurable timeout settings: foreman_request_timeout (default 200s), foreman_connect_timeout, foreman_keep_alive_timeout (default 10s)
• On transport error, replace the broken Net::HTTP with a fresh instance so the next caller gets a clean connection
• Add a single retry for POST /register on stale connection failures (EOFError, ECONNRESET, etc.), scoped to host_register only — Foreman's controller uses Host.find_or_initialize_by, making a single replay safe

Background

Tested under incremental registration load (76→1140 concurrent hosts, 6 capsules, Satellite --tuning large). Key findings:

• net-http-persistent (PR Fixes #XXXXX - Reuse persistent Foreman connections theforeman/smart-proxy#950) was incompatible with WEBrick's thread-per-request model — connection_pool's thread-local tracking caused CLOSE-WAIT accumulation and ConnectionPool::TimeoutError under load
• Two mutex-guarded singleton connections provide TLS reuse without external gems, with traffic isolation between GET cache fills and POST registration
• Apache event MPM can close keep-alive connections before KeepAliveTimeout when workers are busy, causing stale connection EOFErrors on POST /register
• The POST retry is safe because Foreman's find_host uses Host.find_or_initialize_by(name: ...) — a find-or-create that produces the same result on replay

Commits

1. Reuse persistent Foreman connections via mutex-guarded singletons — the transport change
2. Retry POST /register once on stale connection failure — endpoint-specific retry with body capture

Test plan

• bundle exec ruby -Itest test/request_test.rb — 18 tests, 0 failures
• Full test suite — 952 tests, 0 failures (6 pre-existing inotify/macOS errors)
• RuboCop clean on changed files
• Production load testing on 6 capsules (76→1140 concurrent registrations)

🤖 Generated with Claude Code