LLMs are great calibration tools. They are extremely cost-effective for this type of work.
In many systems, situations can be detected deterministically: new intent patterns, traffic shifts, error clusters, or edge-case contexts.
When a new situation is detected, use the LLM to update the policy for that context. Then run that policy until the next shift. This keeps responses appropriate across changing environments without high operating cost.