Back in the day, If i recall, when we went in and modified an agent policy we had an option to save, but not to push out those changes immediately. Currently if you go in and modify a policy, the save button automatically forces an update to all clients running that policy. We've found this to have a very bad effect on all of our servers which are VM's running the Shavlik Agent.
Here is what happens:
- Edit an Agent Policy
- Go into the patching section, change a patch task configured in the agent policy to use a different patch scan template. (We do this every month as we release our patches in packages which are defined by using different patch scan templates).
- Click save.
Immediately this is forced out to all clients running this policy. On workstations this is no big deal but when it comes to virtual machines, it's causing issues. We recently set up Veeam ONE to monitor the performance of our systems and last night was the first time I made a change to a server policy in Shavlik since implementing Veeam ONE. As soon as I hit save, I started to get Datastore latency alarms for tons of VM's, too many to count. This latency was high as well, over thresholds by a large margin which will cause production issues to our end users (slowness). I think what is happening is that all of the VM's are receiving this update at the same time and either downloading something from the Shavlik server or processing something off of their own disk which is just too much at once to handle and keep performance up.
My suggestions:
- Bring back the Save but don't update button and allow the update to go out the next time the machine running the modified policy checks in.
- Build in some form of stagger so that hundreds of machines aren't hit with the update at the same time causing massive disk IO operations at once.
Please let me know if you have any questions.
