Autoscale VMs do not follow ScaleDown Rules after a Node Failure

[btzq]

ISSUE TYPE

• Bug Report

COMPONENT NAME

Autoscale

CLOUDSTACK VERSION

4.19.0

CONFIGURATION

OS / ENVIRONMENT

SUMMARY

STEPS TO REPRODUCE

We are actively using Autoscale Groups which have the following scale down rules.

Autoscale Rule

• Name: ScaleDownPolicy-0
• Duration (in sec) = 60 Seconds
• Quiet Time (in sec) = 30 seconds

Conditions:

• Counter: VM CPU - Average Percentage
• Operator: Less Than
• Threshold: 35

These rules work well on normal days. But today, we had a node failure, which the node was hosting some autoscaleVMs

As a result, we experienced this issue, which have already been reported:

• Autoscale Load Balancing Rule looses VMs during a node failure. #9145

In the case where the VMs were not 'Orphaned', and managed to find its way back to the Autoscale Group (or just werent affected), i noticed that the scale down rule did not work anymore.

I had VMs that:

• Min : 2 Members
• Max : 6 Members
• Available Instances: 6 <- This should be 2 Instead.

And after 5 minutes, it still did not scale down. All VMs will running.
Ive verified the CPU Utilisation of all VMs were only 1% consistently.

In order to have the service resume, i had to:

• Disable the Autoscale Group
• Delete the VMs
• Re-enable the Autoscale VM

After that, the scaledown rule works, and i had the setup below again:

I had VMs that:

• Min : 2 Members
• Max : 6 Members
• Available Instances: 2

NA

EXPECTED RESULTS

NA

ACTUAL RESULTS

In the case where the VMs were not 'Orphaned', and managed to find its way back to the Autoscale Group (or just happened to be in a host that was not affected), i noticed that the scale down rule did not work anymore.

[weizhouapache]

@btzq
thanks for reporting
can we/you close one of the two issues ?

[btzq]

@weizhouapache which issue are you refering to?

I reported 3 issues regarding autoscaling so far, but they seem to be 3 isolated issues?

• Autoscale VMs do not follow ScaleDown Rules after a Node Failure #9336
• Autoscale Group creates Infinite Number Of VMs when unable to startup #9318
• Autoscale Load Balancing Rule looses VMs during a node failure. #9145

If you are refering to #9145 , then i need to point out that that ticket was reporting an issue where the VM can become orphans.

In this ticket #9336 , this ticket reports that in either scenarios (including situations where Orphan scenario doesnt take place), the scale down rule doesnt work anymore.

[weizhouapache]

@weizhouapache which issue are you refering to?

I reported 3 issues regarding autoscaling so far, but they seem to be 3 isolated issues?

• Autoscale VMs do not follow ScaleDown Rules after a Node Failure #9336
• Autoscale Group creates Infinite Number Of VMs when unable to startup #9318
• Autoscale Load Balancing Rule looses VMs during a node failure. #9145

If you are refering to #9145 , then i need to point out that that ticket was reporting an issue where the VM can become orphans.

In this ticket #9336 , this ticket reports that in either scenarios (including situations where Orphan scenario doesnt take place), the scale down rule doesnt work anymore.

thanks for the explanation @btzq

we need to investigate them

[btzq]

Hi @DaanHoogland @weizhouapache , we just encountered this issue again after one of our compute servers died.... Please dont put this into deprecated cause it is a very real issue!

[DaanHoogland]

@btzq , a new githubfeature is sub-issue so you can group related issues ;) hope you are willing to try it.
#9318 has a possible fix , #9574. Can you test that?

I'll see if I can reproduce the other issues

[DaanHoogland]

@btzq , I replayed what I think is your scenario:

• two hosts
• an autoscalegroup with
• min 2, max 6, 30 secs for all timers
• upscale on 50% cpu
• downscale on 35% cpu
  using dd zero>null I created load
  by doing this on all vms I got to the maximum
  I stopped one of the hosts
  extra VMs were being created (note that this is strange behaviour as no VMs were disguarded as being part of the group)
  I started the host again
  the VMs were marked as stopped and the group was now 9 big.
  stopping all the dd jobs the load got less and VMS were being stopped and expunged
  starting the stopped VMs these were also expunged one by one normalising the situation.

I will investigate some more, but I think it works (more or less) as expected. Can you point to any differences in my scenario with respect to your situation?