ITS Services Status
The F5’s have been restored to the previous OS. We believe this will resolve the issue with the failover events. We will continue to monitor the systems.
We were unable to download the hotfix. We will roll the load balancers back to the previous now. We will try to again to download the hotfix tomorrow.
The vendor believes that our problem is a known issue and has provided an engineering hotfix for that issue. We are downloading it now. If time allows, we will attempt to install the hotfix before midnight. However, we need to avoid interfering with current maintenance on another system which uses the load balancers.
The investigation continues. The vendor has confirmed that the issue is occurring on both units.
Since our initial report, there have been three additional failover events. We are continuing to run on the new OS to collect more information for diagnosis. If the issue is not resolved by 10pm, we will roll back to the previous OS.
The vendor has been contacted. System configuration and log files have been uploaded to the vendor’s diagnostic site. The support engineer identified core files that had resulted from crashing processes. The engineer is currently researching the stack traces and will contact us when he has more information.
Starting an hour and twenty minutes after upgrading the OS on the F5 load balancers, we began to observe random failover events between the two units. It appears to be a serious bug in the new OS. We are engaging vendor support.
If we are unable to correct the problem, We will have emergency maintenance later this evening to roll back to the previous OS.
Last updated on Apr 17, 2013 2:56 PM