They run up huge bills and upset users
Mobile network operators are burning fossil fuels and running up huge power bills in thrall to Zombie Computers, rogue servers that could be too risky to take out, according to a Red Hat expert who warned that cloud computing will offer unproductive machinery an even more effective screen from which to hide. Zombie-hunter Holly Cummins, Quarkus senior principal software engineer at RedHat, shared her ‘Kill Bill’ tips with developers with Q Con London. The top line is that the use of applications like FinOps and LightSwitchOps could immediately save telcos a fortune and halt customer defections.
Cummins said a zombie is a box that’s served no information or computing for at least six months. They are not just legacy boxes hiding in a nest of cables but surprisingly modern systems whose presence is undocumented. There’s the legendary Kubernetes cluster schmuck that consumed thousands of dollars for no apparent reason. Sometimes there’ll be ‘orphans’ that nobody claims ownership of. They’re impossible to see and even harder to measure.
A study found that most energy in US data centres powered 12 million servers that do nothing most of the time. On average a server uses 15% capacity and 45% max power. In 2021, 26.6 billion dollars was wasted in the public cloud in one year with always-on cloud resources.
According to sustainability experts at the Antithesis Institute, some people forget to turn off servers while others over-estimate the services, projected ends, change in business processes and isolation needs. Under-used servers might only run batch jobs on the weekend or certain systems during business hours but they’re on and being cooled 24/7. Automatic scaling could solve that but few people bother. Why take the risk when there’s no reward? They would if telcos had a green programme.
But the zombies pass under the radar, according to Cummins. Efficiency algorithms are not relevant for zombies, so you must seek and destroy them. Cummins gave some pointers: Try a scream test. Another technique is tagging, providing meta-data for future review, which is a manual process and only sometimes clear. Operational tools like GreenOps and AIOps and LightSwitchOps can turn off a server and sets it up like a light switch. This must be fast and reliable though.
Scripts can turn servers on and off. GitOps, where infrastructure is recreated as code, can spin servers up and down. GitOps could be an alternative to systems that require redundancy.
The problem could get worse, warned Cummins. The cloud, virtualisation and serverless computing make zombies invisible. Prevention through heavy governance does not help.
It’s not just servers that are zombies, there’s ‘demonic’ data and even spectral network traffic, unhappy packets that roam the infrastructure and never reach their destination. Together they comprise the banshee-like internet background noise.
Tools like Quarkus can start up faster than an LED light bulb but they only solve half the problem, the other half being the processes and infrastructure that make organizations reluctant to stop and start servers. So zombie servers may proliferate in the age of cloud computing.
In March cloud data manager Veritas Technologies, unveiled new research that showed that half (51%) of consumers said they were furious that online data storage wastes energy and produces environmental pollution when, on average, half of the data enterprises store is redundant, obsolete or trivial (ROT) and another 35% is dark with unknown value. More ominously half of consumers (47%) said they would stop buying from a company if they knew it was wilfully causing environmental damage by failing to control how much unnecessary or unwanted data it is storing.