Setonix supercomputer named fourth 'greenest'

Australia’s Setonix supercomputer has been recognised as one of the greenest in the world on the globally recognised Green500 list.

The news was announced by Pawsey Supercomputing Research Centre, home of Setonix, after the list was unveiled at the SC22 international supercomputing conference in Texas. The ranking puts Setonix in company with exascale supercomputers Frontier in the US and LUMI in Finland, which share the same computing architecture.

Setonix was also named the most powerful public research supercomputer in the Southern Hemisphere, ranking 15th in the global Top500 list this week.

The Green500 is the global benchmark for efficient high-performance computing. The ranking means once fully available to researchers in early 2023, Setonix will enable high-impact research in domains such as radio astronomy, energy and resources, engineering, bioinformatics, health sciences and climate science while lowering its environmental impact.

Since the completion of the first stage of the system earlier this year, researchers have already used Setonix to produce a highly detailed image of a supernova remnant captured using CSIRO’s ASKAP radio telescope on Wajarri Country in Western Australia.

At peak performance, Setonix will provide massively parallel compute power, equivalent to hundreds of thousands of standard computers working in unison for research, artificial intelligence and machine learning. Importantly, it will deliver that peak performance at 56.98 GFlops/watt, providing a 30-fold increase in raw compute power over its predecessor systems Magnus and Galaxy while only increasing the power draw only by 50%.

Named after Western Australia’s favourite marsupial, the quokka (Setonix brachyurus), Setonix will deliver enough compute power to do in a second a calculation that would take a human 1.5 billion years to achieve. Part of an Australian Government-funded $70 million technology refresh program, Setonix is an eight-cabinet HPE Cray EX system that includes 217,088 AMD compute cores, 768 graphics cores and 1792 total compute nodes.

The system was designed to deliver high-performance computing with energy efficiency in mind. The supercomputer is a hybrid system of central processing units (CPU) and graphics processing units (GPU). This heterogeneous architecture enables Setonix to reach significantly higher computer power and prepare for next-generation high-performance computing without significantly increasing power draw.

The Top500 and Green500 rankings were made based on the peak and maximum power of Setonix’s GPU partitions.

The energy efficiency of Setonix is made possible by the adoption of the latest AMD CPU and GPU technology together with innovative direct liquid cooling of all system components. In addition to that, the system is cooled by a geothermal solution, pioneered by Australia’s national science agency CSIRO, saving approximately 7 million litres of water every year that might otherwise be wasted, and powered by a 120 kW solar panel system, which helps offset the centre’s carbon footprint.

The new supercomputer is being installed in two stages. The first stage debuted at 314 on the Top500 in November 2021, delivering a peak performance of 2.57 petaFLOPS. The arrival of the final phase of the system and new ranking in the Top500 and Green500 list is a significant milestone in Pawsey’s multi-year technology refresh, aimed at helping elevate Australian research to the global level.

Other important elements of the technology refresh program already deployed are:

• A 130 PB S3 capable multi-tiered storage system including Ceph object storage and Versity-controlled tape storage.
• A move from a monolithic single-core router to a spine-leaf architecture with a 400 Gigabits per second (Gbps) backbone and 100 Gbps links to nodes.
• Dedicated radio astronomy systems including GPU cluster Garrawarla and ingest nodes.
   

“Being named one of the world’s most powerful supercomputers, and one of the greenest, is a significant acknowledgement of Australia’s investment in high-impact research enabled by high-performance computing,” said Pawsey Supercomputing Research Centre’s Executive Director Mark Stickells.

“Setonix will give Australian researchers the computing power and infrastructure, supported by our expert staff, to discover new galaxies, develop new medicines, create new battery technologies and better understand our universe. It will unlock the scientific knowledge needed to support Australia’s standing in global academia and commercial success.

“By delivering the full power of Setonix in an energy-efficient system we can ensure researchers are able to do their work with sustainability in mind. This is a significant milestone for Pawsey and the broader Australian research community, and is the culmination of more than three years of effort from the team to deliver world-class compute facilities sustainably,” Stickells said.

The Pawsey Capital Refresh project is supported by the Australian Government through a $70 million grant. Pawsey is also supported by the Department of Education under the National Collaborative Research Infrastructure Strategy (NCRIS).