Leading the way into what computer scientists call the exascale era, the Frontier supercomputer will debut at the U.S. Department of Energy’s Oak Ridge National Lab (ORNL) in Tennessee in 2021. Just eight teams of scientists have been selected to develop applications for this magnificent new machine now under construction, and one of those teams is led by Sunita Chandrasekaran, an assistant professor of computer and information sciences at the University of Delaware.

Exascale computers can do a billion billion calculations per second, allowing scientists an unprecedented ability to analyze data and make new discoveries.

Chandrasekaran’s team was selected by the Oak Ridge Leadership Computing Facility (OLCF), through its Center for Accelerated Application Readiness (CAAR), to reach predictive capabilities for high electron energy simulations with the plasma physics application PIConGPU. 

PIConGPU enables simulation of laser-plasma physics and laser-matter interactions. An enhanced version of the software for Frontier could unlock new insights that enable the discovery of new products and medical treatments, such as laser-driven radiation therapy for cancer. 

For this project, Chandrasekaran will work with Guido Juckeland, the head of computational science at Helmholtz-Zentrum Dresden-Rossendorf (HZDR), a German research laboratory, and Michael Bussmann, the head of the Center for Advanced Systems Understanding (CASUS) Department at HZDR and the CASUS Scientific Project Leader. Arghya Chatterjee from ORNL will be the project's liaison.

The team will have access to early-generation hardware platforms, software development systems and resources to prepare for Frontier’s launch.

Previously, Chandrasekaran has been using Summit, the most powerful supercomputer currently in existence, which debuted in 2018 at the OLCF. Still, Frontier opens up a whole new world of possibility because of its massive computing power. It also runs on different hardware than Summit, which opens up new challenges and opportunities. PIConGPU runs on Summit, but the software used on Summit will be different to that needed for Frontier.

Chandrasekaran and her colleagues, who have expertise in programming models and directives, will utilize a C++ language abstraction model called Alpaka to upgrade PIConGPU for Frontier.

The software and application are known, but the architecture is still under construction. That means the team will need to run experiments and test their system on other high-performance computers, although none of them will likely measure up to Frontier.

“We’re working for a system that doesn’t exist yet, so we have to imagine a lot of things,” said Chandrasekaran. “We would like to build a local mini-development system close to what Frontier will be in order to start programming that system locally and exploring and running things on it and learning about the system in order to get ready for Frontier.”

The team will also explore Alpaka, OpenMP offloading and AMD’s HIP programming paradigms for Frontier. These models allow one set of code to run in parallel on a number of multi-core processors. 

"We are excited to bring PIConGPU, a previous Gordon-Bell-Award finalist, to Frontier,” said Juckeland. “We look forward to partnering with Sunita Chandrasekaran's group to draw from her experience in validating and optimizing OpenMP and OpenACC applications. By extending Alpaka — an open-source zero-overhead intra-node parallelization framework, which is utilized by PIConGPU — to support the Frontier software ecosystem, UD and HZDR can run PIConGPU simulations on all current high-performance computing platforms without any source-code modifications.”

Said Bussman: “Bringing PIConGPU to Frontier will give us unprecedented opportunities in advancing plasma-based accelerators. We expect great progress towards the energy and quality frontier defined by applications ranging from high energy physics to photon science to radiation therapy of cancer. Being among the first to leverage the power of exascale computing for particle-in-cell simulations of plasmas will enhance the predictive capabilities of our simulations and open up new possibilities in studying the complex plasma physics involved. We expect a wealth of scientific data to come out of our first runs on Frontier. This data will be extremely helpful for mastering the challenges we are facing when putting plasma accelerators into application. Like our codes, we want to give this data to the community to foster research in plasma accelerators.”

PIConGPU will use OpenMP offloading, and Chandrasekaran is in an ideal position to determine its effectiveness and work out any bugs. Since 2017, Chandrasekaran and her team has been leading the validation and verification test suite for OpenMP offloading in collaboration with ORNL and others, part of the Department of Energy-funded Exascale Computing Project (ECP) project, SOLLVE.

“We are definitely going to be able to apply the lessons learned in the past two years on the SOLLVE project for PIConGPU,” said Chandrasekaran.

Students at all levels will also benefit. Chandrasekaran will work with a graduate student on this research, and she has recently hired a doctoral student to join her Computational Research and Programming Lab. In her undergraduate course on parallel programming, Chandrasekaran will sprinkle in lessons from the real-world scenarios she encounters in her research. 

Chandrasekaran and team are also looking for a postdoctoral research associate in computational science who will be hired by ORNL and working from ORNL. The job position is described here. Interested candidates, please contact Arghya Chatterjee at chatterjeea@ornl.go and Sunita Chandrasekaran at schandra@udel.edu.

Chandrasekaran is also bringing her expertise and leadership to a new project to utilize high-performance computing and data science technology to enable new breakthroughs in cancer research. The Frederick National Laboratory for Cancer Research, operated by Leidos Biomedical Research, Inc., for the National Cancer Institute, and UD recently signed a memorandum of understanding to explore avenues for collaboration around using computational and data science to tackle cancer, AIDS, and emerging health challenges. Eric Stahlberg, director of FNL’s Biomedical Informatics and Data Science (BIDS) Directorate, spearheaded the collaboration along with Chandrasekaran. Read more about that effort here.