Journey from Research to User-Friendly Portals: HARP and the SGX3 Experience at Science Gateways 2023

As a Ph.D. student, most of my time is dedicated to literature reviews, data curation, and developing custom AI frameworks for HPC resource allocation. But there is a critical gap between building research code and delivering user-friendly, reproducible software components — and Science Gateways 2023 in Pittsburgh brought that gap into sharp focus.

My research centers on the HARP (HPC Application Resource Predictor) framework, which uses machine learning to estimate resource requirements for scientific workflows on HPC systems like the Ohio Supercomputer Center (OSC) and the Texas Advanced Computing Center (TACC). Researchers on these systems face real challenges: unique workflow demands, complex resource configurations, and associated allocation costs all make efficient job submission a moving target.

During my SGX3 internship at TACC, I focused on the software engineering side of HARP — applying FAIR principles, addressing generalizability, and containerizing components for cyberinfrastructure-agnostic deployment. This work is part of the NSF AI Institute ICICLE, aimed at simplifying and expanding AI usage for analytics, modeling, and automation in scientific workflows (Ai4CI).

The HARP portal showing feature correlations and their influence on walltime prediction.

Sample cost-time output from regression models, parsed against CI policies and billing structures.

ICICLE @ Science Gateways 2023

As an SGX3 fellow, I submitted both a Portal and a Poster and was sponsored to attend the conference. Science gateways are user-friendly, end-to-end solutions that enable researchers to leverage complex computing, data resources, and domain-specific tools — essentially research portals for real-world science.

One of the highlights was participating in a tutorial led by Christian Garcia: “Exploring the Tapis Pods Service and Analyzing Its Initial Performance.” This session illuminated the integration of ICICLE and TAPIS for user-driven workflows like Smart Food Sheds and Animal Ecology Camera Traps. The Tapis Pods service is an open-source API that streamlines Kubernetes container management, networking, and security at TACC.

Our poster, “HARP — Know Your Resource Needs!”, incidentally secured the runner-up spot for Best Poster. The SGX3 interns also introduced their student portals to an enthusiastic audience, each tailored to specific user needs.

Conversations, Mentors, and the 13-Minute Hackathon

One of the most valuable aspects of Gateways23 was the mentor-mentee program — each student was paired with a mentor to share meals, attend sessions, and discuss career paths. Meeting my mentor offered new perspectives on application profiling in academic settings with limited HPC access.

We also wrapped up the conference with a delightful 13-minute hackathon, where participants brainstormed gateway concepts for addressing digitization challenges. A fitting, high-energy close to an inspiring week.

The NSF luncheon with panelists from the NSF Office of Advanced Cyberinfrastructure (OAC) gave us insights into fellowships, career prospects, and funding paths for graduate researchers.

And a personal favorite: a presentation titled “The Truth About Science Communication” by Joe Palca, drawing on four decades of communicating science to the public. When does science become newsworthy? How do you make obscure topics captivating? These are questions every researcher should wrestle with.

More videos about Gateways23 and the Science Gateways Community can be found on the Science Gateways YouTube channel.