A cyberinfrastructure-aware framework that derives a job's resource request from the application and the target system, rather than asking the user to guess it, and revises it as the job runs at the HPC center, while at the edge it selects the model variant that fits the field device before deployment.
Before a job runs, users hand-estimate wall-time, memory, and accelerators. Across leadership-class systems the result is the same shape of waste, and it is documented, not anecdotal.
The waste is not random misconfiguration. It follows from asking people to predict machine behavior on evolving infrastructure that now spans the edge-to-center continuum.
Each failure pattern has one primary component that answers it, with a few drawing support from others. The Reference Architecture composes them into a single closed loop.
| Failure pattern | Answered by | With support from |
|---|---|---|
| Wall-time over-estimation | HARP | iScheduler |
| GPU underutilization in AI jobs | DNN Estimator | HARP |
| Reactive resubmission | HARP | iScheduler |
| Static scheduling, no runtime adaptation | iScheduler | Rules Engine |
| Heterogeneous workflow stages | HARP | CI Database |
| Constant CI / middleware updates | CI Database | Rules Engine |
| Blind edge model deployment | ML Field Planner | CKN telemetry |
Programmable parts that call one another through defined endpoints. Open any one to follow its challenge, solution, and results.
Per-application wall-time, memory, CPU, and GPU from low-cost profiling and an asymmetric loss.
Open → ValidationTurns an estimate into a policy-valid, executable Tapis or SLURM plan, and thresholds into governed events.
Open → EstimationEstimates task-specific CNN and YOLO workloads from the compiled HLO graph plus tabular features.
Open → OrchestrationThe closed loop: reasons over estimates, dispatches through Tapis, monitors, and reschedules.
Open → EdgeSelects a model variant under accuracy, latency, power, and memory limits before field deployment.
Open → IntegrationComposes all six with the ICICLE cyberinfrastructure; animal ecology as the worked example.
Open →High-performance computing underpins modern scientific discovery, yet researchers struggle to use it efficiently. Before submitting a job, users must estimate wall-time, memory, node counts, accelerator needs, and scheduling configurations, usually by hand. The result is chronic overprovisioning, idle accelerators, long queue waits, premature job termination, and repeated resubmission. The problem deepens with AI workloads and extends to the edge, where energy budgets and on-device feasibility decide whether a model can run on field hardware.
This dissertation answers with an AI-driven, cyberinfrastructure-aware framework for resource estimation and adaptive workflow orchestration. It builds estimation, validation, scheduling, monitoring, and edge planning as programmable components that call one another through well-defined endpoints, which also positions the work as a foundation for future agentic systems. The framework contributes six named components, composed into a reference architecture with an animal-ecology workflow as the worked example: edge camera-trap detection that degrades in the field triggers a center-side retraining run that the framework estimates, validates, schedules, and redeploys.