Peregrine HPC Energy Research | NREL Data Catalog

Complete power and performance metric data for 10,000 jobs randomly selected from two years of the Peregrine system’s operation (all jobs prior to October, 2015). Complete power and performance metric data for jobs run on the Peregrine system in April, 2015.

Related Publications:
(1) Bruce Bugbee, Caleb Phillips, Kenny Gruchalla, Ryan Elmore, Avi Purkayastha. Exploring HPC Application Power Usage. Conference on Data Analysis (CoDA) 2016. March 2-4, 2016. Santa Fe, New Mexico.
(2) Ryan Elmore, Kenny Gruchalla, Caleb Phillips, Avi Purkayastha, Nick Wunder. An Analysis of Application Power and Schedule Composition in a High Performance Computing Environment. National Renewable Energy Laboratory. Technical Report. NREL/TP-2C00-65392. January, 2016. http://www.nrel.gov/docs/fy16osti/65392.pdf

2 Data Resources

Name	Size	Type	Resource Description	History
10k.anon.json.bz2	631.72 MB	Document	Complete power and performance metric data from 10,000 jobs randomly selected from two years of the Peregrine system’s operation (all jobs prior to October, 2015). Format: The data is available as a compressed JSON stream — one line per JSON object. Each object is a hash containing three elements, each of which is itself a hash: “power” “ganglia” “job” The “job” object contains metadata for the job including these fields: id unique identifier for this job user_name user running the job — anonymized group_name group running the job — anonymized account allocation the job is charged to — anonymized queue the queue the job is submitted to: https://hpc.nrel.gov/users/systems/peregrine/running-jobs/new-queue-structure submit_time start_time end_time exit_code exit code from Moab nodelist colon separated list of node hostnames used for the job nodes_used processors_used wallclock_used torque_wallclock_used torque_exit_code exit code according to torque cpu_used mem_used vmem_used wallclock_req feature_req nodes_req processors_req feature requests interactive whether or not the job was run interactively node_hours app_name application category app_name_predicted whether or not application category was predicted using a Naïve Bayes classifier The "power" and "ganglia" objects contain time-series data from iLo http://www8.hp.com/us/en/products/servers/ilo/ and NWPerf https://github.com/EMSL-MSC/NWPerf/. The format of the timeseries data is a list of hashes containing, timestamp (epoch time in microseconds), metric name, host and value, e.g.,: { "timestamp": 1461697084441, "host": "n1668", "value": 228, "metric": "power" } Anonymized fields have been SHA1 hashed.
april2015.anon.json.bz2	3.07 GB	Document	Complete power and performance metric data for jobs run on the Peregrine system in April, 2015. Format: The data is available as a compressed JSON stream — one line per JSON object. Each object is a hash containing three elements, each of which is itself a hash: “power” “ganglia” “job” The “job” object contains metadata for the job including these fields: id unique identifier for this job user_name user running the job — anonymized group_name group running the job — anonymized account allocation the job is charged to — anonymized queue the queue the job is submitted to: https://hpc.nrel.gov/users/systems/peregrine/running-jobs/new-queue-structure submit_time start_time end_time exit_code exit code from Moab nodelist colon separated list of node hostnames used for the job nodes_used processors_used wallclock_used torque_wallclock_used torque_exit_code exit code according to torque cpu_used mem_used vmem_used wallclock_req feature_req nodes_req processors_req feature requests interactive whether or not the job was run interactively node_hours app_name application category app_name_predicted whether or not application category was predicted using a Naïve Bayes classifier. The "power" and "ganglia" objects contain time-series data from iLo http://www8.hp.com/us/en/products/servers/ilo/ and NWPerf https://github.com/EMSL-MSC/NWPerf/. The format of the timeseries data is a list of hashes containing, timestamp (epoch time in microseconds), metric name, host and value, e.g.,: { "timestamp": 1461697084441, "host": "n1668", "value": 228, "metric": "power" } Anonymized fields have been SHA1 hashed.

Version	Name	Size	Type	Resource Description	Notes	Date
1	10k.anon.json.bz2	631.72 MB	Document	Complete power and performance metric data from 10,000 jobs randomly selected from two years of the Peregrine system’s operation (all jobs prior to October, 2015). Format: The data is available as a compressed JSON stream — one line per JSON object. Each object is a hash containing three elements, each of which is itself a hash: “power” “ganglia” “job” The “job” object contains metadata for the job including these fields: id unique identifier for this job user_name user running the job — anonymized group_name group running the job — anonymized account allocation the job is charged to — anonymized queue the queue the job is submitted to: https://hpc.nrel.gov/users/systems/peregrine/running-jobs/new-queue-structure submit_time start_time end_time exit_code exit code from Moab nodelist colon separated list of node hostnames used for the job nodes_used processors_used wallclock_used torque_wallclock_used torque_exit_code exit code according to torque cpu_used mem_used vmem_used wallclock_req feature_req nodes_req processors_req feature requests interactive whether or not the job was run interactively node_hours app_name application category app_name_predicted whether or not application category was predicted using a Naïve Bayes classifier The "power" and "ganglia" objects contain time-series data from iLo http://www8.hp.com/us/en/products/servers/ilo/ and NWPerf https://github.com/EMSL-MSC/NWPerf/. The format of the timeseries data is a list of hashes containing, timestamp (epoch time in microseconds), metric name, host and value, e.g.,: { "timestamp": 1461697084441, "host": "n1668", "value": 228, "metric": "power" } Anonymized fields have been SHA1 hashed.	CSCData file id: caae8c64-00ef-4c3f-9c40-c64426c927ff	04-09-2024 14:51:27

Version	Name	Size	Type	Resource Description	Notes	Date
2	april2015.anon.json.bz2	3.07 GB	Document	Complete power and performance metric data for jobs run on the Peregrine system in April, 2015. Format: The data is available as a compressed JSON stream — one line per JSON object. Each object is a hash containing three elements, each of which is itself a hash: “power” “ganglia” “job” The “job” object contains metadata for the job including these fields: id unique identifier for this job user_name user running the job — anonymized group_name group running the job — anonymized account allocation the job is charged to — anonymized queue the queue the job is submitted to: https://hpc.nrel.gov/users/systems/peregrine/running-jobs/new-queue-structure submit_time start_time end_time exit_code exit code from Moab nodelist colon separated list of node hostnames used for the job nodes_used processors_used wallclock_used torque_wallclock_used torque_exit_code exit code according to torque cpu_used mem_used vmem_used wallclock_req feature_req nodes_req processors_req feature requests interactive whether or not the job was run interactively node_hours app_name application category app_name_predicted whether or not application category was predicted using a Naïve Bayes classifier. The "power" and "ganglia" objects contain time-series data from iLo http://www8.hp.com/us/en/products/servers/ilo/ and NWPerf https://github.com/EMSL-MSC/NWPerf/. The format of the timeseries data is a list of hashes containing, timestamp (epoch time in microseconds), metric name, host and value, e.g.,: { "timestamp": 1461697084441, "host": "n1668", "value": 228, "metric": "power" } Anonymized fields have been SHA1 hashed.	CSCData file id: 3c1ab018-661d-4382-abab-c3c9428162c4	04-09-2024 14:55:14
1	april2015.anon.json.bz2	3.07 GB	Document	Complete power and performance metric data for jobs run on the Peregrine system in April, 2015. Format: The data is available as a compressed JSON stream — one line per JSON object. Each object is a hash containing three elements, each of which is itself a hash: “power” “ganglia” “job” The “job” object contains metadata for the job including these fields: id unique identifier for this job user_name user running the job — anonymized group_name group running the job — anonymized account allocation the job is charged to — anonymized queue the queue the job is submitted to: https://hpc.nrel.gov/users/systems/peregrine/running-jobs/new-queue-structure submit_time start_time end_time exit_code exit code from Moab nodelist colon separated list of node hostnames used for the job nodes_used processors_used wallclock_used torque_wallclock_used torque_exit_code exit code according to torque cpu_used mem_used vmem_used wallclock_req feature_req nodes_req processors_req feature requests interactive whether or not the job was run interactively node_hours app_name application category app_name_predicted whether or not application category was predicted using a Naïve Bayes classifier. The "power" and "ganglia" objects contain time-series data from iLo http://www8.hp.com/us/en/products/servers/ilo/ and NWPerf https://github.com/EMSL-MSC/NWPerf/. The format of the timeseries data is a list of hashes containing, timestamp (epoch time in microseconds), metric name, host and value, e.g.,: { "timestamp": 1461697084441, "host": "n1668", "value": 228, "metric": "power" } Anonymized fields have been SHA1 hashed.		04-09-2024 14:51:27

Keywords

Submitted

• Apr • 09 2024

Computational Science

Caleb Phillips

303-275-4297

Center 2C00

ORCID iD 0000-0002-3665-4239

Author Information

Caleb Phillips, Computational Science, ORCID iD: 0000-0002-3665-4239

Cite This Dataset

Phillips, Caleb. 2024. "Peregrine HPC Energy Research." NREL Data Catalog. Golden, CO: National Renewable Energy Laboratory. Last updated: April 10, 2024.

About This Dataset

id 235

Pub Number NREL/TP-2C00-65392

Status Public

Last Updated 04/10/2024

Facilities

Energy Systems Integration Facility (ESIF)

High Performance Computing Center (HPC)

Funding Organization

NREL Internal (LDRD, BD)

Sponsoring Organization

USDOE Office of Energy Efficiency and Renewable Energy (EERE)

Research Areas

Computational Science

Energy Analysis

Energy Systems Integration

License

View License