Overview

HyperQueue is a tool designed to simplify execution of large workflows (task graphs) on HPC clusters. It allows you to execute a large number of tasks in a simple way, without having to manually submit jobs into batch schedulers like Slurm or PBS. You just specify what you want to compute – HyperQueue will automatically ask for computational resources and dynamically load-balance tasks across all allocated nodes and cores. HyperQueue can also work without Slurm/PBS as a general task executor.

If you use HyperQueue in your research, please consider citing it.

Useful links#

• Installation
• Quick start
• Python API
• Repository
• Discussion forum
• Zulip (chat platform)

Features#

Resource management

• Batch jobs are submitted and managed automatically
• Computation is distributed amongst all allocated nodes and cores
• Tasks can specify resource requirements (# of cores, GPUs, memory, ...)

Performance

• Scales to millions of tasks and hundreds of nodes
• Overhead per task is around 0.1 ms
• Task output can be streamed to a single file to avoid overloading distributed filesystems

Simple deployment

• HQ is provided as a single, statically linked binary without any dependencies
• No admin access to a cluster is needed