MatEnsemble Documentation

MatEnsemble orchestrates large batches of Flux-scheduled tasks on HPC systems: Python callables, shell commands, explicit resource requests, and dependency-aware execution graphs from a single Python driver process.

Guides

• Overview
  • High-throughput orchestration and schedulers
  • What MatEnsemble does
  • Core concepts
  • Logging and on-disk layout
  • Adaptive vs. non-adaptive scheduling
  • User-defined strategies
  • Roadmap and stability
  • References
  • Next steps
• Installation
  • Versions and compatibility
  • Container images (recommended on clusters)
  • Where to read next
• Tutorials
  • Example repositories
  • Minimal executable (“exec”) workflow
  • Python chores and OutputReference dependencies
  • Chained dependencies (any acyclic DAG)
  • Nested arguments
  • Third-party imports inside chores
  • Operational tips
  • Further reading
• Design and execution model
  • Runtime prerequisites
  • Objects you interact with
  • Workflow directory layout
  • DAG construction and ordering
  • Resource accounting
  • Main scheduling loop (“super loop”)
  • Failure propagation
  • Dashboard (optional)
• Configuration and behavior reference
  • Pipeline constructor
  • Pipeline.chore decorator factory
  • Pipeline.exec
  • Pipeline.submit
  • status.json schema
  • Per-chore artifacts
  • Failure reason strings (internal)
  • Redis helper (optional)

API

• matensemble
  • matensemble package
    • Submodules