Introduction

MyPBS is an accounting package designed to address the void in per-user/per-project accounting mechanisms for high performance computing (HPC) centers. The software package can address such issues as projects, queue price weights, per-user project permissions, and much more.

The software package was designed to be as portable and diverse as possible, and therefore uses many common applications that are available on a variety of computing platforms:

MyPBS utilizes perl wrappers that are direct replacements for many of the common PBS commands, such as qsub, and are installed into the common path only after the original binaries are moved to another location. The original binaries are required in order for MyPBS to perform the actual job submission to the PBS system. See the document on MyPBS Installation for more information.

System Configurations

MyPBS can be configured to operate in a variety of environments. MyPBS can be split into two different functional components: a statistical monitoring and web interface package, and the PBS plug-ins. Installations have been tested where both functional components were installed on the same machine, and where they were split. Both work.
The requirements for splitting functionality are:

1. The MySQL server must be accessible from the computation and submission nodes, as well as the web server via IP.
2. The MyPBS perl wrappers and moved PBS binaries must be accessible from the computation nodes. This is due to an artifact from PBS that MPI node 0 is commonly a computation node, and will be responsible for launching the job.
3. The prologue and epilogue plug-ins must be available from the computation nodes
4. The perl module, MyPBS.pm must be installed, or accessible from the nodes.

One common solution to items 2-4 are to setup NFS shares to the computation nodes from the head node.

Accounting Mechanisms

HPC centers have a diverse ability to charge or account for used computational cycles on their systems with MyPBS. The unit of measurement in MyPBS is the service unit, where one SU is one CPU/hour.

Queue Weights

Jobs that run in high-priority queues can be assigned a higher overall cost than jobs that run in the standard queues. If your HPC center uses these types of queues, you can impose a greater weight to those jobs that run in the higher priority queues. By default, all queue weighting is set to 1.0. A queue weight of 2.0 would mean that for a job that consumed 10CPU/hours, the project would be deducted 20 SUs for the computational work performed.

Cost Management

Different styles of projects can, and usually do have different costs associated with their use. For example, university students will have a lower $SU cost than a commercial project on a university system. MyPBS allows for unlimited cost structures, each with their own "Normal" and "Over-run" costs.

Over-run aka Overdraft

MyPBS is over-run compatible. Projects can be permitted to continue to work even if any pre-assigned or pre-paid SU balance is consumed. Projects can also be configured to never run a job that could cause them to have a negative balance of SUs. This allows MyPBS to be used as a debit, debit with no overdraft prevention (what we call overrun), or credit based system, all configurable on a per-project basis. In addition, overrun SUs can be charged at a higher rate than prepaid SUs. This allows HPC centers to predict and prepare for large projects; users that want to continue to run can be charged a premium for that ability.

SU Addition

SUs can be added to a project by either a dollar amount (by cross-referencing the cost structure for a particular project) or by entering an arbitrary SU amount. If the project has a negative balance due to over-run computation, funds or SUs are first applied to the negative balance and then if any funds remain, will provide a credit balance to the project.

Project Administration

Multi-User

Projects can have more than one user performing work towards the project, and have their hours be counted under the same project. This is very useful for large HPC centers that commonly want to create a "catch-all" project for undergraduate computational work, where students will commonly be doing work for multiple classes and assignments.

Queue Control

Projects can be limited to specific PBS queues. Any jobs that are submitted to other queues are immediately rejected with an error message stating that the user does not have access to run the job in that queue for the submitted project.

Project Manager

The project manager can be a non-local user to the computational machine, and has the ability to add or delete users, see detailed statistics on the project progression, or request additional time on the system.

Over-run

Administrators have the option of enabling a specific project to be permitted to continue computational work even when the balance of their prepaid SU's are exhausted. Additionally, SUs that are not prepaid have a different cost than pre-paid SUs.