HPC Storage Options
There are various options for storing job data on the HPC, depending on the nature of the data. There are three main factors to consider when thinking about data storage:
- Is the data actually needed for the running a HPC task?
- How important is the data, can it be reproduced or downloaded again?
- How big is the data?
An overview of the available storage is shown in the table below:
|Name||Path||Use for HPC Task||Backed Up||Default Quota|
You have a (local) home directory on the cluster "/gpfs/home/username" which is backed up to tape on a nightly basis and is located on the Storage Area Network (SAN).
The likely uses for this storage area are: Analysed output, model code, and data which can't be easily replicated.
You also have a scratch area "/gpfs/scratch/username" (e.g. /gpfs/scratch/abc14xyz) which is located on a high availability storage array (protecting against individual hard drive failures), but IS NOT BACKED UP.
The likely uses for this area are: data for driving models, model output prior to analysis. Essentially an area where you should store job output and any data which can be recreated or downloaded again.
There is a default quota set for each user on both HPC home and scratch filesystems. You can view your usage and corresponding quota on each filesystem, updated hourly, by running the quotacheck command:
[cc@login00 ~]$ quotacheck
HPC Storage usage at 17:00 on Feb 17
Filesystem Usage (GB) Quota (GB)
scratch 95.45 100.00
home 25.82 30.00
If your HPC storage requirements are likely to exceed the default quotas, please contact firstname.lastname@example.org and we will be happy to discuss your needs and look at the options available.
There is a guide which details uploading and downloading files to and from the cluster can be found under SCP Configuration
There is also an archive area "/oldgpfs/esarchive" for older accounts. This is a read only location.
General Information about HPC Storage
Please consider the following when using HPC storage:
- Filesystem backups are designed for disaster recovery only, please take care with file and data management
- HPC storage is only for HPC data. Please do not use HPC storage as a backup for your desktop/laptop, documents or music collections!