Storage
HPC Storage Options
There are various options for storing job data on the HPC, depending on the nature of the data. There are three main factors to consider when thinking about data storage:
- Is the data actually needed for the running a HPC task?
- How important is the data, can it be reproduced or downloaded again?
- How big is the data?
An overview of the available storage is shown in the table below:
| Name | Path | Use for HPC Task | Backed Up | Default Quota |
|---|---|---|---|---|
| Home | /gpfs/home/username |  |
|
60GB |
| Scratch | /gpfs/scratch/username | |
 |
100GB |
| Archive | /oldgpfs/esarchive/department/username | |
|
- |
| UEA CFS | Example: /home/ressci4/USERNAME | |
- | - |
Home
You have a (local) home directory on the cluster "/gpfs/home/username" which is backed up to tape on a nightly basis and is located on the Storage Area Network (SAN).
The likely uses for this storage area are: Analysed output, model code, and data which can't be easily replicated.
Scratch
You also have a scratch area "/gpfs/scratch/username" (e.g. /gpfs/scratch/abc14xyz) which is located on a high availability storage array (protecting against individual hard drive failures), but IS NOT BACKED UP.
The likely uses for this area are: data for driving models, model output prior to analysis. Essentially an area where you should store job output and any data which can be recreated or downloaded again.
Archive
There is also an archive area "/oldgpfs/esarchive". Your archive area should be used for storing large data sets and alike, which don't need to be regularly accessed. Archive is secure and backed up to tape before finally being archived, data then resides on two archive tapes. There maybe some delay in retrieving data from archive. We suggest using this area for storing data you need to access on the cluster but not on a daily basis.
Quota
There is a default quota set for each user on both HPC home and scratch filesystems. You can view your usage and corresponding quota on each filesystem, updated hourly, by running the quotacheck command:
[cc@login00 ~]$ quotacheck
HPC Storage usage at 17:00 on Feb 17
=======================================
Filesystem Usage (GB) Quota (GB)
scratchsys 95.45 100.00
syshome 25.82 30.00
If your HPC storage requirements are likely to exceed the default quotas, please contact hpc.admin@uea.ac.uk and we will be happy to discuss your needs and look at the options available.
There is a guide which details uploading and downloading files to and from the cluster can be found under SCP Configuration
General Information about HPC Storage
Please consider the following when using HPC storage:
- Filesystem backups are designed for disaster recovery only, please take care with file and data management
- HPC storage is only for HPC data. Please do not use HPC storage as a backup for your desktop/laptop, documents or music collections!