HPC Storage Options

There are various options for storing job data on the HPC, depending on the nature of the data.  There are three main factors to consider when thinking about data storage:

  • Is the data actually needed for the running a HPC task?
  • How important is the data, can it be reproduced or downloaded again?
  • How big is the data?

 

An overview of the available storage is shown in the table below:

 
Name Path Use for HPC Task Backed Up Default Quota
 Home  /gpfs/home/username 250GB
 Scratch  /gpfs/scratch/username 500GB

 

Home

You have a (local) home directory on the cluster "/gpfs/home/username" which is backed up to tape on a nightly basis and is located on the Storage Area Network (SAN).

The likely uses for this storage area are: Analysed output, model code, and data which can't be easily replicated.

Scratch

You also have a scratch area "/gpfs/scratch/username" (e.g. /gpfs/scratch/abc14xyz) which is located on a high availability storage array (protecting against individual hard drive failures), but IS NOT BACKED UP.

The likely uses for this area are: data for driving models, model output prior to analysis. Essentially an area where you should store job output and any data which can be recreated or downloaded again.

Quota

There is a default quota set for each user on both HPC home and scratch filesystems.  You can view your usage and corresponding quota on each filesystem, updated hourly, by running the quotacheck command:

[cc@login00 ~]$ quotacheck
 HPC Storage usage at 17:00 on Feb 17
=======================================
 Filesystem    Usage (GB)    Quota (GB)
 scratch         95.45        100.00
 home            25.82         30.00

If your HPC storage requirements are likely to exceed the default quotas, please contact hpc.admin@uea.ac.uk and we will be happy to discuss your needs and look at the options available. 

There is a guide which details uploading and downloading files to and from the cluster can be found under SCP Configuration

Archive

There is also an archive area "/oldgpfs/esarchive" for older accounts.  This is a read only location.

General Information about HPC Storage

Please consider the following when using HPC storage:

  • Filesystem backups are designed for disaster recovery only, please take care with file and data management
  • HPC storage is only for HPC data. Please do not use HPC storage as a backup for your desktop/laptop, documents or music collections!