Parallel IO for history and restart writing of high-res cases

[janahaddad]

Description

@yunfangsun uncovered a behavior for large high-res meshes that caused hanging in the SCHISM+WW3 simulation. The workaround was to:

• add pio_typename = PNETCDF to the ufs.configure file

CMEPS defaults to serial IO for writing history and restart files...
Also see: ufs-community#2347

Solution

@yunfangsun has run some initial timing tests with different PIO options. Based on performance in more timing tests, could either:

• disable MED history & restart writing
• write them at end of simulation to optimize i/o

Related to

• see detailed dscussion in history and restart write are hanging (or running too slow) in the ATM+SCH+WW3 case CMEPS#1
• somewhat related: the restart capabilities need to be tested explicitly at some point (Test restart capabilities for each model component  #113)

@uturuncoglu adding this as a backlog item for now. @yunfangsun will run the remaining tests you suggested in the issue above in the coming weeks, and we can log the results here to look back on when there's time to work on this.