Regional Weather Forecasting in the 1996 Summer Olympic Games
Using an IBM SP
Zaphiris Christidis
IBM T. J. Watson Research Center, Yorktown Heights, NY
zaphiri@watson.ibm.com
James Edwards
NOAA/Forecast Systems Laboratory, Boulder, CO
and
Cooperative Institute for Research in the Atmosphere
Colorado State University
Fort Collins, CO
jedwards@fsl.noaa.gov
John Snook
NOAA/Forecast Systems Laboratory, Boulder, CO
snook@fsl.noaa.gov
Introduction
A 30-node IBM RS/6000 Scalable Power-parallel (SP) system was installed
at the regional National Weather Service Office (NWS) in Peachtree City,
Georgia, and it was used during the 1996 Summer Olympic Games in order
to provide efficient and accurate regional weather forecasts for the Olympic
Games. These forecasts were produced by the Regional Atmospheric Modeling
System (RAMS), which fully exploited the parallel processing power of the
30-node parallel system. The SP system was configured for maximum redundancy,
and it was customized in order to provide additional computational and
weather visualization support at the regional NWS office. Here we will
discuss the SP hardware and software configuration, as well as specific
system customizations enabling the SP to perform operational regional weather
forecasts with the minimum amount of human intervention.
The Olympic SP Weather Server
The Olympic SP Weather system was configured with 28 66 MHz thin
nodes, each equipped with two GB of local disk and 128MB of memory, and
two 77 MHz wide nodes, with 256 MB of memory and four GB of local
disk each. The wide nodes shared an external eight GB Serial Storage Architecture
(SSA) disk pack, configured with the high availability software (HACMP)
for maximum redundancy. User file-systems residing on the SSA disks, were
physically connected to both wide nodes, while owned by a single preassigned
primary node. In case of a primary node failure, HACMP was designed
to take control automatically by re-appointing the secondary node
as primary, while physically switching ownership of file systems
to the active primary node. (See Figure 1.) The system was also
equipped with two RISC System/6000 39H POWER2 control workstations with
another external eight GB SSA disk pack and HACMP.
Figure 1. Olympic Weather Forecasting Server Configuration.
User home as well as utility file-systems were owned by the primary
control workstation, olcw1, and they were NFS-mounted across
all nodes via the AIX auto-mounter. File-systems containing production
software and RAMS executables were owned by the primary wide node
(ol1n01), and they were also NFS-mounted across all available
nodes.
To help the dissemination of weather forecasts from RAMS, another IBM
RISC System/6000 39H equipped with a GXT1000-2 graphics adapter was configured
as an additional node to the total SP configuration. This workstation shared
the same file-systems with the rest of the SP nodes, and ran the IBM Visualization
Data Explorer (DX) software package. The entire system was assigned its
own private Ethernet network, while the two control workstations and the
two wide nodes were also connected to the local public network, having
direct access to the Internet via a firewall host. The configured system
had a total of 80 GB of storage, 4.2 GB of memory, and it was capable of
delivering over eight GFLOPS of peak performance.
The Local Analysis and Prediction System (LAPS) (Stamus et al, 1997)
was used in order to provide initial meteorological conditions to the RAMS
model. It was scheduled to execute on the secondary SP control workstation,
olcw2, on a 30-minute cycle. LAPS collected observed meteorological
variables from various sources such as satellite, radar, buoy, upper air
and ground-based weather stations, as well as forecasted fields from the
Rapid Update Cycle model (RUC). This information was available in the form
of NFS-mounted file-systems, from various data ingestion systems already
present at the regional NWS office in Peachtree City (Rothfusz et al, 1996).
LAPS produced three dimensional analyses of various meteorological fields,
which in turn were assimilated into the RAMS horizontal and vertical grids,
in the form of a single time-encoded file, assembled on the SP production
file-systems.
Boundary conditions were collected and processed by the secondary wide
node ol2n01, via an automated file transfer process (FTP).
The FTP process was initiated by an alert mechanism, spawned on
a 20 minute cycle via another AIX cron process. Whenever
forecast files from the 29km ETA weather model running at NCEP became available,
the alert software would queue FTP requests on ol2n01,
forcing it to download the appropriate files from NCEP. The file transfer
was done through a dedicated T1 line (1Mb/s) from NCEP to the NWS in Peachtree
City, Georgia. The downloaded files were decompressed upon arrival on ol2n01,
and relevant geographical sectors were extracted and interpolated into
the RAMS grid. In turn, these files were time-encoded on the production
file-systems, and they were used as forecast boundary conditions for the
RAMS simulations. Surface parameters, such as topography, seasonal sea
surface temperatures, soil moisture as well as vegetation types were interpolated
from a 30-second database, and they were generated upon demand on the production
file-systems, for RAMS parameter initialization.
Operational weather simulations were carried out by RAMS, a non-hydrostatic
weather model suitable for Regional, Mesoscale and Cloud-scale atmospheric
predictions (Pielke et al, 1992). RAMS was parallelized on the IBM SP system
(Edwards et al, 1997) using a two-dimensional domain partitioning scheme,
which was facilitated by the Scalar Modeling System (SMS) - Nearest Neighbor
Tool (NNT) (Rodriguez et al, 1996). Thus, different geographical areas
(sub-regions) around Georgia were distributed among available SP nodes
and the full weather equations were solved in parallel on the IBM SP system
for each sub-region separately. Model simulations were initiated on olcw1
on a 60-minute cycle. The ol1n01 was assigned to perform
data I/O functions, involving the reading of the initial and boundary conditions,
as well as storing basic predicted atmospheric state variables, such as
wind, temperature, pressure, moisture, etc, at each model grid point and
for selected time steps.
The LAPS and RAMS outputs were converted into the format required by
N-AWIPS (DesJardins et al, 1997) for dissemination of the forecasts to
the Olympic Support Weather Forecasting team. In addition, Data Explorer
was used by the forecasters to visualize and analyze the two and three-dimensional
meteorological variables as generated by RAMS in a three-dimensional display
environment (Treinish and Rothfusz, 1997). Data Explorer was also used
to obtain animations of selected meteorological parameters for presentation
to the media and distribution on the Internet via the world-wide-web. Furthermore,
selected LAPS and RAMS products were placed in an anonymous FTP repository
on ol2n01, providing the meteorological parameters needed
by the Air Resources Laboratory of NOAA, in order to operate their air
pollution dispersion model as a part of the NWS Olympic Environmental Emergency
Response Program (Rolph, M.D. et al, 1997).
Customization and Automation
Initial and boundary condition processing, as well as RAMS model simulations
were fully automated through the use of Perl scripts executed by
the AIX cron process. Actual weather forecasts made use
of 26 out of the total 30 nodes of the SP parallel system. The rest of
the SP nodes were were mostly used by various data handling processes,
such as N-AWIPS graphics post-processing.
The RAMS model was configured to operate at variable resolutions, upon
demand, and with the minimum amount of human intervention. This was facilitated
through the use of a single control file (schedule.rams),
with contents modifiable by any forecaster. This file was in the form of
a spreadsheet, specifying the forecasting cycle and duration of the RAMS
simulation, as well as the model's horizontal grid location and size (Table
1).
Table 1. Typical format of the file schedule.rams.
Routine operational weather forecasts were usually scheduled on a three-hour-cycle
throughout the day. The first weather simulation was performed on an 8km
grid over the major Georgia area for 24 hours, starting at 0600 UTC. The
second forecast was scheduled at 0900 UTC, and it was performed on a two-km
grid centered around Savannah, Georgia. Subsequent forecasts (location,
duration, cycle and resolution) were decided and finalized by the Olympic
Support Weather Forecasting team, as they saw fit.
Figure 2 shows schematically the hardware and software customization
of the Olympic Weather SP Server, while Figure 3 displays a flow diagram
of the automated forecasting process used in a production mode at the regional
NWS office. This process was a combination of four tightly coupled tasks,
as indicated by the four vertical block structures. To achieve optimal
timing among the tasks, extensive testing and tuning was performed before
the process was fixed for production simulations.
Figure 2. Hardware and Software Customization of the Olympic Weather
Forecasting Server.
Figure 3. Flow diagram illustrating the automation process used for
production runs.
The first vertical block describes the AIX cron task
spawned 20 minutes after every hour on olcw1, driving the
execution of RAMS. A production run was performed only if it was scheduled
via a proper entry in the schedule.rams file, and only
upon availability of suitable initial and boundary conditions. In the process,
various sub-tasks were scheduled appropriately, in order to assure the
smooth execution of RAMS on the SP. The second vertical block shows the
AIX cron task driving LAPS execution on olcw2.
This process was executed on a 30-minute cycle for both, surface, and upper
air data ingestion. RUC data ingestion was executed once per hour. The
third vertical block indicates the alert AIX cron task,
polling for NCEP forecast files. This task was executed three times per
hour, transferring available NCEP ETA files on ol2n01.
These files were decompressed, and appropriate geographical sectors were
extracted and processed in order to be used as boundary conditions by RAMS.
Finally, the fourth block shows the conversion of LAPS, RAMS and ETA products
into the N-AWIPS products. System backups and file archiving were scheduled
every night, when the system was otherwise idle or performed various research
simulations, for further model tuning. Model verification, followed by
a log and data recycling task was scheduled just before the first simulation
of the day.
Summary
We described the complete integration of the IBM SP parallel system
in the infrastructure already present at the Regional NWS office. This
system was configured to perform operational weather forecasts during the
1996 Summer Olympic Games in Atlanta, using LAPS/RAMS with a minimal amount
of human resources. Though the benefits in having such systems operational
in regional weather offices are significant (Snook et al, 1997), it is
worthwhile adding that the combined software/hardware system was:
Inexpensive, with low maintenance costs as compared to high-end supercomputers
used in operational weather forecasting today
Easy to install and integrate with other hardware/software
Reliable, with sufficient amount of hardware and software redundancy
Portable, facilitating its replication in environments with similar infrastructures
Easy to operate manage and automate the various processes involved, thus
making it virtually transparent to the weather forecasters
Acknowledgements
The authors wish to acknowledge IBM Corp. and the help of David Soll,
Karen McPherson, Todd Wiseman and Steven Lebowitz, for making it possible
to allocate and install the SP system at the NWS office in Peachtree City,
Georgia. Special acknowledgment goes to Pete Stamus, Clark Safford, Chris
Johnson, Lloyd Treinish, J.T. Johnson and Lans Rothfusz for significant
contributions to the total system integration. Finally, the authors are
indebted to John McGinley, Bill Cotton and Rick Lawrence for their support
in all phases of this project.
References
DesJardins, Mary L., S. T. Jacobs, D. W. Plummer and S. S. Scholtz.
N-AWIPS: AWIPS at the National Centers for Environmental Prediction.
Proceedings of the AMS 13th IIPS, February 2-8, 1997, Long Beach,
CA. .
Edwards, J., J. Snook and Z. Christidis. Forecasting
for the 1996 Summer Olympic Games with the NNT-RAMS Parallel Model.
Proceedings of the AMS 13th IIPS, February 2-8, 1997, Long Beach,
CA. (Also available
in PDF format.)
Pielke, R. A., W. R. Cotton, R .L. Walko, C. J. Tremback, W. A. Lyons,
L. D. Grasso, M. E. Nicholls, M.-D. Moran, D. A. Wesley, T. J. Lee and
J. H. Copeland.. A Comprehensive Meteorological Modeling System - RAMS.
Meteorol. Atmos. Phys., 49, 69-91, 1992.
Rodriguez, B., L. Hart, and T. Henderson, 1996: NNT 1.0 User's Guide.
In press, Forecast Systems Laboratory Technical Memorandum.
Rolph, G.D., J. McQueen, J. Sanders and D. Soule. The Use of NWS
Olympic Resources in NOAA's Environmental Emergency Response Program.
Proceedings of the AMS 13th IIPS, February 2-8, 1997, Long Beach,
CA.
Rothfusz, L. P., J. T. Johnson, L. C. Safford, M. R. McLaughlin, and
S. K. Rinard. The Olympic Weather Support System. Proceedings
of the AMS 12th IIPS, January 28 - February 2, 1996, Atlanta, GA.
Snook, J., Z. Christidis and J. Edwards.
Forecast Results from the Local-Domain Mesoscale Model Supporting the 1996 Summer
Olympic Games. Proceedings of the AMS 13th IIPS, February 2-8,
1997, Long Beach, CA. (Also
available in PDF format.)
Stamus, P.A., and J. McGinley. The Local
Analysis and Prediction System (LAPS): Providing Weather Support to the
Olympic Games. Proceedings of the AMS 13th IIPS, February 2-8,
1997, Long Beach, CA. (Also available in
PDF format.)
Treinish, L. A., and L. P. Rothfusz. Three-dimensional
Visualization for Support of Operational Weather Forecasting at the 1996
Centennial Olympic Games. Proceedings of the AMS 13th IIPS,
February 2-8, 1997, Long Beach, CA.
zaphiri@watson.ibm.com
