Frequently Asked Questions
What
is Deep Thunder?
Deep Thunder is
a service that provides local, high-resolution weather predictions customized to business applications
for weather-sensitive operations up to a day ahead of time.
In particular, the goal is to provide weather forecasts at a level of precision and fast enough
to address specific business problems. Such forecasts can be used for competitive advantage
or to improve operational efficiency and safety. Users want access to the forecasts in
an "on-demand" fashion -- disseminated to them the way they want them, when they want them.
In reality, improving the effectiveness of a customer's weather-sensitive operations is not
really about the weather. Rather, it is one of optimization of business processes such
as resource allocation, scheduling and routing, which are constrained by specific weather
events. Long-term, the real value is when the forecasts are integrated into business
processes. Having detailed forecasts of the right caliber as a service is a critical
prerequisite to enable optimization of weather-sensitive operations.
We see enormous potential for changing the current reactive approach to
such business operations to being proactive in industries as diverse as
road maintenance and operations, aviation, agriculture, broadcast,
communications, energy, insurance, emergency management, homeland security,
sports, entertainment, tourism, construction, traffic management, etc. where weather is an important
factor in making effective business decisions.
The system behind the Deep Thunder service has several key components:
receiving and processing data, modelling, and post-processing analysis,
visualization and dissemination. Although we are doing work in each of
these areas, our focus has been on the business applications that can be
achieved using high-performance computing, visualization,
and automation while designing, evaluating and optimizing an integrated
system.
The research that has led to the simulation codes used
for weather modelling has taken place for decades. Some of the codes themselves
trace their origins to the 1970s, but have evolved and improved considerably
since then. We are not inventing new weather models as part of this project.
What we are doing is adapting, refining and applying existing models.
As part of the effort, we are developing new methods of
data visualization, analysis and dissemination, and techniques for improving
computational performance and system automation. Part of the rationale
for this focus is practicality. Given the time-critical nature of weather-sensitive
business decisions, if the weather prediction can not be completed fast
enough it has no value. Thus, such predictive simulations need to be completed
at least an order of magnitude faster than real-time (e.g., a hour or so
for a 24-hour forecast.) Rapidly generated fixed and highly interactive
flexible visualizations focused on the applications enable the weather
model data to be utilized quickly, especially for near-real-time
decision making in business operations.
Is Deep Thunder like the
National Weather Service?
Deep Thunder is different than but complementary
to the National Weather Service (NWS). First of all, it would not be possible
without the NWS. Deep Thunder leverages the US Government's significant
investment in observing the atmosphere and simulating the weather by using
the data that NWS makes available.
NWS focuses on uniform services for the whole US by providing
detailed observations (spacecraft, radar, stations, etc.) and global to continental-scale
simulations on a large IBM pSeries Cluster 1600 (e.g., 12 km for
all of North America and the surrounding oceans), for example, which are
considerable tasks.
Hence, they do not have the mission to provide customized,
detailed services for specific industries or geographies.
On the other
hand, Deep Thunder provides local, cloud-scale, high-resolution (e.g.,
100 x 100 km at 1 km) simulations (on a small IBM pSeries Cluster 1600)
focused on business applications using detailed physics and
customization in operations, products and integration.
Is Deep Thunder like Doppler radar?
Doppler radar shows the current weather and can help determine what may happen within
the next hour or so, at most. Deep Thunder has a similar emphasis on the
local weather, but provides a prediction up to a day into the future.
What kind of technology powers Deep Thunder?
The system that enables the Deep Thunder service consists of a sophisticated
infrastructure of hardware and software that is integrated and automated.
The hardware includes a satellite receiver for access to weather data from the
National Weather Service, IBM System p machines for computing and IBM System x
machines for dissemination. The software includes data processing,
weather modelling, data analysis and visualization capabilities.
Where is the Deep Thunder
research headquartered?
We are at the IBM Thomas J. Watson Research Center in
Yorktown Heights, NY. Deep Thunder is a key part of the weather modelling
project within the Deep Computing Systems Department.
Who's on the team?
Tony
Praino and Lloyd Treinish work
full-time on the weather modelling project, spending most of their time on Deep Thunder.
Lloyd
is a space scientist by training, focuses primarily on visualization, modelling, applications
and overall system architecture.
Tony is a research engineer, who has been an active in the meteorological community since 1970.
He works on data ingest, forecast verification and automation with a
particular interest in winter weather phenomena in the northeast United States.
A former member of the Deep Thunder team, Zaphiris Christidis, is on
International Assignment in Beijing, working with government weather center customers in Asia. Zaphiris is a computational meteorologist, and contributed to the modelling, computational,
architecture and system components of Deep Thunder.
When did IBM start the Deep
Thunder project?
It is an outgrowth of a collaboration that Zaphiris and
Lloyd had with the National Weather Service office in Peachtree City, GA
for support of the 1996 Summer Olympic Games in Atlanta, where it achieved
a high level of accuracy and reliability in its weather forecasts. We continued
some of the activities after that looking mostly at general forecasting,
but only as a small part of our other activities that we were pursuing
in our separate departments.
As the technology improved and become more practical,
we started to consider other potential applications. When the initiative
in Deep Computing
began at IBM Research, it made sense to start a more formal effort. Therefore,
a weather modelling project was started within the Mathematical Sciences Department
in 2000. Zaphiris and Lloyd moved to that department.
Later in the year, Tony Praino joined us. Given that one focus of the project is in
high-performance computing, it moved to Deep Computing Systems Department in late 2004.
Why did IBM choose to do this
research project?
This research is part of a larger IBM Research effort
in Deep Computing
-- the concept of analyzing large amounts of data, and using that analysis
to solve complex problems. Since a near-term goal of Deep Thunder
is to enable proactive weather affected decision making by coupling predictive
weather simulations to business processes, analyses and models, it clearly
is a Deep Computing project. In addition, other components of Deep
Thunder are aligned with areas of IBM strengths in high-performance
computing, system integration and visualization.
What can Deep Thunder
do that was not possible before?
Prior to this effort, it really was not feasible to do
near-real-time high-resolution, meso-gamma-scale (or cloud-scale) weather simulations coupled with
specialized visualization focused on business problems. Granted some of the practicality
is a result of recent convergence of several factors.
First, is the relatively low cost of sufficiently powerful
computing. A small supercomputer such as a modestly-sized IBM System p Cluster 1600
can be used and enables cost-effective implementation of a range of forecasting services.
The second factor is the availability of relevant input
data. The National Weather Service makes its modeled and observed data
available to everyone via broadcast satellite technology (called NOAAport).
We have one of the receivers here at the Watson labs in Yorktown that we
purchased from Planetary Data Systems,
Inc.
Next is the maturity of the modelling codes. Although scientific
research continues leading to new or enhanced codes, several popular mesoscale
numerical weather prediction systems have become powerful and flexible
enough to be used for both research and operational efforts. We are using
a couple of these software packages, which we have customized.
The final factor is somewhat related to the first -- emergence
of low-cost platforms for visualization. This not only includes computational
capability, but also specialized hardware for interactive three-dimensional
rendering. Ubiquitous, cheap graphics cards, originally used in computer
games can be used for visualization of data generated by weather models.
These platforms now outperform expensive workstations of only a few years ago.
When we put all of these factors together coupled with a growing understanding of
the weather-sensitivity of many business decisions, we are able to implement highly
targeted forecasts focused on specific operational problems.
This is not the same as taking continental-scale forecasts
from the U.S. Government or private weather companies and tailoring the results to a
particular region or city. Each forecasting environment is customized not only by
the application but by the local geographic conditions and weather concerns, which are
not captured by the broader-scale weather models. In addition, we can avoid the
considerable computational cost associated with trying to scale weather models for the
entire United States to incorporate the more detailed physics.
What is the biggest challenge
for this project?
This project actually has several challenges, which make
it particularly interesting -- scientific, technical and business. All
of these challenges are still on-going. As discussed earlier, considerable
research is ongoing in weather forecasting, and in modelling and data analysis
specifically, the bulk of which is funded by government agencies and pursued
by both government and academic scientists. Even as our ability to observe
and simulate the atmosphere improves, the scientific understanding is still
incomplete, although our knowledge continues to grow.
Similarly, there are also a number of research efforts
in methods of visualizing complex data for a variety of purposes. We contribute
in this area using real-time weather modelling and observation as a testbed
for new ideas. In addition, we need to leverage the investment in observations
and the capability of modelling codes in an operational fashion. This implies
technical challenges in the implementation to make a system that can produce
usable forecast products in a timely manner.
But the biggest challenge for the viability of this project
has been on the business side. The kind of capability that Deep Thunder
represents is a new market. Since it is emerging, there is uncertainty
about its size, the willingness and ability of potential customers to invest,
etc. But given the magnitude of the relevant weather sensitivity, the potential
may be quite large, although not fully proven. Moreover, the complexity
and size can, in itself, intimidate potential investors. However, that
potential has been sufficient for us to work on this project.
Given our progress to date, we have engaged with a number of
potential customers to explore the interest and opportunity, especially
with local government agencies, airlines and surface transportation companies,
energy businesses, etc. We see that the Deep Thunder concept represents
a new market with enormous potential that can leverage IBM strengths in
services, high-performance computing, systems integration, etc.
What's new with Deep Thunder?
To evaluate these ideas, we created an initial operational version of
Deep Thunder for the New York City area. To do that we have put together from scratch
a modest weather laboratory in Yorktown, which we have used to support
our continuing development and enable us to run Deep Thunder operationally.
In particular, our lab consists of a NOAAport satellite receiver (which
provides us raw data from the National Weather Service), a System p
Cluster 1600 supercomputer (five 4-way and one 2-way Power4 nodes), a small RS/6000
SP supercomputer (eleven 4-way and one 8-way Power3 nodes) and nine 3D graphics workstations
(IBM Intellistations).
We used this facility to build an initial prototype system
to create high-resolution forecasts for the New York area. In particular,
we generate "nested" 24-hour forecasts at 16, 4 and 1 km resolution (areas
of 976x976, 244x244 and 61x61 km in size, respectively) centered over New
York City tied to multi-resolution visualizations. Fixed sets of qualitative
products (three-dimensional images and animations) are posted on an internal
IBM web site, updated at least twice per day. In our lab and the IBM Industry
Solutions Laboratory in Hawthorne, NY, we also have several interactive
visualization applications that support more in-depth analysis of the model
data as well as the ability to examine real-time observations made by the
National Weather Service. We have built an operational end-to-end infrastructure
and automation (data ingest, pre-processing, simulation, post-processing,
visualization, dissemination). This has been augmented with
new products geared specifically to users at certain geographic sites
as well as particular applications to enable their use as a service.
Our customers and collaborators access
these products via password-protected web sites outside of the IBM firewall.
Some of our forecasts in particular geographic areas
are now available for public access
via IBM alphaWorks Services.
The idea behind these model runs is to illustrate
how the forecasts can be tailored to the geographic region of interest and
specific applications by building an operational
infrastructure and experience to enable a viable and practical service
with both business and meteorological value. It also has enabled us to engage potential
customers with some real capabilities. Although we are generating regular
forecasts, we have sufficient capacity in the near-term for more runs
per day (e.g., on-demand, in response to customer needs or severe weather)
or to expand our current forecasts by length or geographic area,
as well as provide custom forecasts for partners, and continued development
for Deep Thunder and other weather modelling efforts.
For example,
we have extended these ideas to other metropolitan areas in response to
customer interest to enable services in these regions.
Similar nested
forecasts to 2 km resolution are now being produced operationally for the Chicago, Kansas City, Atlanta, Baltimore and Washington
metropolitan areas, and
1.5 km resolution for the Miami-Fort Lauderdale area.
In addition, experimental forecasts to 1 km resolution are being done for the
San Diego area.
The image below places all but one of these forecasts in a geographic context, which shows a
map of the eastern two-thirds of the continental United States. On the map
are three regions associated with six of the seven aforementioned metropolitan
areas. They correspond to the triply nested, multiple resolution forecasting
domains used to produce each high-resolution weather forecast.
The outer nests are in gray, the intermediate nests are in magenta and
the inner nests are in white.
The areas within the gray borders are covered at 32 km resolution for Kansas City,
Chicago, Atlanta and Baltimore/Washington, 24 km for Miami-Fort Lauderdale and 16 km resolution for New York.
The areas within the magenta borders are covered at 8 km resolution for Kansas City,
Chicago, Atlanta and Baltimore/Washington, 6 km for Miami-Fort Lauderdale and 4 km resolution for New York.
The areas within the white borders are covered at 2 km resolution for Kansas City,
Chicago Atlanta and Baltimore/Washington, 1.5 km for Miami-Fort Lauderdale and 1 km resolution for New York.
To provide a sense of the computational capabilities, all of the computing for any one of our
current, multi-resolution, 24-hour forecasts are completed in 30 to 60 minutes using twenty 1.7 GHz Power4
processors for computing and a single Power4 processor for I/O, which includes the post-processing
visualization to populate the web site. Once a run is initiated, the entire process from data ingest
to updating the web site is fully automated.
What type of companies might
find this type of technology useful?
We see a wide range of possibilities, including travel,
aviation, agriculture, broadcast, communications, energy, insurance, sports,
entertainment, tourism, construction and other industries where weather
is an important factor in making effective business decisions.
In general,
one might ask what is the potential business value of improved weather forecasts? As a start consider the fact that
"weather is not just an environmental issue, it is a major economic factor. At least one trillion dollars of our
economy is weather sensitive," Former US Commerce Secretary
William Daley.
A more recent study reported in the
Bulletin of the American Meteorological Society estimates that
one third
of private industry activities representing some three trillion dollars
annually has some degree of weather and climate risk.. A
partial summary of the market segment economic impacts is available.
For example, consider the more local and short-term impact of weather events.
For example, it has been estimated that the annual cost of under or overpredicting
electricity demand due to poor weather forecasts is several hundred million
dollars in the US alone. The value for weather forecast services for U.S.
households in 2001 was estimated at $11.4B. According to the Air
Transport Association air traffic delays caused by weather were about $4.2B
in 2000, of which $1.3B was estimated to be avoidable.
According to the United States Department of Transportation, about 7000 people are killed and
800,000 are injured each year in weather-related accidents on US highways. The economic impact of
these and other weather-related problems on the roads are estimated to lead to 544M vehicle-hours of delay,
and an economic impact of about $42B annually.
In addition, there is an emerging industry for weather derivatives
(as hedges against weather-related financial risk), which has grown from
nothing in 1997 to tens of billions of dollars today. Initially, this market was for
energy-related commodities, but has expanded to other markets like agriculture and retail. While
it focuses primarily on the seasonal scale, it may evolve to include the dynamics of the
short-term market, as the local impact of energy commodities grows.
A summary of these and other statistics is available from
the US Government.
Granted, these types of weather sensitivities
span a wide range of geographies and temporal scales. But a significant
fraction is within the "window" of one to two days for local weather phenomena.
Thus, we feel that there is significant potential in this emerging marketplace.
Is anyone else doing any research
like this? How does it compare to IBM's work?
There are a large number of groups at universities (e.g.,
University
of Oklahoma, Pennsylvania State University)
and government labs (e.g., National
Center for Atmospheric Research, NOAA
Forecast Systems Laboratory) working on mesoscale weather modelling.
After all, academic and development work in this field has been taking
place for about two decades. In fact, some of these groups are IBM customers
using the RS/6000 SP or pSeries systems as their computational engine. However, our efforts
are focused on the development of services and systems for potential commercial applications
of such simulations, which is quite complementary. Again, we are not inventing
new weather models, but adapting, refining and applying extant ones.
There are also a handful of small companies doing somewhat
related work, most of which began after our initial efforts at the 1996
Olympics and subsequent technology demonstrations. In fact, the existence
of these relatively new efforts is further evidence that there is a market
for such capabilities. A few other companies doing similar work are
established weather service providers who have added custom modelling on a broad geographic scale to
complement their traditional capabilities based upon
data products generated by the National Weather Service. For the most part,
these other efforts have concentrated either on the modelling and/or the
data assimilation portion of the forecasting system often with a focus on
just the meteorology. This is in contrast
with our efforts that have an emphasis on business-oriented services.
What's next with the project?
This is an on-going R&D project in science, technology and business.
While we have already established some
collaborations or pilots with leading-edge customers,
this is activity that will continue to grow.
These organizations are ones for
whom we built/adapted the service for their needs, and hopefully, will be expanded.
We are
currently in discussions with several companies in the energy, aviation
and other industries as well as local government agencies. We are
also exploring how the Deep Thunder concept could be leveraged within
IBM to provide a competitive advantage or to improve efficiencies in the
operations of specific organizations or facilities.
Some of our forecasts will become available to the public in the near
future.
In addition, we are doing some work to help determine the metrics
for measuring business value of our services.
Hopefully this will yield
further capabilities that could be utilized with other customers.
As part of the continuing
efforts, we are refining the quality of the model results, improving the
degree of automation, and developing new methods of visualization and dissemination.
What are some sample visualized
products?
Each of the three images below show aspects of a thunderstorm
predicted by "Deep Thunder". They were produced by a suite of interactive
visualization tools that are integrated with the modelling portion of Deep
Thunder. This suite also generates the visualizations that automatically
populate the New York City forecast web site.
The first image (above) shows a terrain map, colored by
a forecast of total precipitation, where darker shades of blue indicate
heavier accumulations. The map is marked with the location of major cities
or airports as well as river, coastline and county boundaries. In addition,
there are colored lines indicating predicted winds, with the lighter color
being faster winds. The lines flow in the direction of the predicted winds
as indicated by little arrows. Above the terrain is a forecast of clouds,
for which the typical "anvil" shaped structure of a thunderstorm cell is
visible. Within the clouds are cyan surfaces that correspond to rain shafts,
where the precipitation is forming.
The second image (above) illustrates additional details
about the properties of the forecast above and at the ground. Similar to
the previous image, there is a colored but flat surface. The coloring corresponds
to the hourly accumulation of precipitation on the ground. In the back
of the image, the thunderstorm cell shown in the first image is "sliced
open", where we see an irregular colored surface that shows the predicted
reflectivity inside the cell, and thus, the internal stucture of the rain
shaft. The amount of rain in the cloud is also shown via the brown surface.
The third image (above) illustrates additional details
about the properties of the forecast at the ground. The coloring corresponds
to surface temperature. The surface itself shows "lifted index", a variable
that corresponds to relative stability in the atmosphere. The areas where
the surface is more deformed (peaks and valleys) illustrate storm activity.
As in the first image, there are lines showing wind flow and local maps.
In addition, there are contour lines for predicted reflectivity, corresponding
to what a weather radar system would observe. The contours are concentrated
in the area of high lifted index and cooler temperatures, where the thunderstorm
cell has formed according to the model.
An additional image below illustrates the notion of the multi-resolution
nature of the forecasts by showing a forecast for a thunderstorm at 16, 4 and
1 km resolution using techniques similar to those in the first image above.
lloydt@us.ibm.com
Last updated September 25, 2006
