JADS Improves Data Forecasting

Challenge: Developing a Load Balancing Database

A public-owned electric utility in the Western/Northwestern U.S. needed to modernize its software for gathering meteorological and streamflow data. Their existing system was difficult to configure and maintain and could not handle the number of data sources available via the internet. This data can significantly impact plant usage planning and operations. For example, by knowing the depth of the snowpack, river flow, and forecasted temperatures, it is possible to make proactive decisions about dam operations to maximize generation and avoid flooding. The solution the utility envisioned does exactly this.

The program would need the ability to read thousands of sources of relevant data in a variety of formats. The system would gather data at various intervals, ranging from minutes to days, and would process disparate data formats, requiring significant transformation before being loaded into their data analysis systems. Additionally, the utility anticipated that the load on the system could double in the first few years of use. Because of the mission-critical nature of the application, the high-performance requirements, and the need for customization, they sought out IntelliTect to assist their team in developing the architecture and building an initial product.

Solution: Creating Architecture through a JADS

The utility has since successfully partnered with IntelliTect for several other projects and engaged IntelliTect for a Joint Architecture Design Session (JADS). One of IntelliTect’s software architects, Grant Erickson, met with their team, and together they walked through the process of creating an architecture for the project. During this weeklong JADS, the team diagrammed the key use cases, developed component models, and decided on key implementation technologies. Additionally, the team discussed and documented design decisions driven by specific requirements around auditing, performance, and enterprise needs. By the end of the JADS, the utility had a path ahead for implementing the solution.

The electric utility jump-started the project by having IntelliTect build a vertical slice of the solution and all of its components. A vertical slice is an implementation of the basic building blocks of the application. This vertical slice included a running solution with an initial structure for each of the major components in the application.

Based on the architecture, it was clear that a modular framework would be needed to meet both performance requirements and the necessity to add new data sources rapidly. As a result, the utility elected to have IntelliTect continue developing the core framework for the application. During this time, the teams worked jointly to ensure the framework met the utility’s needs. Once the framework was completed, IntelliTect provided on-site training in order to successfully hand off the framework to the utility’s development staff. IntelliTect transitioned to a support role as the utility’s team built out the rest of the application and began setting up thousands of data feeds.

The initial phase of the project included a simple web interface for statistics and a basic WPF-based interface for creating new data feeds. As the project progressed, it became evident that the user interface needed a high degree of sophistication and flexibility in order to create and manage the large number of feeds required. IntelliTect’s WPF expert worked remotely with the utility’s developers and together using pair programming, they built the key pieces of the UI application. As a result, the utility has been able to continue the development and maintenance of the application internally, while retaining IntelliTect for ongoing support and training for existing projects and new technologies.

Results: JADS Solution Doubles Database Projections

The solution is currently in production, handling 9,000 data feeds running 80,000 times daily. The database has more than two billion rows with five million added daily. These numbers more than doubled the utility’s initial projections for the first year. They estimate that the data volume in the next year will double once again. A three-member team at the utility has been maintaining and enhancing the solution on their own for the past year.

The success of this project has led to other engagements with IntelliTect including a compliance application for power transmission and generation, as well as a new enterprise application portal framework built on ASP.NET Core.