The optimization problem behind net congestion
In this blog series we show that our mathematical strength is really an important addition to our programming skills. With a mathematical approach we provide solutions that cannot be achieved with programming alone.
The blogs are based on interviews with colleagues who recently completed a project in which mathematics played a decisive role.
In this second contribution we speak with Max and Bas, who together carried out a nice project for TenneT. “Due to grid congestion, many companies cannot get the electricity they need. The aim of this project was to predict how much capacity of the high-voltage grid is available, and how you can allocate it fairly. You do not want to allocate all capacity only to large parties, but also not only to small ones. This offers customers perspective on the end of the effects of grid congestion.”
How nice that you contributed to this, Max. How was the capacity of the high-voltage grid originally divided among the parties?
“The existing algorithm was an attempt to incorporate a manual, pragmatic approach of the grid strategists into software. In this way, a fair power distribution was calculated, but without mathematical substantiation. Unfortunately, this algorithm sometimes gave incorrect results. That is of course difficult if you want to ensure fairness and transparency.”
What was your approach?
“My colleague Bas has extensively analyzed the problem and formulated it mathematically as an optimization problem. This allows you to automatically calculate the best possible distribution, based on clear rules. This way you can replace an algorithm that is difficult to maintain with a transparent and robust calculation method.
How did you translate this optimization problem into a concrete program?
“For this we used existing calculation libraries such as SciPy. This significantly reduces the amount of code that needs to be maintained. This allowed us to focus on what really counts: the mathematical logic behind the problem. We used this to discuss with the customer about optimal power distribution.”
What did this achieve?
“We managed to solve the problem reliably and efficiently. Thanks to our mathematical knowledge, we were able to significantly improve calculation performance, for example by using techniques that only work if you clearly understand the underlying mathematical model.
We also added smart controls and diagnostics: the system now also shows why a certain solution is good, or where there is still room for improvement. That makes it easier for our customer to have confidence in the outcome and to substantiate decisions.”
How where the domain experts involved?
Bas: “During the project we worked intensively with the domain experts of the grid operator. They could indicate which outcomes were not desirable in practice, and together we adjusted the model accordingly. Consider situations where certain customers must always be given priority, or rules that guarantee that no one is worse off compared to the previous year.”
What did the customer think of the development of the project?
Max: “At first it took some getting used to for the customer, because it was a very new idea. But when we showed that the new approach not only delivered better results, but was also much easier to maintain, they were convinced.”
Bas: “Even after the second phase of this project, the customer is very satisfied, but they still have wishes that we could help them with.”
Why should other customers also ask VORtech for help?
Many organizations struggle with issues in which something must be distributed fairly, optimally or efficiently. This often involves mathematics, and that is where we can help. The special thing about VORtech is that we are good at applied mathematics as well as at creating efficient software for calculations. This combination provides solutions that go beyond a prototype and can really bring operational solutions.
How can they get in touch with you?
The easiest way is to fill in the contact form. We will then contact you directly to see if we can help.
The mathematical details
The original distribution of grid capacity was based on a manually designed algorithm, which included a lot of domain knowledge. Unfortunately, this sometimes led to undesirable results.
We have reformulated the problem as a mathematical optimization problem. We used SciPy, but the standard routines within this library were too slow. By using, among other things, a sparse Jacobian, we were able to significantly reduce the calculation time.
The tolerances also required attention. Stricter tolerances sometimes gave better results without visibly changing the target function. This turned out to be due to the structure of the target function. We added a preprocessing that solved that particular problem.
The objective function itself turned out to have a singularity at zero (when a party gains nothing compared to the current situation). Therefore, we constructed initial values where all parties receive a very small amount of capacity above the current situation, while still meeting all boundary conditions.
In addition, we added diagnostics: for each solution, it is indicated why it cannot be even better, or how far one is from an optimum.
Finally, we have adjusted the method by which we guarantee monotonicity several times: all customers receive at least as much capacity annually as the previous year.