Sometimes Not Everything Runs Smoothly: We Apologize for Today’s Technical Problems
Table of contents
Sometimes things don’t go as planned. At Penta, we unfortunately encountered this today. For a few hours, we had a system failure which meant that you may not have been able to log in for a few hours. This problem has fortunately been solved. However, we would like to apologize for this incident and for any inconvenience it may have caused. We can guarantee that the security of your account and your deposits were protected at all times. Nevertheless, we would like to take a self-critical look at this incident and make sure that something like this does not happen again in the future.
We Care About Transparency, Even When It’s Inconvenient
Our goal is to provide you with hassle-free banking at any time. Unfortunately, we have not been able to do this at times today. We are aware that this situation is unacceptable and that we have made you another promise, which we will try to keep at all times.
Even though we are highly frustrated with today’s incident, we want to be transparent at all times—that’s why we inform you about such incidents. So that we can assure you that we have worked hard on the solution and have fixed the problem.
If we can’t keep our promise, we want to be upfront about it. At Penta, we don’t want to hide when things aren’t running smoothly. Rather, we want to be there for you and find solutions quickly so that you can focus on your business again.
Our business banking platform is designed to serve numerous customers while adding new functions very fast. However, today we encountered an issue with our backend systems caused by a combination of different factors. As a result, login and access to some services could only be granted with degraded performance.
But you don’t have to worry about the security of your business account at any time. This remains guaranteed at all times.
How We Try to Avoid Those Incidents
Our tech team is the first instance to identify technical issues, and all departments are brought in to accurately identify and fix the problem as quickly as possible. In order to move quickly from problem to solution, we at Penta operate under an “Incident Management” procedure.
How “Incident Management” Works at Penta
In every area where software is used, things can sometimes go wrong. For exactly such cases we have built up a so-called “Incident Management” framework at Penta, which consists of three phases to come to a quick problem solution:
- Understand: First, our software engineers try to locate and understand the potential problem.
- Act: Once the problem is discovered, the team develops a possible solution to the problem.
- Communicate: The next step is to communicate internally as well as externally to our customers. With constant updates, you always know what is happening and when restrictions may occur.
What Happened Today and How We Found the Solution
09:55am: Today, we became aware of an issue and immediately formed a taskforce. The goal of the taskforce was to understand why logging into the Penta account was taking longer than usual.
09:55 – 09:56am: As part of such analysis, we make use of various internal tools and mechanisms, for example with the purpose of monitoring technical systems. These systems are the starting point for the analysis. In this situation, we saw that there was a so-called “spike” that gave us the indication that we were dealing with an unusual system behavior. We were able to realize this issue within the first minute of becoming aware of the problem. This part of the process worked out as planned.
09:56 – 10:45am: From this moment on, it’s about testing different hypotheses to find out what’s going wrong from all aspects of our software. Some parts of the system worked flawlessly, which made it harder to find the problem in this case.
10:45 – 11:45am: A problem was fixed and everything was working for about an hour.
11:45 – 12:30am: An additional difficulty was added because the symptoms and the cause were not directly related. So it turned out that we had solved a symptom, but not the cause, which is why we kept having minor setbacks.
Some parts of the process didn’t work as planned. In the next steps, we will work with the team to develop further solutions that will prevent such situations in the future.
12:30 – 12:45am: A core problem was solved and the solution was tested.
1pm: Around 1pm, we were able to bring the system back up for our customers, as a large part of the original problem was solved. We made this decision even though some customers could still experience slow loading times. But we chose to sacrifice speed rather than sacrificing the overall service.
1:45pm: We were able to finally fix the bug and imposed a 30-minute slot of intensive monitoring before we started communicating to you.
2:10pm: Finally, we have informed our customer that no further restrictions are to be expected and all functions are working as planned.
We Are by Your Side
We are very sorry for the temporary system failure. Although our processes worked as planned during the incident, one core step of the process took too long.
We will now invest significantly more resources in this area in order to better understand, together with our developer team, how we can prevent such situations in the future and significantly reduce the time it takes to resolve an issue.
We are always happy to help
Once again, we as the entire Penta team would like to apologize and we are working hard to avoid those circumstances in the future.
Lukas Zörner (Chief Product Officer) & Henrik Jondell (Chief Technology Officer)