In every industry there is a real push to make sure you can be confident in the data that you hold. That means understanding how data is captured and stored, making sure its accurate and complete and then ensuring you have timely backups. An old mentor of mine always said, the challenge with data is, “shit in, shit out”. His comments were as true then as they are today. How many systems and platforms around the world and in every industry (especially financial services) is hampered by poor data integrity, out of date data or even worse, fields of data that just have garbage in them, or nothing at all.
Now how many systems struggle with data because it has been unexpectedly changed? The output therefore is inaccurate because data has been corrupted or worse still, tampered with? In the world of financial services, the ability to ensure your data is all present and correct is almost everything. Because of this challenge, there has, over the past years, been a real drive by certain technologies to push themselves as the de facto solution for immutability, I’m thinking blockchain here. Not only does blockchain make a strong case for it to be the “ledger” of truth for financial services, but it also makes a strong case for its ability to ensure operational resiliency.
On the surface here, blockchain may look like the perfect solution, but there are several challenges associated with that. Is there a better way than using a typical blockchain architecture? Does it really fit with financial service providers, more importantly, does it fit well with financial market infrastructure (FMI) providers? This one is of great interest, as many push a blockchain architecture to address current challenges with cross-border payments. But, is there something else?
Let’s look at the 3 big challenges for ledgers regarding ensuring the data they hold is all present and correct:
Data corruption
When we were setting up ClearBank, I had many a discussion with the Bank of England, the regulator and certain payment systems regarding data corruption – especially with regards to data corruption being replicated across nodes. The point is highly valid, that corruption can in some implementations replicate across your multiple nodes, and therefore into your backups. That makes the issue of understanding your RPO especially hard, simply because you need to understand at what point that corruption took place.
Cyber attack
A big challenge with a cyber-attack, especially when it targets your underlying datastores is, has the attack damaged your data – has it been altered. Again, this can be tough to address, especially with regards to your RPO. Sure, there are ways to capture these changes, looking through logs etc. but you ultimately rely on your data backups at this point, and again, identifying which data backup is accurate / pre-attack. This really can play havoc with your RTO and RPO as there is simply a lot of work that must be done to identify where data has been changed, and the impact that change has had going forward. In the world of financial transactions, you can’t just unwind every debit/credit that occurs after the point of tampering.
Insider threat
This is pretty much the same issue/risk as that which is presented by a cyber-attack. The difference here is, your logs may have easily been tampered with too, since the threat has come from an internal source. Granted this could also be true with a cyber-attack, but it is more commonly associated with an insider threat. So, in this case, how do you prove your data is correct?
The solution: An Immutable datastore, an immutable ledger
Most financial services ledgers are not immutable. There will be lots of operational and some technical workarounds to try and ensure data isn’t being tampered with, but at the end of the day, on the main, the data held in the worlds financial systems is not immutable.
Immutability of a ledger is common with hash chains and blockchains alike. So clearly the technology, which has been around a lot longer than most are aware of could be a great way to prove data is accurate, and that it hasn’t (or even better) or cannot be tampered with. For this section, I am going to split it into a typical blockchain approach and a not so typical blockchain approach…
A traditional blockchain solves the issue of immutability, but it sadly presents a few others, these include:
- The integration overhead (cost and lines of code to maintain)
- Underlying infrastructure requirements
- IT support considerations, good old People Process and Technology (PPT)
- System performance considerations (your choice of blockchain – will it be quick enough)
- Data residency, where is the data being replicated
The five points above are major deal breakers, they render a blockchain based solution as pretty much a non-starter, if your system is a high volume transactional based system that spans different legal jurisdictions (geographies). This though hasn’t stopped many experts, central banks, banks etc investigating just what the technology could deliver in this space, and rightfully so. The quest for immutability and resiliency is always going to be core to any transactional type of system, especially a payment system.
A traditional blockchain solution would provide several nodes across the geographies in which you want the system to operate, so that would mean spanning different legal jurisdictions if you want something that could work for cross-border use cases. Several nodes provide great resiliency and the nature of a blockchain provides that immutable ledger. However, simple distance between nodes introduces latency, that’s before you look at the blockchain implementation. The location of the nodes also causes so many challenges regarding data replication and therefore residency. Can you imagine a regulator in one jurisdiction demanding to see data on all aspects of your solution because data is no longer located just in that specific jurisdiction? We’ve seen numerous examples of this inside and outside of the financial services sector. So right now, data replication and residency is a deal breaker for the technology.
So, while a traditional blockchain solves the immutable ledger challenge, it introduces many other challenges. All too often we see examples of technologies desperately seeking a problem to solve, in some ways this is blockchain in the traditional implementation sense, especially within the financial services sector. I personally believe in trying to solve the problem and identify the technology that best fits….With that in mind, how have we solved this at RTGS.global? What do I personally believe is the right approach/solution?
Database first, immutable ledger second…
A traditional database is suited to high transactional throughput and data storage. It’s proven to be the go-to tech for decades now, and with evolution of this technology, like Microsoft’s in-memory Azure SQL tables, the performance of a database is lightning. Couple this with micro-service-based architectures and really, it’s hard to see a better technology for high transactional based systems. Database technology now also includes concepts of nodes and high availability, couple this with a good old backup strategy, and almost all of the challenges are solved. Now, if you are smart with your implementation, then you can also ensure data residency and that’s true even for cross border solutions like RTGS.global.
The one challenge remaining is verifying that data is accurate, that it hasn’t been tampered with. This is where I need immutability. But the ask here is subtly different, which gives you room to use technology in a different fashion. Here I am asking to prove a traditional data store data is accurate, I am not asking for the primary data store to be immutable and I am not using that primary data store as a form of messaging across geographies…The solution is therefore a way of utilising an immutable ledger (like a blockchain) to confirm / verify the accuracy of my database. Now, if you can do this, then you have all the benefits of traditional database systems, including the ease for developers to use it, coupled with all the benefits of a blockchain. Enter Azure SQL Database Ledger.
Azure SQL Database Ledger
When we first started working with Microsoft on RTGS.global, our ask was a way in which to prove data was accurate. We had assessed many blockchain implementations and concluded that the technology added more obstacles to building out RTGS.global than challenges it solved. So, our ask of Microsoft was how do we prove immutability and verify our data is accurate. Luckily, we were able to work with them on Azure SQL Database Ledger.
Now for those of you who do not know what this is, at a macro level it is essentially an Azure SQL database. However, the data that you capture is also backed off into a blockchain implementation asynchronously. What this means is you don’t have the blockchain slowing you down, in terms of how long it takes you to leverage the technology, but also in terms of you transacting. In this case, the blockchain is in the background building out the immutable ledger.
The ledger can build “digests” which too are in themselves immutable, as they are stored and accessed within an Azure immutable storage. The digest can then be verified against the data held within a database by running a simple Stored Procedure (SP). The output confirms that the data within the DB is all accurate, that it hasn’t been tampered with and therefore, you have solved your immutability challenges.
At RTGS.global we run a distributed approach to our data storage, essentially each jurisdiction in which we operate has its own Azure SQL Database Ledger, that follows a high availability model in that jurisdiction. Data residency is within the specific jurisdiction, with the blockchain that backs off of Azure SQL Database Ledger also being private and located in that jurisdiction only. We also showcase to central banks and regulators the immutability of the data within our ledgers by providing a portal view to the digest verifications that we run.
For us, the solution is simple, its elegant, highly powerful and solves all the challenges of performance, residency, people process and technology and obviously, delivers immutable data storage.
Many of the worlds financial systems ledgers are not immutable, and that’s a grave concern/big risk. However, a traditional blockchain ledger isn’t the solution. The solution is to use blockchain technology to prove data integrity, use the technology for what it is great at doing, delivering immutability.
For more information on how RTGS.global uses Azure SQL Database Ledger, check out the Microsoft case study here: Microsoft Customer Story-RTGS.global automates financial transfers, unlocks trillions in capital with Azure You may also want to look at the technology in more detail yourself, then this is a good place to start, Announcing Azure SQL Database ledger – Microsoft Tech Community
For my very technical readers, if you really want to get under the hood of the technology, then read these two blog posts from Jason Anderson
Azure SQL Database ledger PART 1 by Jason M. Anderson – SQLServerGeeks
Azure SQL Database ledger PART 2 by Jason M. Anderson – SQLServerGeeks