- Expert Views
Could your database be impeding your time to market?
14 January 2013 • Source: Joe Morrissey – VP EMEA, 10gen, the MongoDB company
The overhaul of the US regulatory system with the Dodd-Frank Reform Act rippled across global financial markets. At the heart of this reform lay the comprehensive regulation designed to increase the transparency of financial transactions in an attempt to help avoid a similar crisis in the future. The act and its requirement for compliance has forced the financial services sector to take a much closer look at their back-office systems due to the increased data volumes that are a consequence of the act.
Traditionally, investment banks – as all other organisations large and small – have relied on relational databases, with their rigid and tabular structures to store data. However, it is now becoming clear that these relational data stores are finding it difficult to cope with the enormous increase in data volumes and throughput that are now commonplace. Scaling relational databases is often prohibitively expensive due to the nature of their design.
This has led to many financial services firms reconsidering their default position of a relational model for their database architecture. Instead they are seeking alternatives that not only provide the performance and scalability at lower cost, but also introduce other benefits such as flexibility and agility.
In financial services today we are seeing a massive shift towards non-relational databases. Next generation (often referred to as NoSQL) databases are now moving into production environments in some of the world’s largest investment banks to solve some of the shortcomings that IT staff are experiencing with their current relational database installations.
NoSQL databases offer massive scale-out capability
NoSQL databases are designed from the outset to offer massive scale-out capability on commodity and virtualised platforms. This is distinct from how relational databases are usually scaled through the utilisation of increasingly large and expensive servers. The vertical method of scaling has always been much more expensive than horizontal scaling on low cost servers. This also has the advantage of starting small and then adding more servers as data volumes and throughput increase, thus controlling capital expenditure.
Additionally, the dynamic nature of a next generation database schema means that it’s able to cope much more effectively with the varied and constantly changing nature of data that financial services organisations now process and store on a daily basis.
For instance, back office risk departments looking to create a single view of counterparty exposure across the bank can more easily consolidate varying risk metric feeds from front office systems into a ‘schema-less’ database for further processing. In the relational world this would mean painstakingly creating a rigid schema to store the data ahead of time, and then changing it again every time a format changes. The dynamic nature of the schema within a NoSQL database, like MongoDB, means that the data can be stored without change to the database.
This notion has been used effectively in many other buy-side and sell-side use cases, from the global implementation of reference data systems where information of complex financial products can be represented simply and easily through to the ingesting of high volume FIX messages.
With a dynamic schema, new financial products can be stored and represented in the database without having to restructure many tabular structures in a relational database. Thus non-relational databases enable faster and more agile development, that can bring products to market faster.
In the months and years to come, the pace of the financial industry is going to speed up, competition is going to get fiercer and legislation is going to become more stringent. To deal with this, financial organisations will have to be able to operate in a fast, scalable computer environment without massive expense.
While NoSQL databases may not be the answer to every scenario or application, they do have a lot to offer in terms of meeting the challenges that financial services organisations are facing both now and in the future.