The New Data Imperative

Since 2006 the global economy has been experiencing slowing growth and increasing volatility in asset prices, and we have recently seen a near-complete breakdown of the cheques and balances built into the global financial system since World War II. At this point it is still unclear when and how the recovery will begin. However, the global system is at a juncture where major structural reforms are clearly needed before progress can be made. In an economy, financial markets intermediate the capital flows and allocation of investment pools seeking a predictable risk-adjusted return, and currently neither the projections of returns nor the evaluations of risks to capital are predictable. It would be instructive to review how we got to this state of affairs before we focus on the elements that will form the path to recovery.

The mission of institutions in financial markets can be summarised in three interrelated objectives:

• allocate investment capital
• manage the risk to capital in an optimised way
• service and retain customers.

Sell-side institutions raise capital and create and sell securities to investors to finance economic activity. Over time, new investment products such as derivatives (options, asset-backed securities, securitised loans, etc.) have been designed to mitigate various types of risk and cater to investors' differing return/risk tolerances. Derivatives were developed to enable distribution of risk to those best equipped to assume specific types of risk, thus eliminating extreme concentrations of risk in one or a few firms, or in certain asset classes. Beginning in the early 1990s, the number of derivative instruments proliferated to provide risk mitigation for more complex credit- and asset-backed securities at more granular levels. As derivatives are traded at multiple venues, the transaction volumes and frequency of pre-trade analytical operations in real time increase dramatically. As a result, the information that needs to be maintained online to support a diverse array of user applications grows exponentially.

With the transition to decimalisation in securities pricing in the last decade, the possible price points between two point values increased by orders of magnitude, and the baseline data volume for complete price and quote histories exploded. However, the need for retaining all granular data was not immediately apparent. As the use of quantitative models and algorithmic trading spread, this granular data became critical to understanding historical price trends and volatility. Retaining granular data for pre-trade and post-trade analytics, as well as quantitative strategy development, in turn heightened the demand for very large volume, online information repositories.

The globalisation of trading, increases in the number of trading venues and the cross-listing of shares on multiple exchanges all significantly raise the volume of high-value data that needs to be available for trading, portfolio and risk management decisions. The shift to model-based trading and risk analytics also increases the frequency of access and data consumption of a diverse user community. As an extension of this trend, electronic/program trading evolved into algorithmic trading in the late 1990s. This is now rapidly emerging as a dominant trading strategy (see Table 1).

In relation to trading capacity, quote and message volumes have surged as a result of decimalisation (see Table 2). In addition, electronic communication networks (ECNs) and the ability to route orders to multiple trading platforms have led to a decrease in average trade execution size, as institutional investors exploit effective means for executing large blocks of stock without perturbing market behavior.

Portfolio and trading decisions are being automated faster and rely on sophisticated quantitative models that incorporate real-time market data into the algorithmic processes. The capture rate of the data and the latency with which it is used in the application systems have become key differentiators in determining profitability at the trading desk. Price arbitrage opportunities exist for only small fractions of a second. During these short periods the fastest and most agile market players have a narrow window to secure the best prices across multiple trading venues and thus avoid less profitable trades. Reducing the latency between a market data event and submitting an order or providing a quote has become a key competitive advantage. Both proprietary traders and buy-side clients rely on fast access to markets to help them win the game. Low latency also helps traders mitigate risks by allowing them to unwind positions and pull quotes off the market before they are exploited by others as market conditions change.

In addition, regulatory initiatives such as Regulation NMS (National Market System) in the United States and MiFID (Markets in Financial Instruments Directive) in the United Kingdom are causing market fragmentation. One consequence is exploding amounts of market data that must be captured and processed in real time. Furthermore, frequency of trade order at declining average order sizes is also rising. To give an example of what this all means, the daily rate of quotes on U.S. listed securities recently rose by 250%.

In this environment, profits depend on rapid decisions about whether to trade based upon the ability to monitor market dynamics and the impact on portfolio positions. In addition, changes in market structure are opening the playing field to new exchanges and multilateral trading platforms, even as they drive down latency for transactions from milliseconds to hundreds of microseconds.

The growth in messages on Wall Street, from 120,000 per minute in June 2005 to 907,000 per minute in June 2008, creates a pressing need for enterprises to search speedily and accurately through these unending and ever-expanding data avalanches to perform mission-critical business analysis or meet demands for legal or regulatory compliance. Message growth rises with the frequency and volume of trading. On Wall Street the average number of trades a day has increased from about 5 million in January 2000 to 25m in January 2008. Today, Wall Street is dealing with more than 400m traded shares a day, 150 million shares more a day than in 2000.

These developments in capital markets have driven innovations in technology for real-time data streaming, execution models for running queries continuously against live data, and the ability to correlate real-time data, historical trend data and reference data.

"For traders the real challenge is to make sense of this fire-hose of information that is squirting at them," Brian Traquair, president of capital markets and banking systems company Sungard, told the Financial Times in 2008, indicating that they need more memory in their computers and faster systems that can compensate for interruptions in the data flow. This has meant addressing performance issues to enable tens of thousands of messages per second per server, with processing latency measured in milliseconds. Clustered models have also been developed for streaming data to improve reliability and provide high availability and thus meet critical enterprise requirements.

Low latency has traditionally been very expensive to achieve. Network infrastructure and wide area links in particular have been costly to set up and operate, leading to the sharing of resources; building a dedicated solution for low-latency trading was possible only for the major players. However, this has changed in recent years, and taking an end-to-end view of all the components of the trading platform and market data can deliver both cost savings and lower latency. Low latency is no longer limited to the largest players.

By exploiting low-cost hardware, the ever-decreasing cost of network connectivity and pre-packaged software solutions, cost-effective market data and hosting services can be used to transform the economics of low-latency trading platforms. Although significant investment may still be necessary, an efficient trading infrastructure with consistently low and reliable latency will enable traders to exploit rapidly changing market conditions. With estimates suggesting that a one-millisecond improvement in latency could be worth up to $100m a year to a large investment bank, the size of the prize should not be underestimated. Not surprisingly, the industry as a whole is moving toward lower latency.

As the financial services industry tunes its risk management operations to eliminate the weaknesses exposed by the credit crisis, new regulations may significantly increase official scrutiny of business operations. Adding impetus to this and other likely moves is a new perception that measuring risk is more complex than previously thought. For this reason tighter regulatory reporting requirements and a deeper granular level of detail can be expected. Among the other changes we think we'll see is the need for companies to detail financial results by product line or business unit. This in turn will generate new requirements for the IT organisation to manage earlier information availability and flow to decision support systems.

Organisations might be able to accelerate processes to address latency, but that doesn't necessarily mean that they will be able to do all the right things faster. The growth of trading data alone – according to one estimate, the volume of market data messages will soar from under 4 billion messages per day in 2006 to nearly 130 billon per day by 2010 – is putting a major stress on system capabilities.3

In addition, the biggest problem for risk groups is that they don’t have the systems to evaluate large exposures when banks make very large investments. This can lead to huge losses if the risk is not analysed. Large exposure could result from changes in market variables, such as interest rates, or from counterparty risk. When banks take big positions, as in foreign exchange markets, real-time risk management should indicate when they should hedge their positions. Banks, moreover, need to examine the risk of investing in different securities and how the different securities are tied to each other. If one business unit of the bank has a large position in IBM equities, for instance, and another unit has a position in IBM bonds, the two positions need to be evaluated in tandem.

Thus, the recent changes in capital markets – with new ones expected in 2009 and beyond – will require trades and investments to be processed more rapidly and accurately than ever before. Coupled with this is the need to do risk analysis as rapidly as data on trading and investment positions is compiled.

Historically, innovations in information and communications technology that have improved crisis resolution mechanisms have been a key driver of global productivity growth. This is borne out by the analysis of post–World War II financial crises by economists Carmen Reinhart and Kenneth S. Rogoff, cited in chapter 1. "Multifactor productivity," which includes technological change and spontaneous organisational and production improvements, more than doubled its contribution to the gross domestic product in the United States between 1995–2004, compared with 1980–1995. As a new Wall Street takes shape, the question is whether technology, in the form of risk control and analytics, can hasten the process.

As professors Reinhart and Rogoff show, a key element in every recovery has been finding better ways to turn incomplete, confusing information about new financial risks into profitable knowledge. For today’s capital markets firms, the focus must also be on making market information and trading knowledge available and visible to decision makers far sooner than they have been doing.