Bloomberg｜The Open Source Way of Financial Giants

https://www.bilibili.com/video/BV1Be4y197Ac/?aid=556180904&cid=779797507&page=1

Introduction

Bloomberg has many labels: a global financial media giant; the founder served as the mayor of New York and ran for president of the United States; it is also one of the largest technology companies in the world. Decision-making and trading tools, which still generate 85% of Bloomberg's empire to this day, are the coveted but unshakable cornerstone of both established media conglomerates and up-and-comers like Thomson Reuters.

However, behind these prominent labels, Bloomberg is not only a heavy user of open source software, but also an important contributor to many super open source communities such as OpenStack, Apache Foundation, and C++.

Hello everyone! I am Koala. Bloomberg has a lot of technical investment in cloud computing infrastructure, big data, and programming languages. Today, we will explore the story of it and open source.

What kind of existence is the Bloomberg terminal?

As members of the public, we routinely receive business, financial and news information from Bloomberg. As a professional organization and individual, it may be the Bloomberg Terminal, the Bloomberg Terminal, that we rely on every day.

Bloomberg LP, founded in 1981, launched the Bloomberg Terminal the following year, hoping to use groundbreaking technology to bring comprehensive changes to the slow and inefficient industry at that time, and to provide securities brokers with easier jobs. tool.

Although the annual rental fee for a Bloomberg terminal today is as high as $24,000; although it still retains the old interface and cumbersome keyboard, nearly 330,000 subscribers from around the world rely heavily on it, and Owning a Bloomberg terminal is a symbol of identity.
So, what kind of weapon is the legendary Bloomberg terminal?

Bloomberg Terminal is a unified platform that provides professionals with Bloomberg professional services and solutions, with tens of thousands of databases and applications. It integrates financial and peripheral information on one platform, providing users with authoritative economic comments and opinions; at the same time, with the Bloomberg terminal as the core, it also provides a variety of financial solutions; in addition, it creates an instant messaging system Instant Bloomberg is known as the Facebook of Wall Street, where traders exchange prices and even gossip on various financial products. I IB you has become a code word in the financial circle.

Bloomberg's "news + terminal" model is a moat that is always difficult to imitate and surpass. To this day, the services provided by the Bloomberg Terminal are still the source of 85% of the income of the Bloomberg empire.

The Technical Challenges of Building a Bloomberg Terminal

Every day at 9:29 a.m., when the opening bell of any financial market in the world is about to ring, a flood of market data will also flood into Bloomberg's data infrastructure hub, Ticker Plant.

Bloomberg itself does not create data, but, every day, Ticker Plant stores and transmits transactions and quotes entered by various financial instruments around the world, processing hundreds of billions of data messages, 2 million pieces of news information, and more than 1 billion pieces of data. Email and instant messaging, and then aggregate, enrich, store and transmit these events to the Bloomberg terminal.

Bloomberg's 6,000-person technology development team, right at the Ticker Plant, builds infrastructure and tools including hardware, applications, networking and interfaces.

Due to the regional distribution of the global financial market, Ticker Plant not only needs to operate 24 hours a day and night, but also needs to deal with peak data of multiple periods. These all present Bloomberg with technical challenges of scalability, high performance, low latency, and stability.

Open source users and contributors

Faced with these challenges, Bloomberg adopts the principle of "open source first". As Bloomberg open source community lead and CTO office member Kevin Fleming explained in an interview: "Where possible, we prioritize the use of open source tools over commercial tools in the development of applications and infrastructure. When source code is available, you have the flexibility to understand how it works, modify it, and support it; not affected by vendors disappearing, or priorities changing. Open source tools allow us to control our destiny Important enough that we could even form an internal team to provide support and enhancements to the tool.”

At the same time, Bloomberg will also give back some enhancements to the open source community in due course with the open source concept of Take…and give back.
Next, we will explore their choices and contributions to open source software from three directions: cloud computing infrastructure, big data, and programming languages.

Cloud Era Infrastructure - OpenStack and Ceph

Not long ago, the OpenInfra Foundation behind the open source cloud software OpenStack issued a document welcoming Bloomberg as its gold member.

In 2013, when Bloomberg was building its private cloud infrastructure, the design of the technology stack put forward requirements for high availability, horizontal scalability, robust architecture, and deployment and operation in an offline state.

In the choice of virtualized infrastructure, there were both commercial products and open source software in the market at that time. In the end, Bloomberg chose to deploy and run a series of open source software on the open source OpenStack, such as Hypervisor OS using Ubuntu, object and block storage using Ceph, database using MySQL+Galera, message queue using RabbitMQ3, log using Elasticsearch Kibana.

A year into OpenStack, Bloomberg has hundreds of servers running OpenStack in its data centers.

Speaking of contributions to OpenStack, Bloomberg once said that when he joined OpenStack ten years ago, although there were a large number of vendors and startups in the community, there were not many mature companies. With the addition of Bloomberg, more mature companies have joined the OpenStack community, adding many new functions to the OpenStack ecosystem and supporting the development, development and success of the OpenStack cloud platform.

For storage in cloud infrastructure, Bloomberg chose Ceph.

Ceph is a unified distributed storage system with high performance, high availability, high scalability, and rich features that support multiple storage interfaces for block, file, and object, and support multiple language drivers. And at that time, after years of development, Ceph has been supported and widely used by many cloud computing vendors. Both Red Hat and OpenStack can integrate with Ceph to support backend storage for virtual machine images.

At Bloomberg, its storage engineering team, which designs, builds and maintains storage for Bloomberg Terminal's 6,000 application developers, manages more than 100 petabytes of capacity and internal cloud.

Due to Bloomberg's investment in storage, when the global stock market plummeted due to the epidemic in late March 2020, the data from the global capital market on the Bloomberg terminal almost doubled, and the amount of information on some days even reached 2,400. billion, but Bloomberg's storage system has stood the test.

Apache Software Foundation

Bloomberg's extensive use of open source software has also brought them into contact with the Apache Software Foundation. Currently, Bloomberg is its Gold Patron.

Since 2012, Bloomberg has participated in the development and contribution of dozens of Apache open source projects. These projects support Bloomberg's data science workflows, data processing workflows and web services, as well as other internal and external services operated by the company.

Among these open source projects, there are well-known Hadoop and Solr.

For Bloomberg, the technical challenges they face include large-scale data storage, low-latency retrieval, massive requests, and high availability in a distributed computing and storage environment. Hadoop and HDFS, HBase, Hive, Oozie, YARN/Spark and other open source software are the basis for their large-scale, high-availability data applications.

Currently, Bloomberg's Hadoop infrastructure team manages clusters that can handle 10 petabytes of storage, spread across 1,000 servers, and running hundreds of billions of requests per day.

The search engine is another great feature of the Bloomberg Terminal. Bloomberg's search infrastructure team believes that search isn't just about text, it's about numbers, and lots of numbers. To enable clients to study, judge and make decisions on these numbers, Bloomberg must provide flexible, accurate and timely analytical tools. And Solr is an important open source software that Bloomberg uses to build these analysis tools.

Apache Solr is an enterprise search platform written in Java that provides a range of advanced search-related features and is widely used in enterprise search and analytics use cases because of its strong scalability and fault tolerance.

In 2015, the Bloomberg search infrastructure team, using tools built with Solr, provided search services for more than 100 applications. By 2019, Bloomberg Search, built with Solr, could unify information from thousands of Solr cloud services distributed across hundreds of machines.

The best C++ engineers in the business work for Bloomberg

Recently, the title of an article was this: "The best C++ engineers in the business are working for Bloomberg".
At Bloomberg, there are nearly 6,000 engineers writing C++. Arguably, Bloomberg is one of the biggest C++ users in the financial industry and the number one sponsor.

Bloomberg's love of language stems from its engineers' belief in the importance of choosing the right tools. Most of their software is written in C++, JavaScript, TypeScript and Python, and they are also active contributors to these open source communities.
Bloomberg and C++ have a long history. Initially, the server side of the Bloomberg Terminal was developed in the programming language Fortran, then it was written in C, and since then, both the client and server sides have transitioned to C++ and embedded JavaScript.
C++ is one of Bloomberg's core programming languages. In the millisecond-sensitive financial industry, code must be scalable, reliable, and low-latency. As such, Bloomberg builds everything from low-level libraries, components, to the highest performing financial applications and trading systems in C++.

The widespread use of C++ in Bloomberg is reflected in the fact that Bloomberg has tens of thousands of C++ projects and about 2,000 C++ libraries. It also established a special development environment group BDE, responsible for providing a set of C++ software libraries, development tools, and methodologies for thousands of Bloomberg developers.

Bloomberg's contributions to the C++ community are also unique. While modern C++ offers many new features, the stakes remain high when dealing with large enterprise applications as the language becomes more complex. With the accumulation of historical experience in C++ and practical experience in large-scale applications, at the end of last year, Bloomberg legendary programmer and C++ expert John Lakos and their other three senior C++ experts also co-authored "Embracing Modern C++ safely" book, shares and reminds everyone to be careful about some of the features of modern C++.

ending

For Bloomberg, open source isn't a buzzword, it's an everyday job.

Just two months ago, Bloomberg open-sourced Memray, a Python memory analyzer, and quickly gained attention on GitHub.

At Bloomberg, more than 3,000 engineers use Python to code and run data analysis. Memray is a project designed to enable developers to analyze allocations in applications to help reduce memory usage, find memory leaks, and identify hot spots in code that lead to inefficient allocations.

This example of Memray once again proves Bloomberg's open source first philosophy.

Because of the Bloomberg terminal, Bloomberg has surpassed the 150-year-old Reuters Group to become the world's largest financial information service provider in just 20 years after its establishment.

Forty years have passed, and the Bloomberg terminal is still the evergreen tree that brings the main income to the Bloomberg empire; and no matter how old-fashioned media groups or newcomers want to copy and get a piece of the pie, the Bloomberg terminal is always the one that Bloomberg cannot be used for. Shake the cornerstone.