Back in 2016, traders at RBC Capital Markets were using rules-based computer programs to optimize buy-and-sell decisions. However, fierce competition in the space required continuous re-optimization to maintain the best technology, so they had a decision to make: try to continue to bridge the gap with incremental steps—or attempt a bold leap forward.
The challenge: A trader’s job is to try to get the best price when buying and selling securities for customers under prevailing market conditions. But a financial giant like RBC often has to trade huge quantities of securities for institutional clients. Just dipping a toe into the market can change the price of the securities it’s trading and leak information to other market participants.
The backdrop: The team at RBC Capital Markets was already testing a form of artificial intelligence called supervised learning. In supervised learning, an algorithm is trained to identify patterns in sample data, then uses what it learns to make predictions about real data. But the team was struggling to get the technology to make predictions that were actually useful. What’s more, the algorithm couldn’t adapt on its own if conditions changed—for example, if the market plummeted. “Nothing we were doing at that point had a big enough impact to make it worth proceeding,” says Steven Szeto, head of Global Equity Execution Strategy at RBC Capital Markets. “We needed to find a different approach.”
The promise: The artificial intelligence world had begun buzzing about reinforcement learning (RL), an AI technique that uses an “actor/critic” model: The agent takes an action on its own, is rewarded or penalized for it, and learns to perform better over time. RL is particularly good at operating within fast-changing, complex environments-like a strategy video game, say, or trading.
The inspiration: Video games have become the proving ground for AI. So it was big news in 2016 when AlphaGo, a reinforcement learning program developed by AI pioneer DeepMind, beat the (human) world champion at the strategy game Go. “I took a lot of inspiration from that success,” says Hasham Burhani, lead AI scientist at RBC Capital Markets. Burhani, along with colleague David Shih, would play a key role in turning that inspiration into action.
The new challenge: Reinforcement learning had been slow to get off the ground in real-world applications. It’s only recently been adopted for purposes like cooling data centers and optimizing wind farms. The stock market—a dynamic environment containing a myriad of agents using diverse, changing strategies over time—seemed like it was the perfect fit for RL. But to the consternation of AI researchers, reinforcement learning had flopped when applied to trading.
The collaborators: In 2016 RBC had established Borealis AI, an institute dedicated to state-of-the-art research in machine learning and artificial intelligence. Given their mutual interests, it didn’t take long for Burhani and his colleagues at RBC Capital Markets to start talking with the world-class team at Borealis AI about how they could combine forces and improve trading for the benefit of RBC clients. And in short order—just two or three meetings—they’d agreed on how they were going to tackle the problem.
The decision: “In one step, we went from a rules based system to using reinforcement learning,” says Dr. Foteini Agrafioti, a co-founder of Borealis AI. “We could have tried other machine learning algorithms that are more established—reinforcement learning was brand-new technology that has only recently proven to be successful outside the lab,” she says. “But the Capital Markets team knew that RBC had the right controls in place to attempt something like this. And they don’t really entertain ideas that only have mediocre potential.” The team ultimately leveraged Deep Reinforcement Learning, which uses Reinforcement Learning principles plus a neural network to handle highly complex data relationships to develop a solution to address client needs.
The client connection: For RBC, investing in AI technology has always had one main purpose: helping traders help clients. “AI represents an exciting opportunity to continue to provide unique and differentiated insights to our clients” says Lea Hutton, Managing Director, Equity Trading. “The focus of everything we do has always been our clients, and our work in AI is no exception,” she says.
The result: Together, the teams at RBC Capital Markets and Borealis AI have been able to apply reinforcement learning to a real-world application with the launch of Aiden, an AI-based electronic trading platform. “Trading is such a great example of the kind of problem reinforcement learning is designed for,” says Professor Greg Mori, Senior Research Director at Borealis AI. Learning new trading strategies in dynamic market conditions is very different from playing board games. Successfully applying this science to stock trading execution is a major achievement.”
