The Wolf Hunters of Wall Street

From left: Rob Park, Brad Katsuyama and Ronan Ryan. Credit Stefan Ruiz for The New York Times

Technology had collided with Wall Street in a peculiar way. It had been used to increase efficiency. But it had also been used to introduce a peculiar sort of market inefficiency. Taking advantage of loopholes in some well-meaning regulation introduced in the mid-2000s, some large amount of what Wall Street had been doing with technology was simply so someone inside the financial markets would know something that the outside world did not. The same system that once gave us subprime-mortgage collateralized debt obligations no investor could possibly truly understand now gave us stock-market trades involving fractions of a penny that occurred at unsafe speeds using order types that no investor could possibly truly understand. That is why Brad Katsuyama’s desire to explain things so that others would understand was so seditious. He attacked the newly automated financial system at its core, where the money was made from its incomprehensibility.

Before the collapse of the U.S. financial system in 2008, Brad Katsuyama could tell himself that he bore no responsibility for that system. He worked for the Royal Bank of Canada, for a start. RBC might have been the fifth-biggest bank in North America, by some measures, but it was on nobody’s mental map of Wall Street. It was stable and relatively virtuous and soon to be known for having resisted the temptation to make bad subprime loans to Americans or peddle them to ignorant investors. But its management didn’t understand just what an afterthought the bank was — on the rare occasions American financiers thought about it at all. Katsuyama’s bosses sent him to New York from Toronto in 2002, when he was 23, as part of a “big push” for the bank to become a player on Wall Street. The sad truth was that hardly anyone noticed it. “The people in Canada are always saying, ‘We’re paying too much for people in the United States,’ ” Katsuyama says. “What they don’t realize is that the reason you have to pay them too much is that no one wants to work for RBC. RBC is a nobody.”

Before arriving there as part of the big push, Katsuyama had never laid eyes on Wall Street or New York City. It was his first immersive course in the American way of life, and he was instantly struck by how different it was from the Canadian version. “Everything was to excess,” he says. “I met more offensive people in a year than I had in my entire life. People lived beyond their means, and the way they did it was by going into debt. That’s what shocked me the most. Debt was a foreign concept in Canada. Debt was evil.”

For his first few years on Wall Street, Katsuyama traded U.S. energy stocks and then tech stocks. Eventually he was promoted to run one of RBC’s equity-trading groups, consisting of 20 or so traders. The RBC trading floor had a no-jerk rule (though the staff had a more colorful term for it): If someone came in the door looking for a job and sounding like a typical Wall Street jerk, he wouldn’t be hired, no matter how much money he said he could make the firm. There was even an expression used to describe the culture: “RBC nice.” Although Katsuyama found the expression embarrassingly Canadian, he, too, was RBC nice. The best way to manage people, he thought, was to persuade them that you were good for their careers. He further believed that the only way to get people to believe that you were good for their careers was actually to be good for their careers.

His troubles began at the end of 2006, after RBC paid $100 million for a U.S. electronic-trading firm called Carlin Financial. In what appeared to Katsuyama to be undue haste, his bosses back in Canada bought Carlin without knowing much about the company or even electronic trading. Now they would receive a crash course. Katsuyama found himself working side by side with a group of American traders who could not have been less suited to RBC’s culture. The first day after the merger, Katsuyama got a call from a worried female employee, who whispered, “There is a guy in here with suspenders walking around with a baseball bat in his hands.” That turned out to be Carlin’s chief executive, Jeremy Frommer, who was, whatever else he was, not RBC nice. Returning to his alma mater, the University at Albany, years later to speak about the secret of his success, Frommer told a group of business students: “It’s not just enough to fly in first class; I have to know my friends are flying in coach.”

Installed in Carlin’s offices, RBC’s people in New York were soon gathered to hear a state-of-the-financial-markets address given by Frommer. He stood in front of a flat-panel computer monitor that hung on his wall. “He gets up and says the markets are now all about speed,” Katsuyama says. “And then he says, ‘I’m going to show you how fast our system is.’ He had this guy next to him with a computer keyboard. He said to him, ‘Enter an order!’ And the guy hit Enter. And the order appeared on the screen so everyone could see it. And Frommer goes: ‘See! See how fast that was!!!’ ” All the guy did was type the name of a stock on a keyboard, and the name was displayed on the screen, the way a letter, once typed, appears on a computer screen. “Then he goes, ‘Do it again!’ And the guy hits the Enter button on the keyboard again. And everyone nods. It was 5 in the afternoon. The market wasn’t open; nothing was happening. But he was like, ‘Oh, my God, it’s happening in real time!’ ”

Katsuyama couldn’t believe it. He thought: The guy who just sold us our new electronic-trading platform either does not know that his display of technical virtuosity is absurd or, worse, he thinks we don’t know.

As it happened, at almost exactly the moment Carlin Financial entered Brad Katsuyama’s life, the U.S. stock market began to behave oddly. Before RBC acquired this supposed state-of-the-art electronic-trading firm, Katsuyama’s computers worked as he expected them to. Suddenly they didn’t. It used to be that when his trading screens showed 10,000 shares of Intel offered at $22 a share, it meant that he could buy 10,000 shares of Intel for $22 a share. He had only to push a button. By the spring of 2007, however, when he pushed the button to complete a trade, the offers would vanish. In his seven years as a trader, he had always been able to look at the screens on his desk and see the stock market. Now the market as it appeared on his screens was an illusion.

This made it impossible for Katsuyama to do his job properly. His main role as a trader was to play the middleman between investors who wanted to buy and sell big amounts of stock and the public markets, where the volumes were smaller. Say some investor wanted to sell a block of three million Intel shares, but the markets showed demand for only one million shares: Katsuyama would buy the entire block from the investor, sell off a million shares instantly and then work artfully over the next few hours to unload the other two million. If he didn’t know the actual demand in the markets, he couldn’t price the larger block. He had been supplying liquidity to the market; now whatever was happening on his screens was reducing his willingness to do that.

By June 2007 the problem had grown too big to ignore. At that point, he did what most people do when they don’t understand why their computers aren’t working the way they’re supposed to: He called tech support. Like tech-support personnel everywhere, their first assumption was that Katsuyama didn’t know what he was doing. " ‘User error’ was the thing they’d throw at you,” he says. “They just thought of us traders as a bunch of dumb jocks.”


A facility in Secaucus, N.J., is home to some IEX servers. Credit Stefan Ruiz for The New York Times

Finally he complained so loudly that they sent the developers, the guys who came to RBC in the Carlin acquisition. “They told me it was because I was in New York and the markets were in New Jersey and my market data was slow,” Katsuyama says. “Then they said that it was all caused by the fact that there are thousands of people trading in the market. They’d say: ‘You aren’t the only one trying to do what you’re trying to do. There’s other events. There’s news.’ ”

If that was the case, he asked them, why did the market in any given stock dry up only when he was trying to trade in it? To make his point, he asked the developers to stand behind him and watch while he traded. “I’d say: ‘Watch closely. I am about to buy 100,000 shares of AMD. I am willing to pay $15 a share. There are currently 100,000 shares of AMD being offered at $15 a share — 10,000 on BATS, 35,000 on the New York Stock Exchange, 30,000 on Nasdaq and 25,000 on Direct Edge.’ You could see it all on the screens. We’d all sit there and stare at the screen, and I’d have my finger over the Enter button. I’d count out loud to five. . . .

“ ‘One. . . .

“ ‘Two. . . . See, nothing’s happened.

“ ‘Three. . . . Offers are still there at 15. . . .

“ ‘Four. . . . Still no movement. . . .

“ ‘Five.’ Then I’d hit the Enter button, and — boom! — all hell would break loose. The offerings would all disappear, and the stock would pop higher.”

At which point he turned to the developers behind him and said: “You see, I’m the event. I am the news.”


While You Were Blinking
High-frequency-trading activity is not constant; it occurs in microbursts. The line at the bottom of this graphic is the stock-market activity involving General Electric shares over 100 milliseconds (one-tenth of a second) at 12:44 p.m. on Dec. 19, 2013. The gray box magnifies a five-millisecond window, during which GE experienced heavy bid and offer activity and a total of 44 trades. Credit Graphic: CLEVERºFRANKE. Data source: IEX.

To that, they had no response. Katsuyama suspected the culprit was Carlin’s setup. “As the market problem got worse,” he says, “I started to just assume my real problem was with how bad their technology was.”

But as he talked to Wall Street investors, he came to realize that they were dealing with the same problem. He had a good friend who traded stocks at a big-time hedge fund in Stamford, Conn., called SAC Capital, which was famous (and soon to be infamous) for being one step ahead of the U.S. stock market. If anyone was going to know something about the market that Katsuyama didn’t know, he figured, it would be someone there. One spring morning, he took the train up to Stamford and spent the day watching his friend trade. Right away he saw that, even though his friend was using software supplied to him by Goldman Sachs and Morgan Stanley and the other big firms, he was experiencing exactly the same problem as RBC: He would hit a button to buy or sell a stock, and the market would move away from him. “When I see this guy trading, and he was getting screwed — I now see that it isn’t just me. My frustration is the market’s frustration. And I was like, ‘Whoa, this is serious.’ ”

People always assumed that because he was Asian, Brad Katsuyama must be a computer wizard. In reality, he couldn’t (or wouldn’t) even program his own DVR. What he had was an ability to distinguish between computer people who actually knew what they were talking about and those who didn’t. So he wasn’t exactly shocked when RBC finally gave up looking for someone to run its mess of an electronic-trading operation and asked him if he would take over and try to fix it. He shocked his friends and colleagues, however, when he agreed to do it, because A) he had a safe and cushy $1.5-million-a-year job running the human traders, and B) RBC had nothing to add to electronic trading. The market was cluttered; big investors had use for only so many trading algorithms sold by brokers; and Goldman Sachs and Morgan Stanley and Credit Suisse had long since overrun that space.

So Katsuyama was in charge of a business called electronic trading — with only Carlin’s inferior software to sell. What he had, instead, was a fast-growing pile of unanswered questions. Between the public stock exchanges and the dark pools — private exchanges created by banks and brokers that did not have to report in real time what trading activities took place within them — why were there now nearly 60 different places, most of them in New Jersey, where you could buy any listed stock? Why did one public exchange pay you to do something — sell shares, say — when another exchange charged you to do the same exact thing? Why was the market displayed on Wall Street trading screens an illusion?

He hired Rob Park, a gifted technologist, to explain to him what actually happened inside all these new Wall Street black boxes, and together they set out to assemble a team to investigate the U.S. stock market. Once he had a team in place, Katsuyama persuaded his superiors at RBC to conduct what amounted to a series of experiments. For the next several months, he and his people would trade stocks not to make money but to test theories. RBC agreed to let his team lose up to $10,000 a day to figure out why the market in any given stock vanished the moment RBC tried to trade in it. Katsuyama asked Park to come up with some theories.

They started with the public markets — 13 stock exchanges scattered over four different sites run by the New York Stock Exchange, Nasdaq, BATS and Direct Edge. Park’s first theory was that the exchanges weren’t simply bundling all the orders at a given price but arranging them in some kind of sequence. You and I might each submit an order to buy 1,000 shares of Intel at $30 a share, but you might somehow obtain the right to cancel your order if my order was filled. “We started getting the idea that people were canceling orders,” Park says. “That they were just phantom orders.”


This box kept at the facility in Secaucus, N.J., contains a 38-mile coil of fiber-optic cable that creates a slight delay in the processing of orders, which levels the playing field among traders. Credit Stefan Ruiz for The New York Times

Katsuyama tried sending orders to a single exchange, fairly certain that this would prove that some, or maybe even all, of the exchanges were allowing these phantom orders. But no: To his surprise, an order sent to a single exchange enabled him to buy everything on offer. The market as it appeared on his screens was, once again, the market. “I thought, [expletive], there goes that theory,” Katsuyama says. “And that’s our only theory.”

It made no sense: Why would the market on the screens be real if you sent your order to only one exchange but prove illusory when you sent your order to all the exchanges at once? The team began to send orders into various combinations of exchanges. First the New York Stock Exchange and Nasdaq. Then N.Y.S.E. and Nasdaq and BATS. Then N.Y.S.E., Nasdaq BX, Nasdaq and BATS. And so on. What came back was a further mystery. As they increased the number of exchanges, the percentage of the order that was filled decreased; the more places they tried to buy stock from, the less stock they were actually able to buy. “There was one exception,” Katsuyama says. “No matter how many exchanges we sent an order to, we always got 100 percent of what was offered on BATS.” Park had no explanation, he says. “I just thought, BATS is a great exchange!”

One morning, while taking a shower, Rob Park came up with another theory. He was picturing a bar chart he had seen that showed the time it took orders to travel from Brad Katsuyama’s trading desk in the World Financial Center to the various exchanges.

The increments of time involved were absurdly small: In theory, the fastest travel time, from Katsuyama’s desk in Manhattan to the BATS exchange in Weehawken, N.J., was about two milliseconds, and the slowest, from Katsuyama’s desk to the Nasdaq exchange in Carteret, N.J., was around four milliseconds. In practice, the times could vary much more than that, depending on network traffic, static and glitches in the equipment between any two points. It takes 100 milliseconds to blink quickly — it was hard to believe that a fraction of a blink of an eye could have any real market consequences. Allen Zhang, whom Katsuyama and Park viewed as their most talented programmer, wrote a program that built delays into the orders Katsuyama sent to exchanges that were faster to get to, so that they arrived at exactly the same time as they did at the exchanges that were slower to get to. “It was counterintuitive,” Park says, “because everyone was telling us it was all about faster. We had to go faster, and we were slowing it down.” One morning they sat down to test the program. Ordinarily when you hit the button to buy but failed to get the stock, the screens lit up red; when you got only some of the stock you were after, the screens lit up brown; and when you got everything you asked for, the screens lit up green.


Print this post

Do you like this post?

Showing 3 reactions

commented 2014-04-01 22:04:56 -0400 · Flag
Tom writes:

So my question is ? How does this make you feel ? and the Pathetic answer, What the Hell does any of this have to do with Real Economic Activity ? Answer: It doesn’t, IT IS ALL ABOUT MAKING MONEY
commented 2014-04-01 22:04:10 -0400 · Flag
Rich writes:

The last two sentences describe the whole corrupt system:

“That is why Brad Katsuyama’s desire to explain things so that others would understand was so seditious. He attacked the newly automated financial system at its core, where the money was made from its incomprehensibility.”
commented 2014-04-01 22:02:48 -0400 · Flag
Chris writes:

Anatomy of a fed-encouraged techno-scam.