Down the A.I. Rabbit Hole: Hedge Funds, Wall St, & Machine Learning

Published on
January 6th, 2020
47 minutes

Down the A.I. Rabbit Hole: Hedge Funds, Wall St, & Machine Learning

The Interview ·
Featuring Vasant Dhar and Hari Krishnan

Published on: January 6th, 2020 • Duration: 47 minutes

Artificial intelligence and machine learning are poised to "eat the world" — but how, exactly, do they work together on Wall Street to make money for traders? NYU business professor, data scientist, and hedge fund manager Vasant Dhar sits down with fellow hedge fund manager and mathematician Hari Krishnan to take a deep dive into the cutting edge where technology meets markets. Filmed on December 9, 2019 in New York.



  • PP
    Paul P.
    27 October 2020 @ 03:25
    Hari, This would be a great interview to revisit given the volatility in March. Curious how ML navigated that environment given the long only bias in ML.
  • TM
    Thomas M.
    14 March 2020 @ 02:52
    after years of trading from the screen i decided to go back to school to lean higher mathematics. but when i finally arrived at probability theory i ended up more confused then when i started. Probability is a long run limit the number should in theory converge to that estimated number but how and when is an entirely different ball game. not to mention if your data set is small just toss juju bones.
  • RM
    Ryan M.
    28 January 2020 @ 21:23
    This was really good
    • RM
      Ryan M.
      28 January 2020 @ 21:30
      "Unless you feel the pain and the pleasure from actually trading, it's all hypothetical."
  • MN
    Michael N.
    19 January 2020 @ 15:36
    incredible interview, thank you to both parties!
  • DP
    David P.
    12 January 2020 @ 19:51
    Hari, How much would it cost for me to buy up all the copies of SLD and you to promise to never publish again? P. S. Nice interview. Going to be wild when the live data generating process switches to something not in the training data for a lot of these systems.
    • HK
      Hari K. | Contributor
      13 January 2020 @ 16:56
      Haha, 1 more finance book from me, then I'm out for the foreseeable future! Agreed on your last point.
  • RB
    Richard B.
    10 January 2020 @ 03:36
    I am certain an excellent discussion; however, this seems to have little to do with finance or investing. Thanks.
    • DP
      David P.
      12 January 2020 @ 19:36
      You are playing a game. Don't you want to know how your opponent thinks about the game?
  • MB
    Max B.
    11 January 2020 @ 15:37
    Can anyone help point me to a good resource for getting forward estimates (forward p/e, forward EPS, forward revenue etc.)? Not my normal post but any help would be much appreciated....
  • AR
    Anik R.
    9 January 2020 @ 23:16
    I think this concept of alpha decay is pretty unique to trading which is another reason why applying ML is even tougher in this field. Imagine finding an awesome technique to unblur blurry images in 2019, only to find that it doesn't work for blurry pictures taken in 2020 haha.
  • NA
    Nicolas A.
    9 January 2020 @ 04:57
    Thank you very much, highly interesting! One question, how come at one point Vasant says if you only focus on price data you don't really care about the reason behind the moves and in regards to Fukushima and other acts of God he advocates turning the machine off because the model cannot deal with such events? Presumably such events are included in the data set if its large enough (911 comes to mind)? Actually they might be a great time to make outsized returns (or losses). I'm just curious if anyone has any views, I don't have an answer. But seems to me the true believers always say don't turn off the machine and if I recall Renaissance's experience (as per the recent book) during the quant quake correctly Simons advocated to switch off losing models while others wanted to let the machine decide, in the end they turned it off and would have been better off keeping them on (albeit would have been more volatile...). Anyway, thanks again and interested in any views/experiences on this.
    • JH
      Jannis H.
      9 January 2020 @ 12:43
      So for one the price data model could carry the assumption that all extrinsic factors in the world are already contained in the price data in some way. Such that there would be conditional independence of your prediction to these external events given price information. Another problem I see with it is how to incorporate it as a feature for learning. If you are modelling it as a binary variable that simply means 'catastrophy' (which you would also have to quantifiably define beforehand), you would end up with lots and lots of training data in which that variable is turned off and a few single points where its on. In machine learning you usually want your model to avoid learning overly specified hypotheses as that will very often lead to a problem called overfitting. That way your model would probably either simply ignore it because there is such little evidence for it, or would also pick up lots and lots of other useless random patterns that deteriorate your results.
  • RO
    Riskis O.
    8 January 2020 @ 23:48
    Dear Hari K, thanks. Great interview! I found it is really insightful from a quant prospective. One thing I couldn't understand in the interview is that he said there are two things to look at When he decide whether to trust a back tested result, one being checking performance variance, which i believe is quite standard. Then he mentioned that he want to see impact on "human decisions even tho performance does not change". What human decision is he referring to in this caseL? Thanks.
  • DH
    Daniel H.
    7 January 2020 @ 06:42
    I get a funny feeling that his AI algorithms, since he said they cannot handle the fat tail anomalies, really boil down to being short volatility so the algorithm can pick up a "signal."
    • HK
      Hari K. | Contributor
      7 January 2020 @ 14:39
      Excellent comment, Daniel H. My understanding is that ML has a hard time dealing with "minority" data sets, price patterns that are relatively rare. This is only indirectly related to short vol, as high vol regimes are rarer than quiet ones. However, a good system can manage regime shift risk in ways that are superior to what a trend follower (for example) might do. Trend followers are thought to be long vol, but if you take out the structural fixed income long, they are even more susceptible to vol spikes than a good ML system will be.
  • SP
    Steve P.
    7 January 2020 @ 03:21
    Thing about AI research in general - it’s only as valuable as the people behind it. If the researcher can’t come to an insightful conclusion, that’s wasted $. Coupon day is a great example - those house wives could of answered that question for free.
  • HK
    Hari K. | Contributor
    6 January 2020 @ 15:36
    Hi, although I played a fairly minor role relative to Vasant, I will try and periodically respond to any questions you might have. Best, Hari
    • SG
      Sven G.
      6 January 2020 @ 16:46
      Hi Hari, thanks for a great interview. One thought to share, you asked Vasant about datasets from X period where market variables were different and how they might be applicable to a later time with different variables at play. This is a very difficult nut to crack but one thing that is exactly the same. Humans react the same way. You can see that in any chart from any time for any financial instrument. e.g. mania Tech stocks vs gold vs bitcoin vs X commodity. The charts looks virtually identical, not because the underlying instrument is the same but because people are people are people and tend to react in the same way over and over again. Thanks again!
    • SH
      Scott H.
      6 January 2020 @ 19:45
      That's what amazes Sven, algo's percentage volume is now so large I am surprised that market structure has not changed even more. Even short term (discretionary) strategies that I have traded since 2001 are still profitable (of course with a few tweaks but the basic premise is the same).
    • HK
      Hari K. | Contributor
      7 January 2020 @ 00:26
      Some excellent points here. I think some things have changed qualitatively, e.g. mean reversion in implied vol is much faster than before, but perhaps this is a function of low government yields as well as faster risk sampling and ML. Over longer horizons, you are absolutely correct that the cycles of sentiment dominate. The recent "risk on" phase may have been stretched, but our impulses are largely the same. Ty!
  • JB
    John B.
    6 January 2020 @ 19:38
    Ill make an early call. This will be in the Top 10 videos on RV (non-news/current event oriented) for the year.
    • HK
      Hari K. | Contributor
      6 January 2020 @ 23:30
      Many thanks!
  • PP
    Paul P.
    6 January 2020 @ 21:16
    This is beyond excellent. The proverbial must watch. Hari Krishnan and Vasant Dhar in a very natural conversation. I was a huge fan of Hari as an interviewee. I like him equally as well as an interviewer. I'm blown wway by both these men's intelligence. Look forward to more from both. Can't sing enough praises.
    • HK
      Hari K. | Contributor
      6 January 2020 @ 23:30
      Thank you!
  • DC
    Dan C.
    6 January 2020 @ 22:56
    I'd love to see a week long deep dive into algo trading, market structure, etc. RV has had some great interviews but this topic seems worthy of more development. And don't be afraid to get deeper into the tech details... not everyone will get it but that's ok.
    • HK
      Hari K. | Contributor
      6 January 2020 @ 23:30
      Agreed, IMHO. The issue is can the material be practically useful while appealing to a wide enough audience.
  • FG
    Flavio G.
    6 January 2020 @ 22:26
    I have used ML (with good success) for other kinds of things but never for investment. The arguments made were very reasonable and I enjoyed how Hari steered the conversation to the tricky areas of ML. However, the statement about ML not being useful for time-horizons beyond daily trade really surprised me, especially the given reason: not enough data. Hard to understand how this could be true. I would've liked to dive deeper into this specific point. The problem with ML for alpha generation is that people who have an edge are never going to teach how they do it. I am guessing the setup is expensive too, mainly the data feeds and even more the esoteric ones.
    • HK
      Hari K. | Contributor
      6 January 2020 @ 23:28
      You have made a good point, Flavio. I think Vasant was trying to say that ML requires *lots* of data in a stationary environment. Data for training, data for testing and an assumption that the rules of the game don't change too much, conditional on a variable such as volatility. That makes it hard to use data that a. doesn't arrive frequently and b. is subject to revision. Ty!
  • DS
    David S.
    6 January 2020 @ 18:40
    Solid discussion. All the ETF/index automatic purchases would seem to provide a large number of trades without any nuance to analyze. In addition, the percentage of off-market trades that are excluded from the database should limit what can be identified as reasoned stock picking. This leaves a subset of the market to be analyzed and back tested. AI may tell us more about the open market makers decisions by excluding ETF/index purchases, but still it is less predictable for the market as a whole. Keep up the good work. DLS
  • KL
    Kim L.
    6 January 2020 @ 15:50
    Great video! Would like to see more of these - ones where the nuances of using AI are discussed.
  • CB
    C B.
    6 January 2020 @ 15:25
    A complex subject exceptionally well explained. Thank you.
  • BA
    Bruce A.
    6 January 2020 @ 09:48
    I don't think machine learning would help with my golf swing. Way too much noise and not enough signal.
  • WB
    William B.
    6 January 2020 @ 06:08
    Everyone who watches this will likely take away something different, but that’s what helps make a great video!
  • AA
    A A.
    6 January 2020 @ 05:55
    Great info nicely done.