From my observation, with technology, there will always be a lag between those who understand the fundamentals and those who can explain it and it's value in laymans terms. There's a translation layer that takes time to build. Those close to the machine are optimizing for the machine not the human and those optimizing for the human are not optimizing for the machine. Someone has to be properly incentivized to build that bridge to realize that laymans bridge.

Great point 👆🏻

IMO.. from what I’ve read and understand I don’t think there will be enough data for an adequate training set that will allow true AI to dominate the market for a long time. Those sets won’t have enough data to model regime changes like the destruction of an asset class that we are currently witnessing.

Thanks for updating the IOS app!!

Lazard MANAGEMENT or LABS?

Aaruran, totally agree. Having done this stuff for a bit of time i have found the lack of suitability of GLM models to most contexts I apply them to. Too many breaches of distribution, stationarity, and heteroscedasticity. Not to mention they really do not appear to generalize very well. When the data is then adjusted for those components, the loss of information is often too much to proceed with. ML, and especially classification (GBMs, random forest, even dimensionalily reduction algos used with support vector machines for underspecified datasets) provide a useful tool for initial analysis of a problem. This helps focus in on important features for the next analytical steps in the process. This might explain his bias here. However each problem is quite specific and has differing needs and model suitability, so solutions will differ based on context.

OLS regression models require a host of independent variables else they’re susceptible to the 6-point test and likely will fail atleast 1. Regressions work well for fixed variables, characteristics that are less abstract though the relationship between variables may seem abstract. Also, running a regression is great for deep statistics like R, goodness of fit and T and Z statistics. Far as markets go, nearly everything correlated to SP500 lol

I'm building models for a while and in my experience you'll be able to find a lot of it for free, with caveats: (all this assuming you're doing this programmatically) 1, If you need the basics, Quantopian has them (market data and basic US economic data) in a very easy to access form, surrounded by portfolio management libraries. If your focus is backtesting, you can start there - then eventually port the whole thing to your own setup as you run out of the free processing power there. 2, Most of the US government agencies offer some sort of free (usually Python) API but they all vary greatly and will benefit from another layer built on top to abstract from the indiosyncracies of various libraries gathered from Github. 3, If you can afford it, Quandl did that for you already but I feel it could be a slippery slope of expensive subscriptions :) One good tip: they offer TradingEconomics as a package for less (or more access for the same price) as the actual TE website - if I remember correctly around $700/year? 4, If you're struggling to collect EM data (or anything outside the US really), I can recommend the TE subscription, has most of what you relaly need covered. Otherwise you'll probably keep hunting and writing website scrapers and interfaces forever if you're on your own while really you want to be focusing on your model. Personally: I use Interactive Brokers' free API for market data, a bunch of hand rolled libraries (based on FRED, BLS...etc) for most but I wrote those libraries in mind with Quandl so if I need to, I can swap them out easily once I feel the need to have broader data.

My broker ameritrade has an api that gives free price data and current stats on stocks. I have been scraping yahoo to get fundamental data, but they keep changing their website and so I keep having to redo my program which is a pain. All the SEC data has XBRL now so it would be cool to get setup to scrape directly from them but it's more complicated. I just use excel and an access database.

I thought he meant macro data, that's much more difficult. For price and fundamentals there are even very simple to use Excel plug-ins with no programming need.

Kos, depends on the strategies you run. If you are playing with tick or intraday data, alternative data, or exotic markets, i am yet to find solutions that achieve both timeliness and affordability. APIs such as alphavantage or finnhub can be helpful if you're after intraday (inexpensive and usually reliable, but i have experienced inaccess due to what I suspect was high traffic). Alternative data is available on quandl and likely in other places that I havent yet looked (I havent yet use any alternative data). FRED is also great but you'll sometimes experience delays in availability of numbers. In terms of integration, as mentioned in this thread, quantopian has a brilliant backtesting platform. There is some stock and futures data, and fundamental data and other pieces that are quite helpful. Personally, I have built an environment on my own machine with custom metrics and model selection procedures used prior to backtesting to address potential lookahead bias and overfitting. The quantopian zipline python package is open source on github. You can run your backtests using that if you wish to provide your own data. I have found that this sidesteps any memory issues that may arise from slow models on the platform.