Release 2.8.0

2022-09-22

Release 2.8.0 introduces Apple Silicon support, numerous Zipline enhancements, Sharadar enhancements, margin requirements data, and other enhancements.

Highlights

• Apple Silicon support: Mac users can now run QuantRocket natively on M1 Macs. Follow the standard installation steps, and Docker will automatically install the appropriate QuantRocket containers based on your system architecture. (Interactive Brokers customers: please note that IB Gateway runs under emulation on M1 Macs, not natively, as explained more fully in the installation tutorial.)
• Parameter scans for Zipline: this previously Moonshot-only feature for optimizing strategies is now available for Zipline. See the usage guide.
• Multi-worker parameter scans: both Moonshot and Zipline now offer built-in concurrency for parameter scans, allowing you to cut down scan time by testing multiple parameters simultaneously. See the Moonshot or Zipline usage guide.
• Dry runs for Zipline live trading: optionally write your strategy's orders to file instead of sending orders to the blotter for live execution. Dry runs are useful for troubleshooting, validation, and executing orders manually outside QuantRocket. See the usage guide.
• Query previous fiscal periods when using get_sharadar_fundamentals_reindexed_like or when using Sharadar fundamentals in Pipeline. This makes it easy to compare current and previous periods and calculate changes in fundamental metrics over time. See the fundamentals section or Zipline section of the usage guide.
• Native Sharadar bundle: ingest a prefabricated Sharadar bundle (like the US stock bundle) without having to collect a history database and ingest from there. The new Sharadar bundle supports stocks and ETFs in a single bundle (unlike the Sharadar history database). See the usage guide.
• Modify Zipline strategy parameters on-the-fly when running backtests. This Moonshot-inspired feature, which allows you to test different parameters without editing your algo files, is now available for Zipline. See the usage guide.
• Set the bundle and data frequency for your Zipline strategy directly in the algorithm file instead of having to specify them each time you backtest or trade. See the usage guide.
• Margin requirements data from Interactive Brokers, covering stocks with special margin requirements, with history back to 2018. See the usage guide.
• Support for sending "what-if" orders to Interactive Brokers. What-if orders allow you to check the margin impact and estimated commission of an order without executing the order (similar to the Order Preview window in Trader Workstation). See the usage guide.

Documentation

• A new section of the Zipline usage guide outlines a design pattern for re-using code when running multiple variants of a strategy (such as one variant for large cap stocks and another for mid cap stocks). No more copying and pasting the same code into multiple files. (Note for users who don't plan to update to version 2.8.0 immediately: although this design pattern is based on standard characteristics of Python rather than being a feature built into QuantRocket, it should not be used with earlier versions of QuantRocket, as QuantRocket's method of periodically reloading Zipline workers has been tweaked in version 2.8.0 to ensure the design pattern works as intended.)
• New example code in the Zipline usage guide shows how to plot the performance of a portfolio of Zipline strategies using Moonchart.

API Changes

• This release includes updated versions of Python, pandas, numpy, and other libraries. Although the version number changes are relatively minimal, package updates always have the potential to cause breakages in your code if you interact with those libraries directly. Users should carefully test their algorithms after updating. The version changes of the major packages are shown below:
  • Python 3.8 -> Python 3.9
  • pandas 1.2 -> pandas 1.3
  • numpy 1.19 -> numpy 1.21
• The previous method name for constructing a moonchart parameter scan tear sheet, ParamscanTearsheet.from_moonshot_csv(...), has been renamed to ParamscanTearsheet.from_csv(...), since parameter scan CSVs can now come from Moonshot or Zipline. The previous method name is still available but will raise a deprecation warning.

quantrocket/blotter:2.8.0

• support for sending "what-if" orders to Interactive Brokers. What-if orders allow you to check the margin impact and estimated commission of an order without executing the order (similar to the Order Preview window in Trader Workstation). See the usage guide.
• support for filtering order status queries by datetime (previously when a datetime was passed, the time component was ignored and results were filtered by date)

quantrocket-client:2.8.0

• query previous fiscal periods using the new period_offset parameter for get_sharadar_fundamentals_reindexed_like. This makes it easy to compare current and previous periods and calculate changes in fundamental metrics over time. See the usage guide.

quantrocket/fundamental:2.8.0

• margin requirements data from Interactive Brokers, covering stocks with special margin requirements, with history back to 2018. See the usage guide.

quantrocket/jupyter:2.8.0

• update to JupyterLab 3.4.3

quantrocket/moonshot:2.8.0

• Concurrency in parameter scans: speed up parameter scans by running them with multiple workers. See the usage guide.

quantrocket/postgres:2.8.0

• update to TimescaleDB 2.8.0

quantrocket/zipline:2.8.0

• parameter scans: optimize your strategy by testing and comparing different parameter values. Built-in concurrency allows you to cut down scan time by testing multiple parameters simultaneously. See the usage guide.
• dry runs for Zipline live trading: write your strategy's orders to file instead of sending orders to the blotter for live execution. Dry runs are useful for troubleshooting, validation, and executing orders manually outside QuantRocket. See the usage guide.
• query previous fiscal periods using the period_offset parameter for Sharadar fundamentals in Pipeline. This makes it easy to compare current and previous periods and calculate changes in fundamental metrics over time. See the usage guide.
• native Sharadar bundle: ingest a prefabricated Sharadar bundle (like the US stock bundle) without having to collect a history database and ingest from there. The new Sharadar bundle supports stocks and ETFs in a single bundle (unlike the Sharadar history database). See the usage guide.
• modify strategy parameters on-the-fly when running backtests. See the usage guide.
• support for setting the bundle and data frequency for your strategy directly in the algorithm file. See the usage guide.
• support for creating a bundle from multiple history databases or real-time aggregate databases, as long as all source databases have the same bar size and fields. See the Usage Guide.
• run custom code before a bundle loads, such as registering a custom calendar. See the usage guide.
• fix an issue where context variables set inside scheduled functions weren't persisted to file in live trading. See related forum post.

Release 2.7.1

2021-12-24

This is a patch release that contains a mandatory update of IB Gateway. Due to the Log4j critical security vulnerability, Interactive Brokers announced on December 21, 2021 that they will discontinue operation of earlier versions of IB Gateway on December 29, 2021. This means that earlier versions of QuantRocket will no longer be able to connect to Interactive Brokers after that time. Therefore, all QuantRocket users wishing to connect to their Interactive Brokers account must update to the latest version.

quantrocket/ibg:2.7.1

• update to IB Gateway 981.3g

quantrocket/jupyter:2.7.1

• fix an issue preventing the use of quantrocket realtime stream from the JupyterLab terminal

Release 2.7.0

2021-12-16

Release 2.7.0 provides compatibility with Docker Compose v2 and includes other enhancements, improvements, and bug fixes.

Highlights

• compatibility with Docker Compose v2, which ships with recent versions of Docker Desktop, and which uses hyphens instead of underscores in container names (e.g. quantrocket-ibg1-1). Docker Compose v2 caused problems running the ibg service in earlier versions of QuantRocket. Both Docker Compose v1 and v2 are supported with this release.
• The blotter now automatically incorporate dividends into PNL calculations for US stocks. See the usage guide.
• The blotter provides a new API for applying splits to open positions. See the usage guide.
• Moonshot parameter scan results are now logged to flightlog, to provide feedback sooner during long-running parameter scans. See the usage guide for example output.

API Changes

• The endpoint for collecting Reuters estimates from the IBKR API (quantrocket.fundamental.collect_reuters_estimates/quantrocket fundamental collect-reuters-estimates) has been removed. The IBKR API stopped returning new estimates data in 2020 and stopped returning any estimates data earlier in 2021. The endpoints for querying estimates from your local database (quantrocket.fundamental.download_reuters_estimates/quantrocket fundamental reuters-estimates and quantrocket.fundamental.get_reuters_estimates_reindexed_like) have been retained so that users who previously collected Reuters estimates can continue querying the data.
• When inserting intraday data into a custom database, the quantrocket.db functions insert_or_fail, insert_or_ignore, and insert_or_replace will now cast datetime columns to ISO 8601 format using the 'T' separator (YYYY-MM-DDTHH:MM:SS). Prior to this change, underlying pandas behavior caused a space separator to be used (YYYY-MM-DD HH:MM:SS). This caused some edge cases when querying a custom intraday database using start_date/end_date parameters, as the history service expects a 'T' separator to be used. Users with existing custom intraday databases are advised to convert the existing data to use the 'T' separator, so that all data is stored consistently. The following one-time query should be run (setting DB_CODE to the appropriate database code):

$ DB_CODE=custom-intraday
$ sqlite3 /var/lib/quantrocket/quantrocket.v2.history.$DB_CODE.sqlite 'UPDATE Price SET Date = REPLACE(Date, " ", "T")'

Docker

• compatibility with Docker Compose v2, which ships with recent versions of Docker Desktop, and which uses hyphens instead of underscores in container names (e.g. quantrocket-ibg1-1). Docker Compose v2 caused problems running the ibg service and caused problems with logging in earlier versions of QuantRocket. Both Docker Compose v1 and v2 are supported with this release.

quantrocket/blotter:2.7.0

• automatically incorporate dividends into PNL calculations for US stocks. See the usage guide.
• provide a new API for applying splits to open positions. See the usage guide.

quantrocket/db:2.7.0

• handle an issue where pushing databases to a S3 bucket located in a region other than us-east-1 sporadically failed
• cast datetime columns to ISO format with "T" separator before inserting into database in quantrocket.db functions insert_or_fail, insert_or_ignore, and insert_or_replace. See note above under API changes.

quantrocket/moonshot:2.7.0

• log results to flightlog during parameter scans, to provide feedback sooner during long-running parameter scans. See the usage guide for example output.

moonchart:2.7.0

• print a table of results at end of parameter scan tear sheet
• add optional start_date and end_date parameters to Tearsheet.from_moonshot_csv and Tearsheet.from_pnl_csv. See the API Reference.

quantrocket/postgres:2.7.0

• update to TimescaleDB 2.5.1

trading_calendars:2.7.0

• add Juneteenth holiday to XNYS calendar beginning in 2022

quantrocket/zipline:2.7.0

• fix an issue with minute bar ingestion from a history database when the data crosses over daylight savings time. See related forum post.
• fix an error where data.can_trade failed for assets listed on exchanges that don't have trading calendars. See related forum post.
• fix an occasional FileNotFoundError in US stock ingestion. See related forum post.
• treat IBKR "Inactive" order status as equivalent to "Cancelled". See related forum post.

Release 2.6.1

2021-09-08

This is a patch release that fixes an issue with IBKR historical data collection of intraday bars, and includes several other minor enhancements.

quantrocket/blotter:2.6.1

• force PNL recalculation after recording non-QuantRocket executions

quantrocket/history:2.6.1

• Fix an issue where data gaps occurred at request boundaries in IBKR intraday data collection. See related forum post.

quantrocket/jupyter:2.6.1

• update zipline to 2.6.1 to match zipline service

quantrocket/postgres:2.6.1

• update to TimescaleDB version 2.4.1

quantrocket/realtime:2.6.1

• handle the possibility of Polygon returning ticks for symbols that weren't requested. See related forum post.

quantrocket/zipline:2.6.1

• retry transient S3 sync errors when ingesting usstock
• in replay of executions in live trading, handle the possibility that the execution may reference an order id that doesn't actually exist (which can happen if the execution was triggered manually from TWS)
• increase timeout when polling realtime minute data from 4 to 10 seconds because sometimes Alpaca minute data doesn't start flowing right after the minute ends

Release 2.6.0

2021-06-21

Release 2.6.0 includes an updated version of Zipline that works with modern packages and runs in QuantRocket's standard environment, plus other enhancements and bug fixes.

Highlights

• Zipline modernization. The Zipline package, which was pinned to old versions of pandas, numpy, and other libraries and is no longer maintained by Quantopian, has been streamlined and updated to work with modern package versions. As a result, Zipline is now installed in the standard QuantRocket environment. Benefits include:
  1. In JupyterLab, you no longer need to use Zipline in a separate "Zipline environment" kernel (which has been removed) but can use the standard Python 3 kernel;
  2. You can execute Zipline pipelines in satellite scripts;
  3. You can more easily install custom packages in the zipline container (previously, doing so tended to break Zipline due to Zipline's dependency on old package versions).
• Better futures calendar support in Zipline, including expanded calendar choices and support for calendars with trading breaks. See additional notes below under the quantrocket/zipline and trading_calendars sections.
• Support for recording executions that happen outside of QuantRocket's knowledge. This can be used to keep the blotter aligned with your broker when one of your positions undergoes a tender offer, merger or acquisition, etc. See the usage guide.
• Update to JupyterLab 3.0, the most notable feature of which is a table of contents in the left area when you have a notebook open, allowing you to more easily navigate through long notebooks:

API Changes

• The updated versions of pandas, numpy, etc. that Zipline now uses have the potential to cause breakages in Zipline algorithms if you interact with those libraries directly. For example, in newer versions of pandas you cannot mix timezone-aware and timezone-naive datetimes, while in earlier versions of pandas you could. Users should carefully test their algorithms after updating. The version changes of the major packages are shown below:
  • Python 3.6 -> Python 3.8
  • pandas 0.22 -> pandas 1.2
  • numpy 1.13 -> numpy 1.19
• Several previously deprecated Zipline functions have been removed, including zipline.api.fetch_csv (use quantrocket.get_prices instead), zipline.api.set_do_not_order_list (use zipline.api.set_asset_restrictions instead), and zipline.api.symbol (use zipline.api.sid instead).
• When requesting multiple fields for multiple assets using data.history(), Zipline previously returned the data as a pandas Panel, support for which was removed starting in pandas 1.0. For this type of request, Zipline will now return a DataFrame in which the columns are a multiindex of (field, asset). No code changes should be required, however, as fields and assets can be accessed in the same way as before, by first accessing the desired field then the desired asset(s):

# used to return a Panel, now returns a multiindex DataFrame
prices = data.history(assets, ["close", "open"], 150, '1d')
aapl = algo.sid("FIBBG000B9XRY4")
aapl_closes = prices["close"][aapl]

• the quantrocket_trading_calendars package is now deprecated as all of its functionality has been merged into QuantRocket's fork of the trading_calendars package. Please update any imports of quantrocket_trading_calendars to import trading_calendars instead; no other code changes are required. Importing quantrocket_trading_calendars will still work but will print a deprecation warning.
• Zipline users with a minute bundle associated with the CMES/CME/GLOBEX trading calendar should drop and re-ingest the bundle. Reason: the trading hours for the CMES/CME/GLOBEX calendar have been changed (corrected) from a 24-hour session to a 23-hour session (5pm - 4pm CT). To ensure proper querying of the minute bundle, this calendar change necessitates dropping and re-ingesting the bundle. Please also see the expanded set of choices for CME calendars as described below in the trading_calendars section, as there may be a more optimal calendar choice for your bundle than the standard CMES/CME/GLOBEX calendar.
• With this release, QGrid, the Excel-like widget for filtering and sorting DataFrames in notebooks, has been removed from JupyterLab. QGrid is not compatible with the latest version of JupyterLab, and unlike other Quantopian-created packages that QuantRocket now maintains, we deem QGrid to be too minor in importance to warrant ongoing maintenance. Given the technical proficiency of most QuantRocket users, we believe few if any users are currently using or relying on QGrid. If you were using QGrid, please use standard pandas techniques for filtering and sorting your DataFrames instead.

Documentation

• add a section to the Zipline docs clarifying when functions run and how orders are handled in daily vs minute backtests
• include the exchange hours in the trading calendars documentation to assist with selecting the best calendar

quantrocket-client:2.6.0

• fix a performance issue in get_prices when querying a database with text fields (which is possible starting in version 2.5.0 due to custom database support). See related forum post.
• when querying a real-time aggregate database with get_prices and passing a timezone parameter along with start_date and/or end_date, the timezone will now be passed to the realtime service to aid in the interpretation of start_date/end_date. See related forum post.

quantrocket/blotter:2.6.0

• Add new endpoint for recording executions that happen outside of QuantRocket's knowledge. This can be used to keep the blotter aligned with your broker when one of your positions undergoes a tender offer, merger or acquisition, etc. See the usage guide.

quantrocket/houston:2.6.0

• Update to OpenResty version 1.19.3.1.
• Fix an issue where connecting to the IB Gateway GUI of a cloud deployment incorrectly counted against the concurrent install limit. See related forum post.

quantrocket/ibg:2.6.0

• update to IB Gateway version 981, the current stable version

quantrocket/jupyter:2.6.0

• Remove the separate "Zipline environment" kernel. Zipline is now installed in the standard Python 3 kernel.
• update to JupyterLab 3.0. This release includes a table of contents feature to facilitate navigating through long notebooks.
• update to quantrocket-client 2.6.0
• remove QGrid. See the fuller note above in API Changes.

quantrocket/moonshot:2.6.0

• add a new ACCOUNT_BALANCE_FIELD parameter to the Moonshot class. This parameter allows you to customize which account balance field should be used to calculate order quantities in live trading. See the full description of the parameter in the API Reference.
• fix a bug that prevented specifying integer account numbers when using the --accounts options with quantrocket moonshot trade

quantrocket/postgres:2.6.0

• update to TimescaleDB version 2.3.0. This is a maintenance upgrade with no significant new features.

quantrocket/realtime:2.6.0

• fix an issue where including a large number of sids in the query parameters could result in "Argument too long" errors. See related forum post.
• change the treatment of the end_date parameter in quantrocket.realtime.download_market_data_file to be inclusive when only a date with no time is specified. Previously such a query was exclusive because the end date was automatically normalized to 00:00:00 (midnight). See related forum post.
• improve reliability when automatically restarting Alpaca real-time data collection after a network disconnection

quantrocket/zipline:2.6.0

• modernize zipline to run on Python 3.8, Pandas 1.2, and other modern package versions. See the fuller note above.
• support for using calendars that incorporate trading breaks (applicable to some futures exchanges and some Asian exchanges). The minutes during the break will now be omitted from Zipline, meaning handle_data and other functions will not be called for those minutes.
• respect EODCancel (end-of-day cancel policy) in daily mode. Previously, Zipline applied EODCancel in minute mode but ignored it in daily mode. In daily mode, Zipline now cancels any orders which do not fill in the next session after they are placed. See related forum post.
• improve memory profile of backtests by reducing Zipline's cache sizes. See related forum post.
• when running a backtest or trading using a daily bundle, the --data-frequency/data_frequency parameter will default to "daily" if omitted. Previously, it was necessary to specify the data frequency as daily explicitly.
• fix a bug that prevented specifying integer account numbers when using the --accounts options with quantrocket zipline trade
• log a warning in live trading when a position has been delisted, instructing the user to manually record the execution (using the blotter's new API for recording executions). Previously, a NotImplemented error was raised when this situation arose. See related forum post.

trading_calendars:2.6.0

• incorporate all functionality that was previously provided by quantrocket_trading_calendars. (The quantrocket_trading_calendars package is now deprecated and will print a deprecation warning if imported.)
• add new calendars for different CME-operated exchanges and products to better support futures, including trading breaks where appropriate. See the usage guide for the associated hours and breaks for each new or updated exchange:
  • CMES/CME/GLOBEX (FX and interest rates)
  • CME_EQUITY (E-minis)
  • CME_EQUITY_LIQUID (E-mini liquid hours as defined by Interactive Brokers)
  • NYMEX (energy futures)
  • CBOT (agricultural futures)
• include the 11:30am - 12:30pm lunch break in the calendar for the Tokyo stock exchange (XTKS/TSEJ)

Release 2.5.0

2021-03-11

Release 2.5.0 introduces support for custom data, real-time data from Alpaca, multi-account support for Alpaca, an update to IB Gateway, and other enhancements.

Highlights

• Load custom data in a history database then query it using QuantRocket's standard APIs. See the usage guide.
• Real-time data from Alpaca. See the usage guide.
• Connect to multiple live and/or paper Alpaca accounts. See the usage guide.
• Update to IB Gateway version 981. See the note below.

IB Gateway Version Update

This release updates the installed version of IB Gateway to version 981, the current latest version from Interactive Brokers. This version of IB Gateway includes a new daily auto-restart feature which runs at 11:45 PM New York time, and which users should be aware of. With this feature, users with two-factor authentication enabled will only need to manually re-authenticate once a week, on Sundays. The new auto-restart feature also renders unnecessary the recommendation given in previous versions of QuantRocket to restart IB Gateway daily via the countdown service crontab. Learn more about auto-restart.

In addition, users with two-factor authentication no longer need to manually update IB Gateway settings after logging in. QuantRocket will now adjust settings automatically regardless of whether two-factor authentication is used.

Library updates

• Starting with this version, images (other than the quantrocket/zipline image) are based on the Miniconda distribution instead of the full Anaconda distribution. Core data science packages are still installed in containers. However, it is possible that a particular Python package that was included in the full Anaconda distribution and was utilized by custom user code is no longer installed. If this happens, please install the package as needed in a satellite container.
• The main Python 3 environment has been updated from Python 3.7 to Python 3.8 and from pandas 1.0 to pandas 1.2. (This does not apply to the Zipline environment or container.)

Complete 2.5.0 release notes

quantrocket/account: 2.5.0

• collect balances and portfolios from multiple Alpaca accounts, if connected

quantrocket/blotter:2.5.0

• support for multiple Alpaca accounts. Orders are routed to the proper API connection based on the account number.
• fix errors that resulted from Alpaca switching from alphanumeric to numeric account numbers for live accounts. See related forum post.

quantrocket-client:2.5.0

• add new utility functions in the quantrocket.db module for inserting custom data into a database. See the usage guide.
• add new function quantrocket.get_prices_reindexed_like. Primarily intended for use with custom data, this function can query any database queryable with get_prices and return the data in the same shape as an input DataFrame. See the usage guide.
• fix an issue where querying multiple databases failed if any of the databases returned no data. See related forum post.

quantrocket/history:2.5.0

• add support for creating custom databases. See the usage guide.
• add a free parameter to quantrocket history create-usstock-db/quantrocket.history.create_usstock_db for requesting free sample data. This parameter, which mirrors a similar parameter in quantrocket.zipline.create_usstock_bundle, is now the preferred parameter for requesting free sample data. The previously used parameter, universe="FREE", is still available but is deprecated and will print a deprecation warning.

quantrocket/ibg:2.5.981.0

• update to IB Gateway version 981. See the fuller note above.

quantrocket/jupyter:2.5.0

• pyfolio update: treat short positions in Moonshot results as reducing cash in pyfolio tear sheets. See related forum post.

quantrocket/license-service:2.5.0

• add a parameter for specifying your Alpaca data permission when entering your API key. See the usage guide.

quantrocket/master:2.5.0

• when collecting listings from Interactive Brokers, and IB Gateway is not running, the master service will now wait for IB Gateway to start, instead of failing immediately as in previous versions. This mirrors the existing behavior of the history and fundamental services when collecting data from Interactive Brokers. This change does not apply to data collection for options, which will still fail immediately if IB Gateway is not running.

quantrocket/moonshot:2.5.0

• add support for combining a mix of intraday and daily strategies in the same backtest. See related forum post.

quantrocket/postgres:2.5.0

• update to TimescaleDB version 2.1.0

quantrocket/realtime:2.5.0

• add support for collecting real-time data from Alpaca. See the usage guide.

quantrocket/satellite:2.5.0

• return output from custom Python functions accessed using dot notation. See the usage guide.

quantrocket/zipline:2.5.0

• add support for loading custom databases into Pipeline. See the usage guide.
• automatically send GTC (Good-till-canceled) orders in live trading if the algorithm's cancel policy is set to NeverCancel. See the usage guide.
• in live trading, if querying a real-time database based on native minute aggregates (which is the recommended approach for Alpaca real-time databases), Zipline will query the real-time database repeatedly until all of the minute data has arrived. Zipline monitors for two successive queries to return the same number of records as an indication that all minute data has arrived. This design avoids a potential race condition where Zipline might query for minute data before the minute data has fully arrived from the data provider.
• fix an issue where backtests using the US stock minute bundle failed with a segmentation fault or other C error if the backtest spanned a particular date in May 2020 and utilized certain Zipline API functions. See related forum post.
• fix an issue where querying daily data in Zipline live trading using data.history(... '1d') caused an error if real-time data collection was initiated before the market opened
• fix an issue where NULL order statuses in live trading (meaning the broker has not yet sent an order status) caused Zipline to fail with an unhandled exception. See related forum post.

Release 2.4.1

2021-01-21

This is a patch release that fixes a regression affecting data collection of the US Stock minute bundle. Users of the US Stock minute bundle who collected the bundle prior to the release of version 2.4.0 and are now collecting bundle updates using version 2.4.0 are affected by this regression and should update to 2.4.1. Users who first collected the minute bundle using version 2.4.0 are not affected. Users who are only collecting the daily bundle are also not affected.

quantrocket/zipline:2.4.1

• Fix a regression introduced in version 2.4.0 where collecting updates for an existing minute bundle resulted in only the daily updates being synced from the cloud. Background: before version 2.4.0, the stored US Stock bundle config did not specify "minute" or "daily" frequency, since only "minute" was supported. With the introduction of "daily" support starting in 2.4.0, "daily" was incorrectly being assumed as the data frequency for existing bundles that did not specify a data frequency.

Important: After updating to 2.4.1, users should run a "force" update of their minute bundle to sync any missing minute data, which can be done as follows by substituting the correct bundle name:

$ BUNDLE_NAME=usstock-1min
$ curl -X POST "http://houston/zipline/ingestions/$BUNDLE_NAME?force=true"

Release 2.4.0

2021-01-14

Release 2.4.0 includes a number of enhancements related to real-time data, the US Stock dataset, and closer feature parity with Quantopian.

Highlights

• There are two new additions to the Code Library:
  • Quant Finance Lectures. This is the Quantopian Lecture Series updated for QuantRocket.
  • Pipeline tutorial. This tutorial provides an in-depth walkthrough of the Pipeline API.
• It is now possible to collect real-time data for large universes, such as the entire US stock market, by collecting minute or second aggregate data from Polygon.io instead of full tick data. See the usage guide and the updated configuration guidance for using Polygon.io data with Zipline.
• There are several new securities master fields for the US Stock dataset:
  • Sector and Industry. See an example record in the usage guide.
  • Primary share. A new field distinguishes primary from secondary share classes for companies with multiple share classes. See the usage guide.
• You can now query daily data from a minute bundle using the new data_frequency parameter on quantrocket.get_prices and quantrocket.zipline.download_bundle_file. See the usage guide.
• Users who don't want to collect the full US Stock minute bundle can ingest only the daily portion of the bundle for use in Zipline, the Pipeline API, and throughout QuantRocket. See the usage guide for a comparison of the different ways to access the US Stock dataset.
• There are two Python API enhancements for the securities master service:
  • a new convenience function, quantrocket.master.get_securities, queries the securities master database and loads the results into a DataFrame. (Previously, downloading the master file and loading it into a DataFrame were two separate steps.) See the API Reference.
  • The function quantrocket.master.create_universe now accepts a sids parameter that makes it easier to create a universe from a DataFrame without having to write the DataFrame to a CSV. See the API Reference for an example.

Documentation

• A new section of the usage guide documents how to replicate the QTradableStocksUS universe, a popular Pipeline filter on Quantopian.
• A detailed note has been added to the docstring for quantrocket.master.download_master_file and quantrocket.master.get_securities explaining how multiple parameters are combined when querying the securities master. See the API Reference.
• A note has been added to usage guide mentioning that print statements will show up in the detailed logs.

API Changes

• the function quantrocket.zipline.download_minute_file has been renamed download_bundle_file since it can now be used to download minute or daily data, using the new data_frequency parameter. The old function is still available but will print a deprecation warning.

Sample Data

• SPY is now included in the free sample data

Complete 2.4.0 release notes

quantrocket/blotter:2.4.0

• reduce memory usage in PNL calculation when there is a large number of executions

quantrocket/flightlog:2.4.0

• make quantrocket.flightlog.FlightlogHandler a singleton to prevent users from unintentionally logging duplicate messages in their code

quantrocket/fundamental:2.4.0

• better error handling when the IBKR API returns invalid XML indicating no Reuters estimates data is available

quantrocket/master:2.4.0

• add Sector and Industry fields to US Stock dataset. See an example record in the usage guide.
• add a field to distinguish primary vs secondary share class in the US Stock dataset. See the usage guide.
• add a new convenience function quantrocket.master.get_securities, which queries the securities master database and loads the results into a DataFrame. (Previously, downloading the master file and loading it into a DataFrame were two separate steps.) See the API Reference.
• add a sids parameter to quantrocket.master.create_universe to make it easier to create a universe from a DataFrame without having to write the DataFrame to a CSV. See the API Reference for an example.

quantrocket/moonshot:2.4.0

• add new paramater DB_DATA_FREQUENCY which can used for querying daily data from a Zipline minute bundle. See the API Reference.

quantrocket/postgres:2.4.0

• changes the default PostgreSQL log level in order to silence the excessively noisy TimescaleDB logging related to aggregate databases seen in previous versions

quantrocket/realtime:2.4.0

• add support for collecting Polygon.io minute or second aggregates, which facilitates real-time data collection for large universes. See the usage guide, the updated note on database performance, and the updated configuration guidance for using Polygon.io data with Zipline.

quantrocket/zipline:2.4.0

• support for querying daily data from a minute bundle using the new data_frequency parameter on quantrocket.get_prices and quantrocket.zipline.download_bundle_file. See the usage guide.
• support for ingesting only the daily portion of the US Stock minute bundle. This is provided as a convenience for users who don't want to collect the full minute bundle. See the usage guide for a comparison of the different ways to access the US Stock dataset.
• add new US Stock master fields (usstock_Sector, usstock_Industry, usstock_PrimaryShareSid, and usstock_CIK) to SecuritiesMaster Pipeline dataset
• roll-up real-time minute data in live trading to support queries for partial day daily data using data.hist(... '1d'). See related forum post.

quantrocket_trading_calendars:2.4.0

• add a trading calendar for US extended hours which runs from 4 AM to 8 PM: us_extended_hours. This can be used when querying trading hours from the master service or when ingesting a history database with extended hours data into Zipline.

Release 2.3.2

2020-12-21

This is a patch release that fixes a difference in behavior between Zipline backtesting and live trading. See the fuller note below.

quantrocket/master:2.3.2

• allow querying calendar without first collecting listings. This is a minor change to support the QuickStart. See related forum post.

quantrocket/zipline:2.3.2

• In live trading, return real-time data from the previously completed minute, not from the current, partially completed minute.
  • Background: In Zipline backtesting, the most recent data available to strategies is from the previously completed minute, but in live trading, real-time data from the current, partially completed minute was being returned, if available. For example: suppose a Zipline strategy queries data.current(asset, ['open', 'high', 'low', 'close', 'volume']) at 9:32:00. In a backtest, the OHLCV of the 9:31:00 bar is returned, representing the trades that occurred from 09:31:00-09:31:59. In live trading, prior to this update, the same request would return the OHLCV of the 9:32:00 bar, representing trades that occurred from 09:32:00-09:32:59. Since this data was returned only a few seconds after 09:32:00, there wasn't much data to return (because the minute had just started), which often resulted in NaNs. This update aligns the behavior of live trading with that of backtesting so that a query at 9:32:00 always returns the 09:31:00-09:31:59 OHLCV. See related forum post.

Release 2.3.1

2020-12-02

This is a patch release that fixes an issue preventing some Mac users from using the Zipline environment kernel in JupyterLab. If you are not a Mac user or do not use the Zipline environment in JupyterLab, this update will not affect you and is optional.

quantrocket/jupyter:2.3.1

• Fix an issue affecting some Mac users where, after starting the Zipline environment kernel in JupyterLab and importing the zipline module, the kernel would immediately restart.

Release 2.3.0

2020-10-22

Release 2.3.0 includes improvements for teams and financial advisors, and other enhancements and bug fixes.

Highlights

• improvements for teams, including a dedicated documentation section, support for linked license keys, and support for read-only S3 credentials. See the teams documentation
• new dedicated documentation section for financial advisors, and support for placing FA Group/FA Profile orders through Interactive Brokers. See the documentation.
• new endpoint: quantrocket flightlog wait/quantrocket.flightlog.wait_for_message. This function allows you to search for a log message and block until the message appears. This can be useful when you need your code to wait until a background process such as data collection finishes. See the usage guide.
• add continuous_future function to Zipline research environment. See the API Reference.

Documentation

• Document continuous futures in Zipline
• Document how to use Linux programs on your crontab
• Document how to incorporate non-Pipeline datasets in Zipline strategies

Complete 2.3.0 release notes

quantrocket/blotter:2.3.0

• support for placing FA Group/FA Profile orders through Interactive Brokers. See the usage guide.
• fix an issue where only a subset of order IDs might be returned when running Alpaca live and paper trading simultaneously and querying order status with the --open/open_orders=True parameter

quantrocket/db:2.3.0

• add a --region/region parameter to quantrocket db s3config/quantrocket.db.set_s3_config to support pushing to and pulling from S3 buckets in regions other than us-east-1. See the API Reference.
• read-only S3 credentials are now supported and can be used for pulling databases from S3 (without allowing pushing to S3). Intended for teams.

quantrocket/flightlog:2.3.0

• new endpoint: quantrocket flightlog wait/quantrocket.flightlog.wait_for_message. This function allows you to search for a log message and block until the message appears. See the usage guide.

quantrocket/history:2.3.0

• fix an issue reported in a forum post where IBKR data collection failed because ibkr_ConIds were not treated as integers.
• fix an issue where multiple corporate actions occurring for the same security on the same day resulted in the second corporate action being incorrectly adjusted by the amount of the first adjustment factor, in EDI history databases

quantrocket/license-service:2.3.0

• license keys are now encrypted at rest and are obfuscated in display output.
• support linked license keys for teams.

quantrocket/realtime:2.3.0

• fix an issue reported in a forum post where IBKR data collection failed because ibkr_ConIds were not treated as integers.
• automatically re-connect when PostgreSQL connection is disconnected after being idle

quantrocket/zipline:2.3.0

• add continuous_future function to research environment. See the API Reference.
• fix futures contracts being out of order in data.current_chain(). See related forum post.
• modify continuous futures "calendar" roll to roll on RolloverDate, not LastTradeDate. See related forum post.
• fix an issue reported in a forum post where Zipline backtests could fail with KeyError due to stale trading calendars in the calendar cache.
• make REPORTPERIOD and CALENDARDATE Sharadar fundamentals fields available in Pipeline API

Release 2.2.0

2020-09-03

Release 2.2.0 contains Zipline enhancements, performance improvements, usability improvements, and bug fixes.

Highlights

• live trading of end-of-day strategies that use daily data. Previously live trading required the use of minute data. See the usage guide.
• add Zipline pipeline integration for Alpaca easy-to-borrow and IBKR shortable shares datasets. See the usage guide.
• data quality enhancements for US Stock dataset. See Data Corrections below.

API Changes

• The preferred syntax for constructing Zipline pipelines with Sharadar or Reuters data has changed. The new syntax utilizes slicing to provide a cleaner syntax that better aligns with the conventions used on quantopian.com. For example, the syntax SharadarQuarterlyFundamentals.REVENUE is now deprecated in favor of sharadar.Fundamentals.slice(dimension="ARQ", period_offset=0).REVENUE. However, the deprecated conventions will still work. See the usage guide.

Documentation

• a new Zipline example strategy in the Code Library demonstrates live trading of an intraday sell-on-gap strategy

Data Corrections

Corrections have been applied to the US Stock dataset to link together certain price series that were previously split up due to corporate restructurings. For example, in 2015 Google restructured and formed a new parent company, Alphabet. The US Stock dataset previously provided Google's price history under two separate sids: one for the pre-2015 Google and one for the post-2015 Alphabet. These are now linked together under one sid. This also applies to a number of other securities. For additional background, see related forum post.

To update your local copy of the dataset with these corrections, follow these instructions:

US Stock end-of-day dataset

Please drop and re-collect the dataset.

US Stock intraday bundle

Since version 2.1.0, the US Stock Zipline bundle can ingest data either sid by sid or day by day. The sid by sid method is used for initial collection or if there are more than 5 days' worth of updates to ingest. The day by day method is used if there are fewer than 5 days' worth of updates to ingest, as it is much faster in these cases.

The data corrections will be automatically ingested the next time the sid by sid method is used. To force this to happen, you can use the force parameter. This parameter is not included in the client library, so use curl as shown below:

$ BUNDLE_NAME=usstock-1min
$ curl -X POST "http://houston/zipline/ingestions/$BUNDLE_NAME?force=true"

Complete 2.2.0 release notes

quantrocket/blotter:2.2.0

• fix an issue where the blotter would continue to request the order status of orders already rejected by Alpaca

quantrocket/codeload:2.2.0

• add a GIT_KEEP_METADATA environment variable which can be used to automatically load a Git repository when your deployment first launches. (This is an advanced use case.) See the usage guide.

quantrocket/fundamental:2.2.0

• gracefully handle a new situation where the IBKR API may indicate that no Reuters estimates data is available by returning an invalid XML file instead of the usual API error code. See support forum post for background.

quantrocket/history:2.2.0

• improve error handling when you request historical data from IBKR and have not yet collected the security listings from IBKR

quantrocket/jupyter:2.2.0

• Add the QuantRocket version number to the JupyterLab menu bar, with an indicator showing when new updates are available.
• fix a performance issue with quantrocket.fundamental.get_sharadar_sp500_reindexed_like. See forum post for background.

quantrocket/postgres:2.2.0

• update to TimescaleDB 1.7.3

quantrocket/realtime:2.2.0

• improve error handling when you request real-time data from IBKR and have not yet collected the security listings from IBKR

quantrocket/zipline:2.2.0

• live trading of end-of-day strategies that use daily data. Previously live trading required the use of minute data. See the usage guide.
• add Zipline pipeline integration for Alpaca easy-to-borrow and IBKR shortable shares datasets. See the usage guide.
• improve query performance of quantrocket.zipline.download_bundle_file
• add Order.open attribute as a shortcut for checking if an order is open. See API reference.
• The preferred syntax for constructing Zipline pipelines with Sharadar or Reuters data has changed. The new syntax utilizes slicing to provide a cleaner syntax that better aligns with the conventions used on quantopian.com. For example, the syntax SharadarQuarterlyFundamentals.REVENUE is now deprecated in favor of sharadar.Fundamentals.slice(dimension="ARQ", period_offset=0).REVENUE. However, the deprecated conventions will still work. See the usage guide.
• fix an error when trying to load a context file containing a ContinuousFuture in live trading
• fix a bug that could cause the incremental ingestion of daily usstock updates to terminate early if encountering sids with no data for a particular date
• update pandas version from 0.20.1 to 0.22.0 on the zipline service. This matches the pandas version in the "Zipline environment" kernel in JupyterLab.

Release 2.1.1

2020-07-27

This is a patch release that extends Zipline pipeline support to include futures and non-US equities.

quantrocket/jupyter:2.1.1

• update Zipline environment with the changes described under quantrocket/zipline:2.1.1

quantrocket/zipline:2.1.1

• add pipeline support for futures
• fix an issue preventing the use of pipelines with non-US equities. See related forum post.
• automatically derive the pipeline domain from the trading calendar so that a domain need not be specified

Release 2.1.0

2020-07-23

Release 2.1.0 contains a number of enhancements and usability improvements for Zipline strategy development, as well as bug fixes.

Highlights

• This release introduces a new Zipline Research API which greatly improves usability by allowing you to perform much of your Zipline strategy development within the interactive environment of a research notebook. It includes support for running pipelines and accessing the data objects used in Zipline strategies. Along with this change, a Zipline kernel has been added to JupyterLab. See the usage guide.
• A progress meter has been added to Zipline backtests that provides progress and performance statistics during long-running backtests. See the usage guide.
• The new Zipline Intro tutorial in the Code Library walks you through the improved Zipline workflow.
• The runtime when ingesting daily updates of the US Stock 1-minute dataset has been signficantly improved. See the fuller note in the API Changes section.

API Changes

• The runtime for collecting daily updates of the US Stock 1-minute bundle has significantly improved. Previously, collecting an update took several hours but should now take only 10-15 minutes per day if updated daily (longer if collecting multiple days' updates). To take advantage of this speed improvement, you must drop and re-ingest the entire bundle. We regret this inconvenience, but previously ingested bundles did not store enough metadata to take advantage of the newly redesigned ingestion strategy. (Background: Previously, data was ingested by syncing one security at a time from the cloud to your local deployment. On updates, only the diff needed to be synced but determining the diff still took time. Now, if the Zipline service determines it will be faster, it will utilize an alternate strategy of loading individual daily update files, that is, ingesting data day by day instead of sid by sid. This is signficantly faster when you only need to ingest a day or a few day's worth of updates.)
• In Zipline strategies, commissions and slippage are now disabled by default. See the usage guide for reasoning and examples. The examples are based on the previous default behavior and can be used by users who wish to preserve the previous default behavior. (Note for futures traders: the dictionary of exchange fees by root symbol in the linked example only includes a subset of root symbols for simplicity. The full list of exchange fees by root symbol that was previously used for default futures commissions is available in GitHub.)
• In Zipline strategies, storing asset objects to context in the initialize() function is no longer supported and will throw an error. Instead, if you need to store asset objects to context, please do so in before_trading_start (or another function called after initialize). Storing asset objects to context in initialize() causes the asset objects to become stale in live trading because the asset objects are always loaded from the stored context instead of being re-instantiated each day. Daily re-instantiation of asset objects is necessary to ensure that the asset's metadata (particularly as concerns the asset's end date) stays up-to-date. Otherwise, assets will not be tradeable in live trading because Zipline will (incorrectly) think they are stale. We are choosing to immediately enforce this change instead of printing a deprecation warning due to the potential adverse effect on live trading.

The following is no longer allowed:

def initialize(context):

    context.aapl = algo.sid("FIBBG000B9XRY4")
    ...

Replace it with this:

def before_trading_start(context, data):

    context.aapl = algo.sid("FIBBG000B9XRY4")
    ...

Documentation

• Added Zipline Intro tutorial to Code Library.
• Added API reference sections for Alphalens, Pyfolio, and quantrocket-trading-calendars, and improved API reference for Zipline.

Complete 2.1.0 release notes

quantrocket/account:2.1.0

• fix a bug where checking your portfolio before collecting securities master listings for your broker caused an error

quantrocket/blotter:2.1.0

• improve reliability of Alpaca execution monitoring by always requesting an overlapping buffer of executions in case execution records arrive out of order. See related forum post.

quantrocket/history:2.1.0

• fix an issue caused by a change in the IBKR API behavior which resulted in missing dates when collecting 1-day bars from IBKR. See related forum post.
• query results will now respect the use of a datetime string (YYYY-MM-DD HH:MM:SS) for the start_date or end_date parameter when querying an intraday history database. Previously, only the date part was extracted and used in the query, even if a datetime string was passed.

quantrocket/ibg:2.1.972.0

• improve error handling during rollback when setting IBKR credentials fails. See related forum post.

quantrocket/jupyter:2.1.0

• add Zipline Research API, with support for running pipelines and accessing data objects in interactive research. See the usage guide.
• add a "Zipline environment" kernel to support using the Research API for Zipline.
• add example templates to JupyterLab Launcher for Zipline strategies, Moonshot and Zipline allocation files, countdown crontabs, rollover rules configuration files, and IB Gateway permission files. Previously Launcher templates were only available for Moonshot and MoonshotML strategies.

quantrocket/postgres:2.1.0

• update to TimescaleDB 1.7.2

quantrocket/realtime:2.1.0

• fix an unhandled exception that occurred when you try to drop ticks from a real-time database that has no data

quantrocket/zipline:2.1.0

• add progress meter to Zipline backtests. See the usage guide.
• add endpoint to check bundle config. See API reference.
• use multithreading during initial ingestion of US Stock 1-minute data bundle, reducing runtime from 15-18 hours previously to 12-15 hours currently.
• improve runtime when ingesting daily updates of the US Stock 1-minute dataset. See the fuller note in the API Changes section above.
• ingest data one security at a time when ingesting from a daily history database. Previously the entire database was ingested in one go which failed on large databases. See related forum post.
• default to applying no commissions or slippage on backtests. See the usage guide for reasoning and examples. The examples are based on the previous default behavior and can be used by users who wish to preserve the previous default behavior.
• fix an issue where the presence of illiquid securities with no price data caused an error when querying the US Stock 1-minute data bundle without specifying sids or universes. See related forum post.
• fix an issue where ingestion failed if a security had a NULL symbol. See related forum post.
• improve error message when you query a Zipline bundle using a date range that is not present in the trading calendar
• fix an issue where ingesting a history database failed if you hadn't previously collected listings from Interactive Brokers
• provide a helpful error message if user tries to ingest two distinct futures chains (for example that trade on different exchanges) that use the same root symbol

Release 2.0.0

2020-06-09

Highlights

• New data: this release includes numerous new data providers and datasets, including end-of-day, intraday, real-time, and fundamental data. See the Data Library for available datasets.
• Zipline live trading. See the usage guide.
• New broker: Alpaca. See how to connect and how to place and manage orders.

API changes

QuantRocket version 2 is a major product upgrade from version 1. Due to the scale of changes, there are many breaking API changes that version 1 users should be aware of. See the migration guide.

Complete 2.0.0 release notes

quantrocket/account:2.0.0

• support for querying account balances and portfolios from Alpaca
• store all available exchange rates from ECB
• derive GBX exchange rate from GBP

quantrocket/blotter:2.0.0

• support for Alpaca. See usage guide
• fix a SQL error that happens when you query summary pnl with a date range only and no other parameters

quantrocket/db:2.0.0

• encrypt S3 credentials at rest

quantrocket/fundamental:2.0.0

• new data: Sharadar insider holdings
• new data: Sharadar institutional ownership
• new data: Sharadar SEC Form 8-K
• new data: Sharadar S&P 500 constituents
• new data: Alpaca easy-to-borrow stocks

quantrocket/history:2.0.0

• new data: US Stock prices (end-of-day and intraday)
• new data: EDI global end-of-day stock prices
• new data: Sharadar end-of-day fund prices

quantrocket/ibg:2.0.972.0

• encrypt IBKR credentials at rest

quantrocket/ibgrouter:2.0.0

• renamed service from launchpad to ibgrouter

quantrocket/jupyter:2.0.0

• update to JupyterLab 1.2.7
• add support for a .bashrc file. See usage guide
• document attaching VS Code to the jupyter container

quantrocket/license-service:2.0.0

• add endpoint to connect to Alpaca
• add endpoint to connect to Polygon.io
• add endpoint to connect to Quandl

quantrocket/master:2.0.0

• new vendor: collect Alpaca listings
• new vendor: collect EDI listings
• new vendor: collect FIGI listings
• new vendor: collect US Stock listings
• consolidate securities master file from multiple vendors

quantrocket/postgres:2.0.0

• update to PostgreSQL 12 and TimescaleDB 1.7.1

quantrocket/realtime:2.0.0

• new real-time data provider: Polygon.io
• add ability to drop ticks from tick databases. See usage guide
• automatic compression of real-time databases

quantrocket/satellite:2.0.0

• support for custom setup scripts to customize environment

quantrocket/theia:2.0.0

• update to latest version of Eclipse Theia

quantrocket/zipline:2.0.0

• Zipline live trading. See the usage guide
• redesign data collection API to mirror the API of the history service. See usage guide
• support for incremental ingestion of history databases. See usage guide
• Add pipeline support for Sharadar fundamentals, institutions, S&P 500 constituents, Reuters financials and estimates, and securities master. See the usage guide