The Dangers of Cut & Paste: The 5 Biggest Challenges of Optimizing Futures Execution Algorithms

Some execution algorithm providers approach the design of futures algorithms by starting with an equity algo base and editing the details to tailor to the specificities of trading futures. While this approach may create a fair foundation, on average, it leads to unnecessarily high trading costs—sometimes in extreme fashion—because futures come with their own unique market structure and market microstructure challenges. The design of futures algorithms should be treated thoughtfully as a result. Here, we include just a few examples of the unique challenges specific to futures trading that make repurposed equity algorithms problematic. 

Estimating daily volume for futures contracts is uniquely challenging because of contract expiration and roll dynamics

A fundamental example of the failure of cut and paste when it comes to using equities algorithms for futures trading is volume prediction. In order to make educated decisions around order size and timing, every algorithm needs estimates of the volume it will encounter throughout the day. Equities algorithms often rely on historical volume to make judgments about the volume expected during the day ahead, for example, taking an average of the last month of daily trading volumes to estimate the next day’s volume. 

The typical approach to estimating volume for equities involves two steps. In the first step, a daily volume estimate is calculated. This process itself can be simple or complex, but a common simplistic approach takes the typical daily volume over a period of time, perhaps a 21-day (previous month) median or average. Many algorithms would also include some methodology to adjust for expected high-volume days as well, like options expiration or news days. In equities, this approach generally creates a reasonable proxy for daily volume, which can then be broken into estimates of volume corresponding to specific times of day in a second step, and those estimates then drive algorithm decisions—with measurable precision—throughout the life of an order. 

Using this strategy for futures, however, can be dangerous because of volume changes during the “roll” period of a contract. While the equities market introduces its complexity to the trading process with its heavy fragmentation of market volume across dozens of execution venues, trading futures is inherently complex because of the structure of the contracts being traded. 

Each futures contract represents an agreement over a specific period of time, with an expiration date. While some traders are interested in only a single contract expiration, many want to continue to hold exposure to a contract after it expires and must “roll” their position into the “next” contract to expire after the current one. As firms roll their investments from a soon-to-expire contract to the next active contract, the daily volume distribution of each contract is changing, and hence, the historical volume averages we might use for equities are meaningless in predicting what’s to come for a futures contract. 

The roll takes place before expiry and is a thoughtfully measured phenomenon since so many investors participate. As a contract nears expiration, its trading volume declines as traders sell off their positions. Trading volume in the next active contract increases as positions roll from a soon-to-be-expired contract into the next one. Relying on historical volume for a nearly expired contract that is declining in volume can severely overpredict volume, creating more market impact than expected when trading on these estimates. Relying on historical volume for a rising contract can likewise underpredict volume and result in the transition of an investment more slowly than is necessary. 

The roll process and the associated volume changes can take a few days or much longer, depending on the contract. And if a contract takes 10 days to complete its roll, investors need to know how volume is changing during that period—a lot in the first three days and then slower, for example, or slowly at first followed by a steep decline. A contract with monthly expiration where every month is heavily traded can have a quick roll, as illustrated for Crude Oil (CL) contracts in Figure 1A below, or it can decline quickly and then linger for a few more days, as illustrated for Heating Oil (HO) contracts in Figure 1B. A quarterly contract or a monthly contract heavily traded only on quarterly expiries can take much longer to complete its roll. 

Volume prediction becomes challenging during the roll for these reasons, and certainly a historical average volume will not generate realistic volume estimates. Unrealistic volume estimates render the foundation of an algorithm’s decision-making unreliable, yielding poor trading decisions and increased costs on average.

Figure 1AB. This figure illustrates the roll behavior for Crude Oil (CL) (1A) and Heating Oil (HO) (1B) futures contracts in early 2022. Two roll periods are illustrated for each, with the volume of the earliest contract illustrated in navy blue, declining toward January expiry. The volume of the second contract (February expiry, shown in blue) is rising as January’s contract falls and then declines as the March contract (in green) grows in volume in February. Heating Oil (HO) contracts (1B) appear to decline in volume more slowly than the Crude Oil (CL) contracts in 1A.

In addition to the volume estimation challenges associated with the roll period, there are additional challenges introduced by the idiosyncratic behaviors of individual base symbols. Outside of the roll period, some contracts have heavy trading volume in only a single contract at a time, the nearest to expiration. But others have volume in multiple contracts simultaneously, like Natural Gas, as shown in Figure 2. This compounds the complexity of volume estimation, as multiple contracts trade daily and each participates in synchronized, heavily correlated roll behavior as expiration dates approach. The volume behavior (and roll behavior) of contracts is native to the specific base symbol, yet another reason why volume estimation for futures is so complex. And, of course, calendar spreads that are used to roll contracts from one expiry to another and traded by relative value traders experience the same obstacles related to volume estimation. 

Figure 2. This figure illustrates a base symbol with volume in multiple contracts with varying expiration dates, Natural Gas. On each trading day there is volume in multiple contracts, with more than one contract having significant volume simultaneously. As one contract nears expiration, the next contract rises in volume, but a second contract slowly rises in volume as well, two months from expiration.

Because of this idiosyncratic roll behavior, the traditional approach to equities volume estimation of taking a 21-day view of daily volume, for example, is a poor fit and ignores important features of futures market structure. Correcting this mismatch in estimation methodologies can unlock far greater accuracy in estimating daily volumes for futures trading, which improves every detail of decision making by an execution algorithm and yields dramatically improved execution costs, on average. 

The phrase “garbage in, garbage out” certainly applies to algorithmic trading where key analytics are the foundation of performance—for better or worse. Poor estimates of pre-trade analytics yield significantly higher execution costs, but this can be significantly improved if futures-specific and contract-specific behaviors are accounted for in estimation. 

Some futures contracts have exceptionally long queues, making passive execution a challenge

Like volume estimation, the diverse queue times of futures’ limit order books can drive increased costs if mishandled by equities algorithms masquerading as futures algorithms. 

As in equities trading, there are two types of costs that add up to increased execution costs–market impact costs and spread costs. Specifically, each slice of a larger parent order pays some amount of the bid-offer spread as it executes on an exchange. Buy orders placed at the bid and sell orders placed at the offer earn half of the spread when executed. This is generally a good thing, though it can indeed lead to increased adverse selection costs¹. Market orders to buy and sell “cross the spread”, paying half of the spread as a premium for the benefit of immediate execution. 

In equities trading, where there are many queues for limit order placement due to market fragmentation, and each queue with its own rules (see our recent research on queue-jumping), orders can be executed more quickly relative to some futures contracts, which have exceptionally long queues. But each futures contract trades on a single exchange, and algorithms aiming to earn the spread must wait in a single queue to buy or sell. An order may be waiting with many other orders at the same price—hundreds, thousands, even millions, depending on the contract—for the pleasure of earning the corresponding spread premium. 

Research indicates that bid-offer spreads are proportional to the volatility of a contract. In addition, spreads are inversely proportional to contract volume. Therefore, volatility and volume should be used to determine minimum tick size—the minimum allowed spread increment. For futures (unlike equities), exchanges can determine their own minimum tick size requirement, which may be far larger than the “fair” bid-offer spread of a contract that is based on its volatility and volume. When the fair spread is narrower than the minimum allowed spread, the cost of crossing the spread (and reward for earning the spread) is higher. As a result, there are many market participants’ limit orders waiting at the prices aligned with the minimum tick increments, forming long queues. 

In the case of long-queue contracts, algorithms’ limit orders often wait endlessly in a queue of other orders, never making it to the front of the queue for execution. The algorithm must then cross the spread to remain on its planned trading schedule—yielding execution costs that are heavy in spread costs with few passive executions. And if waiting in a long queue does end happily in a passive execution, the portion of spread paid over the entire order is still quite large as that single passive execution is a relative anomaly. 

Long queues are a common feature of some futures contracts where the market is heavily crowded, as is the case for 10-Year T-Note contracts (ZN) as shown in Figure 3. Other contracts shown in Figure 3 have much shorter queues, where algorithms can earn the spread more easily. This phenomenon of long queues is less common in most equity markets, except for very low-priced, highly liquid stocks, as a result of tick size specifications². In equities markets, one can mitigate the issue by trading inside the bid-offer spread in dark pools or utilizing queue jumping techniques (as detailed in our paper referenced above), but tactics enabled (and required!) by the market fragmentation in equities are simply not available in futures markets. 

Figure 3. This figure illustrates the queue size for a variety of contracts, specifically highlighting the long queues for the 10-year T-note contract (ZN). Long-queue contracts require differentiated handling by execution algorithms because they can reduce the likelihood of passive execution and increase adverse selection.

If execution algorithm designers ignore the queue-length issue in equities algorithms, execution costs suffer in a few cases and are not overly disruptive to average performance. However, in the case of an equity algorithm’s foundation used for futures execution, execution costs can be heavily inflated as algorithms that execute frequently on the passive side for equities will cross the spread for most executions on long-queued futures contracts. Making matters worse, if algo designers customize behaviors to long-queued futures contracts and apply a broad-brush approach to all contracts, overrepresenting orders in the queue to increase passive executions, performance will still be poor. 

For contracts where the fair spread is higher than the minimum tick size increments due to low volumes and high volatility—for example, most commodity contracts such as Heating Oil or Crude Oil—the limit order book is sparse, and orders must be layered within it to optimize execution. In such contracts, placing large orders at the best bid or offer has the opposite effect, increasing costs. Large orders often create market impact and come with increased adverse selection, as fills are likely received when prices are moving in the orders’ favor. 

The issue of higher volatility coupled with lower volume does not apply only to commodity futures contracts. It also applies to most mid-cap and small-cap stocks, and it is well known that most traditional equity algorithms don’t work well for these categories. But the effect is exacerbated in futures, where the volatility of some contracts can be far higher than mid- and small-cap stocks tend to be. Most firms concentrate their trading in less than fifty distinct contracts, and as a result, inefficient trading can easily impact fund performance. 

Here again, typical approaches to order placement for equities can be dangerous when applied to futures. We often hear from clients that certain offerings work well for less volatile contracts and others work well for more volatile contracts; this is likely because the associated providers have designed their order placement strategies with the broad brush described above. For optimal futures execution, algorithms must dynamically address the contract-specific relationship between fair spread (which depends on volatility and volume) and minimum tick size, which can lead to contracts with uniquely long queues, sparse limit order books, and everything in between.

Futures contracts have wildly different volatility profiles, and volatility impacts every aspect of execution

Another example of a failure of equities algorithms to adjust to the unique market structure of futures contracts is the handling of diverse volatility profiles across contracts. The distribution of inherent volatility across futures contracts is very different from that of equities. While for most stocks annualized volatility can be around 20-40%, for futures contracts it can range from over 100% (as in Heating Oil) to very close to 0% (as in short-term interest rate futures and calendar spreads). 

As described above, tick sizes determined by exchanges often do not take into account the volatility of the underlying product. For some products, tick sizes are too large with respect to volatility, leading to very long queues and making passive fills hard to come by. On the other hand, for products with very high volatility like most commodity futures, for example, managing adverse selection costs and market impact becomes extremely challenging. Algorithms that give themselves a lot of “flexibility” around a trade plan tend to improve performance for low volatility products, but not for high volatility products where the cost of adverse selection and market impact is extremely high, requiring more skill around limit order placement. Clients must be able to rely on algorithms to dynamically adjust, optimizing for volatility across products and for each product throughout the day, rather than having to choose different vendors for each product they trade to ensure best execution.

Execution on futures exchanges isn’t homogeneous; pro-rata versus FIFO execution can impact execution costs if not handled appropriately

Equities exchanges operate in a “First In, First Out” style across the board, where those limit orders posted first in the queue at each price level receive time priority. The order arriving first is executed first, and this is referred to as FIFO. But some futures contracts execute in a pro-rata execution style, where a subgroup of parties (or all parties) waiting in the queue receive some portion of an arriving limit order. In that event, a limit order waiting deep in a queue for ten contracts may receive a partial fill of one or more contracts while the remainder of the order continues to wait in the queue. As this matching structure does not exist for equities, an equity algorithm is not equipped to handle this style of execution—or more importantly, to handle it strategically. This is a clear case where a futures-specific market structure difference has the potential to increase costs if not properly accommodated. And the style of execution is not exchange-specific, but rather, contract-specific, making each futures contract’s optimal execution strategy highly distinguished. 

Futures trade 24 hours a day, globally

The final point in this list is certainly not the least challenging. A critical data management issue in the design of effective futures algorithms is market timing. In the US equities market, exchanges operate from 9:30am to 4:00pm ET each day. While there is trading outside of continuous market hours, it represents only about 3% of daily trading volume. Futures markets trade around the clock, around the globe. There tends to be higher volume traded during US, European, and APAC trading hours when trading in other asset classes like equities is most active, with lower volumes between these sessions. This phenomenon is illustrated in Figure 4A, where normalized volume is shown for both Australian Dollar FX contracts (6A) and the S&P 500 e-mini contract (ES). Volume in the e-mini is heavily targeted to US equity trading hours (9:30am-4:00pm ET), while the Australian Dollar contract trades much more outside US equity trading hours, including heavier volume while APAC and European markets are active. 

When institutions trade outside the liquid trading hours for a particular contract, volume can be very low and spreads very wide, leading to increased execution costs. As a result, even designing algorithms specific to each contract is not sufficient. Trading Crude Oil (CL) during illiquid hours may be akin to trading Heating Oil (HO) during liquid hours, as is shown in Figure 4B. Both contracts have heavy volume during US equity trading hours, but outside of those hours, traders can still find liquidity in Crude Oil while Heating Oil is far less active. 

Thus, futures algorithms must dynamically adjust for intraday volume, volatility, spread, and depth. While this is always the case for any execution algorithm, when volume is particularly low, poor pre-trade analytics (for example, volume estimation) or failure to adjust to current market conditions can result in dramatically increased execution costs. An equity algorithm repurposed for futures with basic, static pre-trade analytics can create expensive consequences for investors.

Figure 4AB. This figure illustrates the trading volume in Australian Dollar (6A) and S&P E-mini (ES) contracts (4A) and Crude Oil (CL) and Heating Oil (HO) contracts (4B) distributed over a 24-hour period (Eastern Time). Figure 4A shows the normalized portion of volume traded in these contracts is highest during US equities trading (9:30-16:00 ET), with 6A trading relatively more outside those hours than ES. Figure 4B shows the raw volume of Crude Oil and Heating Oil, rather than normalized volume, illustrating how liquidity varies dramatically across contracts and during a 24-hour period. Both contracts trade heavily during US equities hours, but Crude Oil trades somewhat heavily outside those hours as well.

Poor accommodation of futures market structure leads to increased costs

The issues detailed above are part of a long list of trading challenges associated with the unique market structure of futures and the idiosyncratic behavior of individual contracts. We recommend asking your execution algorithm providers how they tailor execution strategies to futures and how their algorithms dynamically adjust to each contract, market, time zone, queue-length, and exchange matching rules, among other important adjustments. 

At BestEx Research, our futures algorithms are designed exclusively for futures, with connectivity at exchanges around the world and more coming online each month. The four pillars of cost optimization—volume, volatility, spread, and depth—drive spread costs, adverse selection costs, and market impact costs and must be considered in conjunction with the market structure of each asset class and market microstructure of each product individually. BestEx Research algorithms are tested against our high-speed backtesting platform that allows realistic simulation of each exchange’s specific order matching rules. We’d love an opportunity to introduce you to our trading tools and show you what’s under the hood. 

‍

End notes

¹ The tradeoff between earning the spread and avoiding adverse selection for optimal placement of passive limit orders is an important topic that is always front of mind in our algorithm design. There is more to come on this topic from us in the near future.

² Read our thoughts about equity tick size requirements in our letter to the SEC regarding proposed changes to RegNMS.

‍

‍

At BestEx Research, your trading costs keep us up at night. We know from experience that systematic, quantitative decision-making around algorithm design contributes to globally optimal execution and results in significantly reduced execution costs. Reach out to us with questions at research@bestexresearch.com or learn more about us at bestexresearch.com.

This research paper reflects the views and opinions of BestEx Research Group LLC. It does not constitute legal, tax, investment, financial, or other professional advice. Nothing contained herein constitutes a solicitation, recommendation, endorsement, or offer to buy or sell securities, futures, or other financial instruments or to engage in financial strategies which may include algorithms. This material may not be a comprehensive or complete statement of the matters discussed herein. Nothing in this paper is a guarantee or assurance that any particular algorithmic solution fits you, or that you will benefit from it. You should consider whether our research is suitable for your particular circumstances and needs and, if appropriate, seek professional advice. 

©2022 BestEx Research. All rights reserved.

‍