Commodity future carry

The category group contains daily indicators of commodity futures’ carry, based on the price ratio between the first and second front future contracts and specific rolling rules. For commodities with seasonal supply and demand fluctuations and significant storage costs, this carry has a seasonal pattern during the year and seasonally-adjusted carry indicators are estimated.

Nominal commodity future carry

Ticker : COCRY_NSA / COCRY_VT10

Label : Nominal commodity future carry: % annualized / % annualized for 10% volatility target.

Definition : Estimated carry on continuously rolling futures, based on difference between front and back contracts: % annualized of notional of the contract / % annualized of risk capital on position scaled to 10% (annualized) volatility target.

Notes :

• The nominal carry is equivalent to the return earned by going long in the front contract and going short in the back contract for an unchanged futures curve. The carry is annualised using the ratio of total business days in a year to the difference in the days to maturity between the two contracts. Thus, the carry metric accounts for differences in the number of trading days between front and second contracts.

• For volatility-targeted carry, positions are scaled to a 10% vol target based on historic standard deviation of the commodity future returns for an exponential moving average with a half-life of 11 days. Positions are rebalanced at the end of each month.

• We construct a continuously rolling future series directly from the individual contracts that are part of the regular trading cycle for that commodity. The return calculation assumes that the future position is rolled (from front to second) on the first day of the month when the old front contract becomes deliverable. Not all commodities have monthly contracts.

• For information on the components comprising each commodity group, see Appendix 1 .

Nominal seasonally-adjusted commodity future carry

Ticker : COCRY_SA / COCRY_SAVT10

Label : Nominal seasonally-adjusted commodity future carry: % annualized / % annualized for 10% volatility target.

Definition : Seasonally-adjusted carry estimated from nominal unadjusted carry, based on point-in-time estimates of seasonal factors: annualized of notional of the contract / % annualized of risk capital on position scaled to 10% (annualized) volatility target.

Notes :

• The purpose of seasonal adjustment is to remove regular seasonal price differences from carry calculations, because those do not plausibly reflecte premia, such as those arising from convenience yields and hedging pressure.

• JPMaQS applies additive seasonal adjustment to nominal carry at each point in time based on the concurrently available vintage. The seasonal adjustment uses the U.S. Census Bureau’s X-13 seasonal adjustment tools.

• In order to estimate seasonal factors, nominal carry is downsampled from trading daily to monthly frequency. For all the months in the vintage bar the last one, we use the average monthly value of the carry as basis of seasonal adjustment. Given that JPMaQS carry metrics always assume contract roll at the end of the month the settlement date of the underlying contract is equal across the month and clearly identified. The seasonal component that is estimated for each point in time is then subtracted from the daily carry values that are based on future prices. Although it is applied daily, the seasonal factor only changes monthly in accordance with changes ins settlement dates.

• For commodities whose settlement dates do not change regularly at monthly frequency but remain unchanged for a few months we still estimate monthly carry adjustment factors. Although the season of the settlement dates does not always change on these occasions, the seasonal factor may still depend on the time distance to the settlement date, affecting marketing uncertianty about demand and supply conditions.

• JPMaQS requires a minimum of 42 months for out-of sample seasonal adjustment. This implies that the initial 3 years and half of seasonally adjusted carry are estimated in sample and have a lower grading.

Real commodity future carry

Ticker : COCRR_NSA / COCRR_VT10

Label : Real commodity future carry: % annualized / % annualized for 10% volatility target.

Definition : Estimated real carry on continuously rolling futures, based on difference between front and back contracts: % annualized of notional of the contract / % annualized of risk capital on position scaled to 10% (annualized) volatility target.

Notes :

• Real carry means that the conventional (nominal) carry is increased by expected U.S. inflation over the period between first and second contract. If inflation is expected to be positive between the two dates, the real (inflation-adjusted) difference between the first and second contract is higher than the nominal one.

• The inflation expectation proxy is the 1-year ahead estimated inflation expectation according to Macrosynergy methodology. See notes on category tickers INFE1Y_JA on the page “Inflation expectations (Macrosynergy methodology)”.

• See the important notes on ‘nominal commodity future carry’ above.

Real seasonally-adjusted commodity future carry

Ticker : COCRR_SA / COCRR_SAVT10

Label : Real seasonally-adjusted commodity future carry: % annualized / % annualized for 10% volatility target.

Definition : Estimated real carry on continuously rolling futures, based on the seasonally-adjusted difference between front and back contracts: % annualized of notional of the contract / % annualized of risk capital on position scaled to 10% (annualized) volatility target.

Notes :

• The purpose of seasonal adjustment is to remove regular seasonal price differences from carry calculations, because those do not plausibly reflect premia, such as those arising from convenience yields and hedging pressure.

• We apply the seasonal and inflation adjustment procedures described above.

Imports

Only the standard Python data science packages and the specialized macrosynergy package are needed.

import os
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import math

import json
import yaml

import macrosynergy.management as msm
import macrosynergy.panel as msp
import macrosynergy.signal as mss
import macrosynergy.pnl as msn


from macrosynergy.download import JPMaQSDownload

from timeit import default_timer as timer
from datetime import timedelta, date, datetime

import warnings

warnings.simplefilter("ignore")

The JPMaQS indicators we consider are downloaded using the J.P. Morgan Dataquery API interface within the macrosynergy package. This is done by specifying ticker strings , formed by appending an indicator category code <category> to a currency area code <cross_section> . These constitute the main part of a full quantamental indicator ticker, taking the form DB(JPMAQS,<cross_section>_<category>,<info>) , where <info> denotes the time series of information for the given cross-section and category. The following types of information are available:

• value giving the latest available values for the indicator

• eop_lag referring to days elapsed since the end of the observation period

• mop_lag referring to the number of days elapsed since the mean observation period

• grade denoting a grade of the observation, giving a metric of real time information quality.

After instantiating the JPMaQSDownload class within the macrosynergy.download module, one can use the download(tickers,start_date,metrics) method to easily download the necessary data, where tickers is an array of ticker strings, start_date is the first collection date to be considered and metrics is an array comprising the times series information to be downloaded.

cids_nfm = ["GLD", "SIV", "PAL", "PLT"]
cids_fme = ["ALM", "CPR", "LED", "NIC", "TIN", "ZNC"]
cids_ene = ["BRT", "WTI", "NGS", "GSO", "HOL"]
cids_sta = ["COR", "WHT", "SOY", "CTN"]
cids_liv = ["CAT", "HOG"]
cids_mis = ["CFE", "SGR", "NJO", "CLB"]

cids = cids_nfm + cids_fme + cids_ene + cids_sta + cids_liv + cids_mis
cids_ns = cids_nfm + cids_fme + ["BRT", "WTI", "CFE"]  # presumed non-seasonal commodities
cids_s = list(set(cids) - set(cids_ns + ["NGS"]))  # presumed seasonal commodities

main = ["COCRY_NSA", "COCRY_SA", "COCRY_VT10", "COCRY_SAVT10", 
        "COCRR_NSA", "COCRR_VT10", "COCRR_SA", "COCRR_SAVT10"]
econ = []
mark = ["COXR_NSA", "COXR_VT10"]

xcats = main + econ + mark

# Download series from J.P. Morgan DataQuery by tickers

start_date = "1990-01-01"
tickers = [cid + "_" + xcat for cid in cids for xcat in xcats]
print(f"Maximum number of tickers is {len(tickers)}")

# Retrieve credentials

client_id: str = os.getenv("DQ_CLIENT_ID")
client_secret: str = os.getenv("DQ_CLIENT_SECRET")

# Download from DataQuery

with JPMaQSDownload(client_id=client_id, client_secret=client_secret) as downloader:
    start = timer()
    df = downloader.download(
        tickers=tickers,
        start_date=start_date,
        metrics=["value", "eop_lag", "mop_lag", "grading"],
        suppress_warning=True,
        show_progress=True,
    )
    end = timer()

dfd = df

print("Download time from DQ: " + str(timedelta(seconds=end - start)))

Maximum number of tickers is 250
Downloading data from JPMaQS.
Timestamp UTC:  2023-11-08 12:13:30
Connection successful!
Number of expressions requested: 1000

Requesting data: 100%|██████████| 50/50 [00:16<00:00,  3.11it/s]
Downloading data: 100%|██████████| 50/50 [00:15<00:00,  3.20it/s]

Download time from DQ: 0:00:52.046036

Availability

cids_exp = cids
msm.missing_in_df(dfd, xcats=main, cids=cids_exp)

Missing xcats across df:  []
Missing cids for COCRR_NSA:  []
Missing cids for COCRR_SA:  []
Missing cids for COCRR_SAVT10:  []
Missing cids for COCRR_VT10:  []
Missing cids for COCRY_NSA:  []
Missing cids for COCRY_SA:  []
Missing cids for COCRY_SAVT10:  []
Missing cids for COCRY_VT10:  []

Commodity carry data mostly date back to the mid-1990s. Real carry data is only available from 2000 onwards.

For the explanation of currency symbols, which are related to currency areas or countries for which categories are available, please view Appendix 1 .

xcatx = main
cidx = cids_exp

dfx = msm.reduce_df(dfd, xcats=xcatx, cids=cidx)
dfs = msm.check_startyears(
    dfx,
)
msm.visual_paneldates(dfs, size=(20, 5))

print("Last updated:", date.today())

Last updated: 2023-11-08

plot = msm.check_availability(
    dfd, xcats=main, cids=cids_exp, start_size=(20, 5), start_years=False
)

plot = msp.heatmap_grades(
    dfd,
    xcats=main,
    cids=cids_exp,
    size=(19, 5),
    title=f"Average vintage grades from {start_date} onwards",
)

xcatx = main
cidx = cids_exp
msp.view_ranges(
    dfd,
    xcats=xcatx,
    cids=cidx,
    val="eop_lag",
    title="End of observation period lags (ranges of time elapsed since end of observation period in days)",
    start="2000-01-01",
    kind="box",
    size=(16, 4),
)
msp.view_ranges(
    dfd,
    xcats=xcatx,
    cids=cidx,
    val="mop_lag",
    title="Median of observation period lags (ranges of time elapsed since middle of observation period in days)",
    start="2000-01-01",
    kind="box",
    size=(16, 4),
)

History

Nominal commodity carry

Unlike financial assets, commodities have significant storage costs. This means that intertemporal substitution can be expensive and dependent on storage capacity. All other things equal, the costlier the storage, the greater is the segmentation and the variability of carry. This has consequences for carry dynamics.

• Commodities with seasonal supply or demand fluctuations often display a seasonal pattern in recorded carry. This can be seen clearly in livestock and gasoline contracts.

• There can be huge outliers in carry if storage limits are reached. These can be seen in many commodities.

• Commodities with relatively low storage costs, such as gold and silver, tend to have the most stable carry since substitution between futures dates is inexpensive.

xcatx = ["COCRY_NSA", "COCRY_SA"]
cidx = cids_exp

msp.view_ranges(
    dfd,
    xcats=xcatx,
    cids=cidx,
    sort_cids_by="mean",
    start="2000-01-01",
    title="Means and standard deviations of nominal commodity carry, since 2000",
    kind="bar",
    size=(16, 8),
    xcat_labels=['Outright', 'Seasonally adjusted']
)

xcatx = ["COCRY_NSA", "COCRY_SA"]
cidx = cids_exp

msp.view_timelines(
    dfd,
    xcats=xcatx,
    cids=cidx,
    start="2000-01-01",
    title="Nominal commodity carry, outright and seasonally adjusted, since 2000",
    title_adj=1.03,
    title_xadj=0.52,
    title_fontsize=27,
    cumsum=False,
    ncol=4,
    same_y=False,
    size=(12, 7),
    aspect=1.5,
    all_xticks=True,
    xcat_labels=['Outright', 'Seasonally adjusted'],
)

Standard seasonal adjustment of carry does not remove all seasonal influences. It typically removes or at least mitigates the directional bias of carry provided seasonal influences of price differences across delivery dates do not have strong trends of their own. However, seasonal adjustment cannot prevent seasonal increases in volatility and absolute carry values, which may arise from weather and harvest conditions.

xcatx = ["COCRY_NSA", "COCRY_SA"]
cidx = ["CAT", "GSO", "SGR", "WHT"]

msp.view_timelines(
    dfd,
    xcats=xcatx,
    cids=cidx,
    start="2000-01-01",
    title="Carry of selected seasonal commodities, outright and seasonally adjusted",
    title_adj=1.03,
    title_xadj=0.52,
    title_fontsize=15,
    cumsum=False,
    ncol=2,
    same_y=False,
    size=(12, 7),
    aspect=1.5,
    all_xticks=True,
    xcat_labels=['Outright', 'Seasonally adjusted'],
)

Overall carry is not strongly correlated across commodities. There is groupwise positive correlation in the metals and energy groups.

msp.correl_matrix(
    dfd,
    xcats="COCRY_SA",
    cids=cids_exp,
    title="Cross-sectional correlations of nominal commodity carry, since 2000",
    cluster=True,
    size=(20, 14),
)

Real commodity carry

Real carry is equal to nominal carry plus expected U.S. inflation. Thus, it is on average a little bit higher, but otherwise shows a very similar dynamic.

xcatx = ["COCRR_SA", "COCRR_NSA"]
cidx = cids_exp

msp.view_ranges(
    dfd,
    xcats=xcatx,
    cids=cidx,
    sort_cids_by="mean",
    start="2000-01-01",
    title="Means and standard deviations of commodity carry, since 2000",
    xcat_labels=["Nominal", "Real"],
    kind="bar",
    size=(16, 8),
)

xcatx = ["COCRR_NSA", "COCRR_SA"]
cidx = cids_s

msp.view_timelines(
    dfd,
    xcats=xcatx,
    cids=cidx,
    start="2000-01-01",
    title="Real carry of seasonal commodities, outright and with seasonal adjustment",
    title_adj=1.03,
    title_fontsize=27,
    title_xadj=0.47,
    legend_fontsize=17,
    cumsum=False,
    ncol=3,
    same_y=False,
    size=(12, 7),
    aspect=1.7,
    all_xticks=True,
    xcat_labels=['Outright', 'Seasonally adjusted']
)

Vol-targeted commodity carry

Even when positions are adjusted for return volatility, differences in carry variations are vast. Lean hogs, cattle, gasoline and soy have posted the widest standard deviations. Seasonality significantly contributed to heteroscedasticity across contracts’ carry.

xcatx = ["COCRY_VT10", "COCRY_SAVT10"]
cidx = cids_exp

msp.view_ranges(
    dfd,
    xcats=xcatx,
    cids=cidx,
    sort_cids_by="mean",
    start="2000-01-01",
    kind="bar",
    title="Means and standard deviations of nominal commodity carry, at 10% vol-target, since 2000",
    size=(16, 8),
    xcat_labels=['Outright', 'Seasonally adjusted']
)

xcatx = ["COCRY_NSA", "COCRY_VT10"]
cidx = cids_exp

msp.view_timelines(
    dfd,
    xcats=xcatx,
    cids=cidx,
    start="2000-01-01",
    title="Nominal commodity carry, without and with volatility targeting",
    title_adj=1.03,
    title_fontsize=27,
    title_xadj=0.47,
    legend_fontsize=17,
    cumsum=False,
    ncol=4,
    same_y=False,
    size=(12, 7),
    aspect=1.7,
    all_xticks=True,
    xcat_labels=['Outright', '10% vol-target']
)

Importance

Research links

“Across assets, carry is defined as return for unchanged prices and is calculated based on the difference between spot and futures prices (view post here). Unlike other markets, commodity futures curves are segmented by obstacles to intertemporal arbitrage. The costlier the storage, the greater is the segmentation and the variability of carry. The segmented commodity curve is shaped prominently by four factors: [1] funding and storage costs, [2] expected supply-demand imbalances, [3] convenience yields and [4] hedging pressure. The latter two factors give rise to premia that can be received by financial investors. In order to focus on premia, one must strip out apparent supply-demand effects, such as seasonal fluctuations and storage costs. After adjustment both direction and size of commodity carry should be valid, if imprecise, indicators of risk premia. Data for 2000-2018 show clear a persistent positive correlation of the carry with future returns.” Macrosynergy

“Two key aspects of commodity pricing are (1) a rational pricing model based on the present value of future convenience yields of physical commodity holdings, and (2) the activity of financial investors in form of rational short-term trading and contrarian trading. Since convenience yields are related to the scarcity of a commodity and the value of inventories for production and consumption they provide the fundamental anchor of prices. The trading aspect reflects the growing “financialization” of commodity markets. The influence of both fundamentals and trading is backed by empirical evidence. One implication is that adjusted spreads between spot and futures prices, which partly indicate unsustainably high or low convenience yields, are valid trading signals.” Macrosynergy

“Seasonal fluctuations are evident for many commodity prices. However, their exact size can be quite uncertain. Hence, seasons affect commodity futures curves in two ways. First, they bias the expected futures price of a specific expiry month relative that of other months. Second, their uncertainty is an independent source of risk that affects the overall risk premia priced into the curve. Integrating seasonal factor uncertainty into an affine (linear) term structure model of commodity futures allows more realistic and granular estimates of various risk premia or ‘cost-of-carry factors’. This can serve as basis for investors to decide whether to receive or pay the risk premia implied in the future curve.” Macrosynergy

Empirical clues

Real commodity carry, when adjusted for seasonal effects and intertemporal segmentation, often reflects a convenience yield paid to efficient financial investors by producers and/or consumers of a commodity. For commodities with little or no seasonal fluctuations (precious and ferrous metals, coffee and international crude), there has been a significant positive relation between carry and subsequent monthly and quartery returns.

cidx = cids_ns

cr = msp.CategoryRelations(
    dfd,
    xcats=["COCRR_NSA", "COXR_NSA"],
    cids=cidx,
    freq="Q",
    lag=1,
    xcat_aggs=["mean", "sum"],
    fwin=1,
    start="2000-01-01",
)
cr.reg_scatter(
    title="Real commodity carry and subsequent future returns (non-seasonal products)",
    labels=False,
    coef_box="lower right",
    xlab="Real commodity carry, % annualized, quarterly means",
    ylab="Subsequent quarterly commodity future returns",
    prob_est="map",
)

For seasonal commodities, i.e. , agricultural commodities, livestock and regional energy contracts, the relation there has not been a signficant predictive relation between (unadjusted) future curry and subsequent returns.

cidx = cids_s

cr = msp.CategoryRelations(
    dfd,
    xcats=["COCRR_NSA", "COXR_NSA"],
    cids=cidx,
    freq="Q",
    lag=1,
    xcat_aggs=["mean", "sum"],
    fwin=1,
    start="2000-01-01",
)
cr.reg_scatter(
    title="Seasonal commodities: unadjusted commodity carry and subsequent future returns",
    labels=False,
    coef_box="lower right",
    xlab="Real commodity carry, % annualized, quarterly means",
    ylab="Subsequent quarterly commodity future returns",
    prob_est="map",
)

However, using seasonally-adjusted carry, there has a stronger predictive relation between carry and return, even if it is not statistically significant. Lack of significance in this simple setting probably reflects that across the panel carry-return relations are very different and that without volatility adjustments the most volatile contracts dominate the results.

cidx = cids_s

cr = msp.CategoryRelations(
    dfd,
    xcats=["COCRR_SA", "COXR_NSA"],
    cids=cidx,
    freq="Q",
    lag=1,
    xcat_aggs=["mean", "sum"],
    fwin=1,
    start="2000-01-01",
)
cr.reg_scatter(
    title="Seasonal commodities: seasonally-adjusted commodity carry and subsequent future returns",
    labels=False,
    coef_box="lower right",
    xlab="Commodity carry, % annualized, quarterly means",
    ylab="Subsequent quarterly commodity future returns",
    prob_est="map",
)

The combination of seasonal adjustment and regular volatility adjustment reveals a clear and positive relation between carry and returns even for the seasonal commodities at a monthly or quarterly frequency.

cidx = cids_s

cr = msp.CategoryRelations(
    dfd,
    xcats=["COCRR_SAVT10", "COXR_VT10"],
    cids=cidx,
    freq="Q",
    lag=1,
    xcat_aggs=["mean", "sum"],
    fwin=1,
    start="2000-01-01",
)
cr.reg_scatter(
    title="Seasonal commodities: seasonally-adjusted vol-targeted commodity carry and subsequent future returns",
    labels=False,
    coef_box="lower right",
    xlab="Real commodity carry, seasonally adjusted and vol-targeted, % annualized, quarterly means",
    ylab="Subsequent vol-targeted (10%) quarterly commodity returns",
    prob_est="map",
)

Based on seasonally adjusted and volatility-target carry and volatility targeted returns one can detect a highly significant predictive carry-return relation at a monthly or quarterly frequency.

cidx = cids_s + cids_ns

cr = msp.CategoryRelations(
    dfd,
    xcats=["COCRR_SAVT10", "COXR_VT10"],
    cids=cidx,
    freq="Q",
    lag=1,
    xcat_aggs=["mean", "sum"],
    fwin=1,
    start="2000-01-01",
)
cr.reg_scatter(
    title="All commodities: seasonally-adjusted vol-targeted commodity carry and subsequent future returns",
    labels=False,
    coef_box="lower right",
    xlab="Commodity carry, % annualized, quarterly means",
    ylab="Subsequent quarterly commodity future returns",
    prob_est="map",
)

Appendices

Appendix 1: Commodity group definitions and symbols

The commodity groups considered are: energy, base metals, precious metals, agricultural commodities and livestock.

• The energy commodity group contains:

  • BRT : ICE Brent crude

  • WTI : NYMEX WTI light crude

  • NGS : NYMEX natural gas, Henry Hub

  • GSO : NYMEX RBOB Gasoline

  • HOL : NYMEX Heating oil, New York Harbor ULSD

• The base metals group contains:

  • ALM : London Metal Exchange aluminium

  • CPR : Comex copper

  • LED : London Metal Exchange Lead

  • NIC : London Metal Exchange Nickel

  • TIN : London Metal Exchange Tin

  • ZNC : London Metal Exchange Zinc

• The precious metals group contains:

  • GLD : COMEX gold 100 Ounce

  • SIV : COMEX silver 5000 Ounce

  • PAL : NYMEX palladium

  • PLT : NYMEX platinum

• The agricultural commodity group contains:

  • COR : Chicago Board of Trade corn composite

  • WHT : Chicago Board of Trade wheat composite

  • SOY : Chicago Board of Trade soybeans composite

  • CTN : NYBOT / ICE cotton #2

  • CFE : NYBOT / ICE coffee ‘C’ Arabica

  • SGR : NYBOT / ICE raw cane sugar #11

  • NJO : NYBOT / NYCE FCOJ frozen orange juice concentrate

  • CLB : Chicago Mercantile Exchange random length lumber

• The (U.S.) livestock commodity group contains:

  • CAT : Chicago Mercantile Exchange live cattle composite

  • HOG : Chicago Mercantile Exchange lean hogs composite