Commodity future carry #
The category group contains daily indicators of commodity futures’ carry, based on the price ratio between the first and second front future contracts and specific rolling rules. For commodities with seasonal supply and demand fluctuations and significant storage costs, this carry has a seasonal pattern during the year and seasonally-adjusted carry indicators are estimated.
Nominal commodity future carry #
Ticker : COCRY_NSA / COCRY_VT10
Label : Nominal commodity future carry: % annualized / % annualized for 10% volatility target.
Definition : Estimated carry on continuously rolling futures, based on difference between front and back contracts: % annualized of notional of the contract / % annualized of risk capital on position scaled to 10% (annualized) volatility target.
Notes :
-
The nominal carry is equivalent to the return earned by going long in the front contract and going short in the back contract for an unchanged futures curve. The carry is annualised using the ratio of total business days in a year to the difference in the days to maturity between the two contracts. Thus, the carry metric accounts for differences in the number of trading days between front and second contracts.
-
For volatility-targeted carry, positions are scaled to a 10% vol target based on historic standard deviation of the commodity future returns for an exponential moving average with a half-life of 11 days. Positions are rebalanced at the end of each month.
-
We construct a continuously rolling future series directly from the individual contracts that are part of the regular trading cycle for that commodity. The return calculation assumes that the future position is rolled (from front to second) on the first day of the month when the old front contract becomes deliverable. Not all commodities have monthly contracts.
-
For information on the components comprising each commodity group, see Appendix 1 .
Nominal seasonally-adjusted commodity future carry #
Ticker : COCRY_SA / COCRY_SAVT10
Label : Nominal seasonally-adjusted commodity future carry: % annualized / % annualized for 10% volatility target.
Definition : Seasonally-adjusted carry estimated from nominal unadjusted carry, based on point-in-time estimates of seasonal factors: annualized of notional of the contract / % annualized of risk capital on position scaled to 10% (annualized) volatility target.
Notes :
-
The purpose of seasonal adjustment is to remove regular seasonal price differences from carry calculations, because those do not plausibly reflecte premia, such as those arising from convenience yields and hedging pressure.
-
JPMaQS applies additive seasonal adjustment to nominal carry at each point in time based on the concurrently available vintage. The seasonal adjustment uses the U.S. Census Bureau’s X-13 seasonal adjustment tools.
-
In order to estimate seasonal factors, nominal carry is downsampled from trading daily to monthly frequency. For all the months in the vintage bar the last one, we use the average monthly value of the carry as basis of seasonal adjustment. Given that JPMaQS carry metrics always assume contract roll at the end of the month the settlement date of the underlying contract is equal across the month and clearly identified. The seasonal component that is estimated for each point in time is then subtracted from the daily carry values that are based on future prices. Although it is applied daily, the seasonal factor only changes monthly in accordance with changes ins settlement dates.
-
For commodities whose settlement dates do not change regularly at monthly frequency but remain unchanged for a few months we still estimate monthly carry adjustment factors. Although the season of the settlement dates does not always change on these occasions, the seasonal factor may still depend on the time distance to the settlement date, affecting marketing uncertianty about demand and supply conditions.
-
JPMaQS requires a minimum of 42 months for out-of sample seasonal adjustment. This implies that the initial 3 years and half of seasonally adjusted carry are estimated in sample and have a lower grading.
Real commodity future carry #
Ticker : COCRR_NSA / COCRR_VT10
Label : Real commodity future carry: % annualized / % annualized for 10% volatility target.
Definition : Estimated real carry on continuously rolling futures, based on difference between front and back contracts: % annualized of notional of the contract / % annualized of risk capital on position scaled to 10% (annualized) volatility target.
Notes :
-
Real carry means that the conventional (nominal) carry is increased by expected U.S. inflation over the period between first and second contract. If inflation is expected to be positive between the two dates, the real (inflation-adjusted) difference between the first and second contract is higher than the nominal one.
-
The inflation expectation proxy is the 1-year ahead estimated inflation expectation according to Macrosynergy methodology. See notes on category tickers
INFE1Y_JA
on the page “Inflation expectations (Macrosynergy methodology)”. -
See the important notes on ‘nominal commodity future carry’ above.
Real seasonally-adjusted commodity future carry #
Ticker : COCRR_SA / COCRR_SAVT10
Label : Real seasonally-adjusted commodity future carry: % annualized / % annualized for 10% volatility target.
Definition : Estimated real carry on continuously rolling futures, based on the seasonally-adjusted difference between front and back contracts: % annualized of notional of the contract / % annualized of risk capital on position scaled to 10% (annualized) volatility target.
Notes :
-
The purpose of seasonal adjustment is to remove regular seasonal price differences from carry calculations, because those do not plausibly reflect premia, such as those arising from convenience yields and hedging pressure.
-
We apply the seasonal and inflation adjustment procedures described above.
Imports #
Only the standard Python data science packages and the specialized
macrosynergy
package are needed.
import os
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import math
import json
import yaml
import macrosynergy.management as msm
import macrosynergy.panel as msp
import macrosynergy.signal as mss
import macrosynergy.pnl as msn
from macrosynergy.download import JPMaQSDownload
from timeit import default_timer as timer
from datetime import timedelta, date, datetime
import warnings
warnings.simplefilter("ignore")
The
JPMaQS
indicators we consider are downloaded using the J.P. Morgan Dataquery API interface within the
macrosynergy
package. This is done by specifying
ticker strings
, formed by appending an indicator category code
<category>
to a currency area code
<cross_section>
. These constitute the main part of a full quantamental indicator ticker, taking the form
DB(JPMAQS,<cross_section>_<category>,<info>)
, where
<info>
denotes the time series of information for the given cross-section and category. The following types of information are available:
-
value
giving the latest available values for the indicator -
eop_lag
referring to days elapsed since the end of the observation period -
mop_lag
referring to the number of days elapsed since the mean observation period -
grade
denoting a grade of the observation, giving a metric of real time information quality.
After instantiating the
JPMaQSDownload
class within the
macrosynergy.download
module, one can use the
download(tickers,start_date,metrics)
method to easily download the necessary data, where
tickers
is an array of ticker strings,
start_date
is the first collection date to be considered and
metrics
is an array comprising the times series information to be downloaded.
cids_nfm = ["GLD", "SIV", "PAL", "PLT"]
cids_fme = ["ALM", "CPR", "LED", "NIC", "TIN", "ZNC"]
cids_ene = ["BRT", "WTI", "NGS", "GSO", "HOL"]
cids_sta = ["COR", "WHT", "SOY", "CTN"]
cids_liv = ["CAT", "HOG"]
cids_mis = ["CFE", "SGR", "NJO", "CLB"]
cids = cids_nfm + cids_fme + cids_ene + cids_sta + cids_liv + cids_mis
cids_ns = cids_nfm + cids_fme + ["BRT", "WTI", "CFE"] # presumed non-seasonal commodities
cids_s = list(set(cids) - set(cids_ns + ["NGS"])) # presumed seasonal commodities
main = ["COCRY_NSA", "COCRY_SA", "COCRY_VT10", "COCRY_SAVT10",
"COCRR_NSA", "COCRR_VT10", "COCRR_SA", "COCRR_SAVT10"]
econ = []
mark = ["COXR_NSA", "COXR_VT10"]
xcats = main + econ + mark
# Download series from J.P. Morgan DataQuery by tickers
start_date = "1990-01-01"
tickers = [cid + "_" + xcat for cid in cids for xcat in xcats]
print(f"Maximum number of tickers is {len(tickers)}")
# Retrieve credentials
client_id: str = os.getenv("DQ_CLIENT_ID")
client_secret: str = os.getenv("DQ_CLIENT_SECRET")
# Download from DataQuery
with JPMaQSDownload(client_id=client_id, client_secret=client_secret) as downloader:
start = timer()
df = downloader.download(
tickers=tickers,
start_date=start_date,
metrics=["value", "eop_lag", "mop_lag", "grading"],
suppress_warning=True,
show_progress=True,
)
end = timer()
dfd = df
print("Download time from DQ: " + str(timedelta(seconds=end - start)))
Maximum number of tickers is 250
Downloading data from JPMaQS.
Timestamp UTC: 2023-11-08 12:13:30
Connection successful!
Number of expressions requested: 1000
Requesting data: 100%|██████████| 50/50 [00:16<00:00, 3.11it/s]
Downloading data: 100%|██████████| 50/50 [00:15<00:00, 3.20it/s]
Download time from DQ: 0:00:52.046036
Availability #
cids_exp = cids
msm.missing_in_df(dfd, xcats=main, cids=cids_exp)
Missing xcats across df: []
Missing cids for COCRR_NSA: []
Missing cids for COCRR_SA: []
Missing cids for COCRR_SAVT10: []
Missing cids for COCRR_VT10: []
Missing cids for COCRY_NSA: []
Missing cids for COCRY_SA: []
Missing cids for COCRY_SAVT10: []
Missing cids for COCRY_VT10: []
Commodity carry data mostly date back to the mid-1990s. Real carry data is only available from 2000 onwards.
For the explanation of currency symbols, which are related to currency areas or countries for which categories are available, please view Appendix 1 .
xcatx = main
cidx = cids_exp
dfx = msm.reduce_df(dfd, xcats=xcatx, cids=cidx)
dfs = msm.check_startyears(
dfx,
)
msm.visual_paneldates(dfs, size=(20, 5))
print("Last updated:", date.today())
Last updated: 2023-11-08
plot = msm.check_availability(
dfd, xcats=main, cids=cids_exp, start_size=(20, 5), start_years=False
)
plot = msp.heatmap_grades(
dfd,
xcats=main,
cids=cids_exp,
size=(19, 5),
title=f"Average vintage grades from {start_date} onwards",
)
xcatx = main
cidx = cids_exp
msp.view_ranges(
dfd,
xcats=xcatx,
cids=cidx,
val="eop_lag",
title="End of observation period lags (ranges of time elapsed since end of observation period in days)",
start="2000-01-01",
kind="box",
size=(16, 4),
)
msp.view_ranges(
dfd,
xcats=xcatx,
cids=cidx,
val="mop_lag",
title="Median of observation period lags (ranges of time elapsed since middle of observation period in days)",
start="2000-01-01",
kind="box",
size=(16, 4),
)
History #
Nominal commodity carry #
Unlike financial assets, commodities have significant storage costs. This means that intertemporal substitution can be expensive and dependent on storage capacity. All other things equal, the costlier the storage, the greater is the segmentation and the variability of carry. This has consequences for carry dynamics.
-
Commodities with seasonal supply or demand fluctuations often display a seasonal pattern in recorded carry. This can be seen clearly in livestock and gasoline contracts.
-
There can be huge outliers in carry if storage limits are reached. These can be seen in many commodities.
-
Commodities with relatively low storage costs, such as gold and silver, tend to have the most stable carry since substitution between futures dates is inexpensive.
xcatx = ["COCRY_NSA", "COCRY_SA"]
cidx = cids_exp
msp.view_ranges(
dfd,
xcats=xcatx,
cids=cidx,
sort_cids_by="mean",
start="2000-01-01",
title="Means and standard deviations of nominal commodity carry, since 2000",
kind="bar",
size=(16, 8),
xcat_labels=['Outright', 'Seasonally adjusted']
)
xcatx = ["COCRY_NSA", "COCRY_SA"]
cidx = cids_exp
msp.view_timelines(
dfd,
xcats=xcatx,
cids=cidx,
start="2000-01-01",
title="Nominal commodity carry, outright and seasonally adjusted, since 2000",
title_adj=1.03,
title_xadj=0.52,
title_fontsize=27,
cumsum=False,
ncol=4,
same_y=False,
size=(12, 7),
aspect=1.5,
all_xticks=True,
xcat_labels=['Outright', 'Seasonally adjusted'],
)
Standard seasonal adjustment of carry does not remove all seasonal influences. It typically removes or at least mitigates the directional bias of carry provided seasonal influences of price differences across delivery dates do not have strong trends of their own. However, seasonal adjustment cannot prevent seasonal increases in volatility and absolute carry values, which may arise from weather and harvest conditions.
xcatx = ["COCRY_NSA", "COCRY_SA"]
cidx = ["CAT", "GSO", "SGR", "WHT"]
msp.view_timelines(
dfd,
xcats=xcatx,
cids=cidx,
start="2000-01-01",
title="Carry of selected seasonal commodities, outright and seasonally adjusted",
title_adj=1.03,
title_xadj=0.52,
title_fontsize=15,
cumsum=False,
ncol=2,
same_y=False,
size=(12, 7),
aspect=1.5,
all_xticks=True,
xcat_labels=['Outright', 'Seasonally adjusted'],
)
Overall carry is not strongly correlated across commodities. There is groupwise positive correlation in the metals and energy groups.
msp.correl_matrix(
dfd,
xcats="COCRY_SA",
cids=cids_exp,
title="Cross-sectional correlations of nominal commodity carry, since 2000",
cluster=True,
size=(20, 14),
)
Real commodity carry #
Real carry is equal to nominal carry plus expected U.S. inflation. Thus, it is on average a little bit higher, but otherwise shows a very similar dynamic.
xcatx = ["COCRR_SA", "COCRR_NSA"]
cidx = cids_exp
msp.view_ranges(
dfd,
xcats=xcatx,
cids=cidx,
sort_cids_by="mean",
start="2000-01-01",
title="Means and standard deviations of commodity carry, since 2000",
xcat_labels=["Nominal", "Real"],
kind="bar",
size=(16, 8),
)
xcatx = ["COCRR_NSA", "COCRR_SA"]
cidx = cids_s
msp.view_timelines(
dfd,
xcats=xcatx,
cids=cidx,
start="2000-01-01",
title="Real carry of seasonal commodities, outright and with seasonal adjustment",
title_adj=1.03,
title_fontsize=27,
title_xadj=0.47,
legend_fontsize=17,
cumsum=False,
ncol=3,
same_y=False,
size=(12, 7),
aspect=1.7,
all_xticks=True,
xcat_labels=['Outright', 'Seasonally adjusted']
)
Vol-targeted commodity carry #
Even when positions are adjusted for return volatility, differences in carry variations are vast. Lean hogs, cattle, gasoline and soy have posted the widest standard deviations. Seasonality significantly contributed to heteroscedasticity across contracts’ carry.
xcatx = ["COCRY_VT10", "COCRY_SAVT10"]
cidx = cids_exp
msp.view_ranges(
dfd,
xcats=xcatx,
cids=cidx,
sort_cids_by="mean",
start="2000-01-01",
kind="bar",
title="Means and standard deviations of nominal commodity carry, at 10% vol-target, since 2000",
size=(16, 8),
xcat_labels=['Outright', 'Seasonally adjusted']
)
xcatx = ["COCRY_NSA", "COCRY_VT10"]
cidx = cids_exp
msp.view_timelines(
dfd,
xcats=xcatx,
cids=cidx,
start="2000-01-01",
title="Nominal commodity carry, without and with volatility targeting",
title_adj=1.03,
title_fontsize=27,
title_xadj=0.47,
legend_fontsize=17,
cumsum=False,
ncol=4,
same_y=False,
size=(12, 7),
aspect=1.7,
all_xticks=True,
xcat_labels=['Outright', '10% vol-target']
)
Importance #
Research links #
“Across assets, carry is defined as return for unchanged prices and is calculated based on the difference between spot and futures prices (view post here). Unlike other markets, commodity futures curves are segmented by obstacles to intertemporal arbitrage. The costlier the storage, the greater is the segmentation and the variability of carry. The segmented commodity curve is shaped prominently by four factors: [1] funding and storage costs, [2] expected supply-demand imbalances, [3] convenience yields and [4] hedging pressure. The latter two factors give rise to premia that can be received by financial investors. In order to focus on premia, one must strip out apparent supply-demand effects, such as seasonal fluctuations and storage costs. After adjustment both direction and size of commodity carry should be valid, if imprecise, indicators of risk premia. Data for 2000-2018 show clear a persistent positive correlation of the carry with future returns.” Macrosynergy
“Two key aspects of commodity pricing are (1) a rational pricing model based on the present value of future convenience yields of physical commodity holdings, and (2) the activity of financial investors in form of rational short-term trading and contrarian trading. Since convenience yields are related to the scarcity of a commodity and the value of inventories for production and consumption they provide the fundamental anchor of prices. The trading aspect reflects the growing “financialization” of commodity markets. The influence of both fundamentals and trading is backed by empirical evidence. One implication is that adjusted spreads between spot and futures prices, which partly indicate unsustainably high or low convenience yields, are valid trading signals.” Macrosynergy
“Seasonal fluctuations are evident for many commodity prices. However, their exact size can be quite uncertain. Hence, seasons affect commodity futures curves in two ways. First, they bias the expected futures price of a specific expiry month relative that of other months. Second, their uncertainty is an independent source of risk that affects the overall risk premia priced into the curve. Integrating seasonal factor uncertainty into an affine (linear) term structure model of commodity futures allows more realistic and granular estimates of various risk premia or ‘cost-of-carry factors’. This can serve as basis for investors to decide whether to receive or pay the risk premia implied in the future curve.” Macrosynergy
Empirical clues #
Real commodity carry, when adjusted for seasonal effects and intertemporal segmentation, often reflects a convenience yield paid to efficient financial investors by producers and/or consumers of a commodity. For commodities with little or no seasonal fluctuations (precious and ferrous metals, coffee and international crude), there has been a significant positive relation between carry and subsequent monthly and quartery returns.
cidx = cids_ns
cr = msp.CategoryRelations(
dfd,
xcats=["COCRR_NSA", "COXR_NSA"],
cids=cidx,
freq="Q",
lag=1,
xcat_aggs=["mean", "sum"],
fwin=1,
start="2000-01-01",
)
cr.reg_scatter(
title="Real commodity carry and subsequent future returns (non-seasonal products)",
labels=False,
coef_box="lower right",
xlab="Real commodity carry, % annualized, quarterly means",
ylab="Subsequent quarterly commodity future returns",
prob_est="map",
)
For seasonal commodities, i.e. , agricultural commodities, livestock and regional energy contracts, the relation there has not been a signficant predictive relation between (unadjusted) future curry and subsequent returns.
cidx = cids_s
cr = msp.CategoryRelations(
dfd,
xcats=["COCRR_NSA", "COXR_NSA"],
cids=cidx,
freq="Q",
lag=1,
xcat_aggs=["mean", "sum"],
fwin=1,
start="2000-01-01",
)
cr.reg_scatter(
title="Seasonal commodities: unadjusted commodity carry and subsequent future returns",
labels=False,
coef_box="lower right",
xlab="Real commodity carry, % annualized, quarterly means",
ylab="Subsequent quarterly commodity future returns",
prob_est="map",
)
However, using seasonally-adjusted carry, there has a stronger predictive relation between carry and return, even if it is not statistically significant. Lack of significance in this simple setting probably reflects that across the panel carry-return relations are very different and that without volatility adjustments the most volatile contracts dominate the results.
cidx = cids_s
cr = msp.CategoryRelations(
dfd,
xcats=["COCRR_SA", "COXR_NSA"],
cids=cidx,
freq="Q",
lag=1,
xcat_aggs=["mean", "sum"],
fwin=1,
start="2000-01-01",
)
cr.reg_scatter(
title="Seasonal commodities: seasonally-adjusted commodity carry and subsequent future returns",
labels=False,
coef_box="lower right",
xlab="Commodity carry, % annualized, quarterly means",
ylab="Subsequent quarterly commodity future returns",
prob_est="map",
)
The combination of seasonal adjustment and regular volatility adjustment reveals a clear and positive relation between carry and returns even for the seasonal commodities at a monthly or quarterly frequency.
cidx = cids_s
cr = msp.CategoryRelations(
dfd,
xcats=["COCRR_SAVT10", "COXR_VT10"],
cids=cidx,
freq="Q",
lag=1,
xcat_aggs=["mean", "sum"],
fwin=1,
start="2000-01-01",
)
cr.reg_scatter(
title="Seasonal commodities: seasonally-adjusted vol-targeted commodity carry and subsequent future returns",
labels=False,
coef_box="lower right",
xlab="Real commodity carry, seasonally adjusted and vol-targeted, % annualized, quarterly means",
ylab="Subsequent vol-targeted (10%) quarterly commodity returns",
prob_est="map",
)
Based on seasonally adjusted and volatility-target carry and volatility targeted returns one can detect a highly significant predictive carry-return relation at a monthly or quarterly frequency.
cidx = cids_s + cids_ns
cr = msp.CategoryRelations(
dfd,
xcats=["COCRR_SAVT10", "COXR_VT10"],
cids=cidx,
freq="Q",
lag=1,
xcat_aggs=["mean", "sum"],
fwin=1,
start="2000-01-01",
)
cr.reg_scatter(
title="All commodities: seasonally-adjusted vol-targeted commodity carry and subsequent future returns",
labels=False,
coef_box="lower right",
xlab="Commodity carry, % annualized, quarterly means",
ylab="Subsequent quarterly commodity future returns",
prob_est="map",
)
Appendices #
Appendix 1: Commodity group definitions and symbols #
The commodity groups considered are: energy, base metals, precious metals, agricultural commodities and livestock.
-
The energy commodity group contains:
-
BRT : ICE Brent crude
-
WTI : NYMEX WTI light crude
-
NGS : NYMEX natural gas, Henry Hub
-
GSO : NYMEX RBOB Gasoline
-
HOL : NYMEX Heating oil, New York Harbor ULSD
-
-
The base metals group contains:
-
ALM : London Metal Exchange aluminium
-
CPR : Comex copper
-
LED : London Metal Exchange Lead
-
NIC : London Metal Exchange Nickel
-
TIN : London Metal Exchange Tin
-
ZNC : London Metal Exchange Zinc
-
-
The precious metals group contains:
-
GLD : COMEX gold 100 Ounce
-
SIV : COMEX silver 5000 Ounce
-
PAL : NYMEX palladium
-
PLT : NYMEX platinum
-
-
The agricultural commodity group contains:
-
COR : Chicago Board of Trade corn composite
-
WHT : Chicago Board of Trade wheat composite
-
SOY : Chicago Board of Trade soybeans composite
-
CTN : NYBOT / ICE cotton #2
-
CFE : NYBOT / ICE coffee ‘C’ Arabica
-
SGR : NYBOT / ICE raw cane sugar #11
-
NJO : NYBOT / NYCE FCOJ frozen orange juice concentrate
-
CLB : Chicago Mercantile Exchange random length lumber
-
-
The (U.S.) livestock commodity group contains:
-
CAT : Chicago Mercantile Exchange live cattle composite
-
HOG : Chicago Mercantile Exchange lean hogs composite
-