Testing Venn’s Analytics During the Coronavirus Crisis

The novel coronavirus has been spreading globally, impacting financial markets severely. Venn can help investors understand and manage the risks in their portfolios and investments by using several types of analysis, including Scenario Analysis and Venncast. 

Given we are in truly unprecedented times, we decided to test the accuracy of these two analyses by conducting in versus out of sample analysis using mutual fund investments and anonymized user master portfolios.¹

The results we’ll discuss below for these two types of analysis highlight:

• The extent to which these analyses are reliable for Venn’s subscribers, especially in times of notable market stress
• Venn’s design choices in constructing the methodologies behind these analyses
• The instability of factor correlations during the coronavirus crisis period

Scenario Analysis

What is Scenario Analysis, and how are subscribers applying it today?

Scenario Analysis was designed to help subscribers estimate how their portfolios and investments could react to certain market shocks that are intended to be realized on an approximately one-month-long time horizon. It targets shocks that are not particularly extreme, as it currently only allows subscribers to input a scenario shock that is between +/- two standard deviations of the scenario index’s average monthly return historically. 

Lately, according to our analytics data, subscribers have been running scenario analyses to test how potential market shocks from coronavirus could impact their portfolios or investments in the future.

Construction of the Scenario Analysis Accuracy Test

To test the accuracy of Scenario Analysis during this volatile time, we measured the sensitivity of various mutual fund investments and anonymized subscriber master portfolios to the S&P 500 Index.² We used returns data through January 31, 2020 for the investments and portfolios to understand their exposures to the factors in the Two Sigma Factor Lens and residual returns. We also used returns data for the S&P 500 Index through January 31, 2020 to calculate its impact on the Two Sigma Factor Lens’ factors and the investments’ and portfolios’ residual returns. We then specified a shock to the index for the next month (February 2020) where the size of the shock was the realized return for the index in February 2020 (-8.23%). For each investment or portfolio we tested, Scenario Analysis outputs an estimated return with error bands, and we then determine whether the actual returns of the investment or portfolio fell within those error bands. For example, say Scenario Analysis estimated that investment A would return -5% (with error bands of +/- 1%) if the S&P 500 returned -8.23% (again, these estimates are generated using data ending on January 31, 2020). If investment A’s actual return in February 2020 was between -6% and -4%, we would say that this investment successfully fell within the Scenario Analysis error bands.

We did the same analysis for the month of January 2020 (using data through December 31, 2019) and December 2019 (using data through November 30, 2019). 

Results³

Source: Venn-generated analysis as of March 20, 2020. 

How can you interpret these results and determine whether they are acceptable? First, we should mention that the error bands in Scenario Analysis are calculated using one standard deviation, implying that the actual returns (assuming a normal distribution) should fall within the error bands approximately 68% of the time.⁴ If the returns fall meaningfully above/below that number, especially as the sample size increases, the error bands were too widely/narrowly constructed and/or the actual return was far away from what the model predicted. 

The results in the two columns furthest to the right in the table above show that for each of the six scenario and investment/portfolio group pairings, the actual returns fell within the error bands relatively close to 68% of the time. In fact, the average observed across all six pairings was 66%. In some cases, the percent that fell within the error bands was greater than 68%, namely for the scenario in December 2019 that was least impacted by the coronavirus-related market events. And in other cases, the percent was below 68%. The lowest percent observed was for investments during the most extreme shock (as defined by its -2.17 standard deviation) in February 2020.

Second, the results highlight the importance for this type of linear analysis to have “guardrails” in place. As mentioned earlier, Scenario Analysis in Venn does not allow users to shock an index more than +/- 2 standard deviations from its monthly average return. The current methodology was designed with these guardrails in order to focus on regimes where linear analysis is appropriate and correlations are relatively stable. During volatile times like what we’ve seen the past few weeks, returns aren’t normally distributed. Additionally, factor correlations are unstable. 

The factor correlation matrix that Scenario Analysis uses is calculated over the preceding year with a half-life of 30 days (to emphasize more recent data). Unfortunately, factor correlations are changing right now by levels not seen since 2008. Here is a graph of how the factor correlation matrix has changed by comparing correlations over the most recent 50 days to the correlations 50 days prior. The average absolute correlation change among the factors over the most recent 50 days compared to the 50 days prior is 0.33.

Source: Venn’s factor returns data as of March 19, 2020. Time period: May 19, 2000 - March 16, 2020.

In an effort to enhance Scenario Analysis, the Venn research team is developing a methodology that would allow subscribers to test their portfolio’s sensitivity to extreme scenarios (i.e., greater than 2 standard deviations from their historical averages). Stay tuned!

Venncast

What is Venncast, and how are subscribers applying it today?

Venncast provides subscribers with a return estimate for their investments and portfolios in the absence of daily updated returns data.⁵ For example, say you only have returns for a manager through January 31, 2020. You might be wondering how that manager is faring throughout this crisis. Venncast can help. 

Construction of the Venncast Accuracy Test

To test the accuracy of Venncast during this volatile time, we used returns data for the same investments and anonymized user master portfolios through January 31, 2020 to calculate factor exposures and residual return. We then used Venncast to estimate the investment and portfolio returns for February 2020 and compared the Venncast return estimates to the actual realized returns.

Results

Source: Venn generated analysis as of March 16, 2020.

61% of investments and 77% of portfolios experienced actual returns in February 2020 that fell within the Venncast error bands. Similar to Scenario Analysis, the Venncast error bands are calculated using one standard deviation, so the actual returns should also fall within the error bands approximately 68% of the time. Venncast uses the actual factor returns for the out of sample period, so even though performance was extreme for the factors, Venncast still fell within the error bands relatively close to 68% of the time. 

Here’s an example of one of the investments we tested whose actual returns successfully fell within the error bands -- it’s a mutual fund called GMO Emerging Markets Debt III Fund.⁶ The actual returns of the investment are in blue, while the Venncasted return estimates and error bands are in gold:

Source: Venn as of March 23, 2020. Time period: January 1, 2020 - February 28, 2020.

Here is an example of a fund (PIMCO Income Institutional) that fell outside the Venncast error bands in February 2020.⁷ The actual fund performance is in blue; it narrowly missed the lower Venncast error band by about 0.1%:

Source: Venn as of March 25, 2020. Time period: January 1, 2020 - February 28, 2020.

Read more about these two types of analyses and other functionality on Venn that can be helpful in your understanding and management of risk, especially in these uncertain times.

REFERENCES

¹Portfolio analysis is available to Venn Pro users only.

²The mutual fund investments selected for the analysis include the most frequently analyzed mutual funds on Venn that have enough return history (i.e., three years) to run Scenario Analysis and that had returns updated through February 2020. The anonymized subscriber master portfolios were those in USD that had enough return history (also 3 years) to run Scenario Analysis and that had returns updated through February 2020.

³Scenario Analysis in Venn allows scenario shocks that are within 2 standard deviations of the scenario index’s historical mean return. Therefore, the analysis for the month of February 2020 would be restricted in Venn.

⁴According to the empirical rule.

⁵Venncast return estimates are provided up to the last available date of factor returns on Venn, which is typically a one business day lag.

⁶We selected this particular investment as it was the most frequently analyzed mutual fund in the Venn investment library that successfully fell within the Venncast error bands and was not sub-advised by Two Sigma.

⁷We selected this particular investment as it was the most frequently analyzed mutual fund in the Venn investment library that fell outside the Venncast error bands.

This article is not an endorsement by Two Sigma Investor Solutions, LP or any of its affiliates (collectively, “Two Sigma”) of the topics discussed. The views expressed above reflect those of the authors and are not necessarily the views of Two Sigma. This article (i) is only for informational and educational purposes, (ii) is not intended to provide, and should not be relied upon, for investment, accounting, legal or tax advice, and (iii) is not a recommendation as to any portfolio, allocation, strategy or investment. This article is not an offer to sell or the solicitation of an offer to buy any securities or other instruments. This article is current as of the date of issuance (or any earlier date as referenced herein) and is subject to change without notice. The analytics or other services available on Venn change frequently and the content of this article should be expected to become outdated and less accurate over time. Any statements regarding planned or future development efforts for our existing or new products or services are not intended to be a promise or guarantee of future availability of products, services, or features.  Such statements merely reflect our current plans.  They are not intended to indicate when or how particular features will be offered or at what price.  These planned or future development efforts may change without notice. Two Sigma has no obligation to update the article nor does Two Sigma make any express or implied warranties or representations as to its completeness or accuracy. This material uses some trademarks owned by entities other than Two Sigma purely for identification and comment as fair nominative use. That use does not imply any association with or endorsement of the other company by Two Sigma, or vice versa. See the end of the document for other important disclaimers and disclosures. Click here for other important disclaimers and disclosures.