DECODING CHAINS - PART 2

Overview

This is the second part of the Decoding chains article. In this part I present two Java example applications that demonstrate the different concepts explained in Decoding chains - Part 1.

Disclaimer: The source code presented in this article and available in the related Github repository has been written by Thomson Reuters for the only purpose of illustrating the concepts explained in the Decoding Chains articles part 1 and part 2. It has not been tested for a usage in production environments.

Content

Application design

Build and Run

Using the EMA Chain Toolkit

More about the Chain Toolkit

Application design

The source code of these example applications has been designed for easy reuse by other example applications. It is made of three distinct parts:

  1. The EMA Chain Toolkit

This module implements the complete chain decoding logic and algorithms explained in the 1st part of the article. These features have been isolated in the EMA Chain Toolkit that is contained in the com.thomsonreuters.platformservices.ema.utils.chain package. The source code of this package is available on Github and is reused by other Thomson Reuters example applications. The toolkit also comes with a javadoc that fully describes the exposed API.

  1. The EmaChainToolkitExample application

This is an example application that demonstrates the EMA Chain Toolkit capabilities and how to use them. The application starts by creating an EMA OmmConsumer and uses it with the toolkit to expand different kinds of chains. Chain examples are expanded one by one in 10 individual steps.  Before each step, explanatory text is displayed and you are prompted to press <Enter> to start the step.

The EmaChainToolkitExample application also implements utility methods that are used to dispatch the OmmConsumer in different situations: until a chain is complete, until the user presses <Enter> or until a certain amount of time is elapsed.

Note: If you do not know yet about the Elektron Message API (EMA) and how to program an EMA consumer application I recommend you follow this EMA Quick Starts and tutorials.

Demonstrated features

The EmaChainToolkitExample application demonstrates the following EMA Chain Toolkit features:

  • Step 1: Builds and opens the Dow Jones chain (0#.DJI). Waits for the chain to complete by polling the isComplete() method. Gets and displays the chain elements.

  • Step 2: Builds and opens the Dow Jones chain. Leverages the ChainCompleteFunction functional interface to wait for completion (no polling), to get the chain elements and to display them.

  • Step 3: Builds and opens the Dow Jones chain. Leverages the ElementAddedFunction functional interface to display new chain elements as soon as they are decoded.

  • Step 4: Same as Step 2 but skips the summary links (first link .DJI in the case of the Dow Jones).

  • Step 5: Opens a very long chain (NASDAQ Basic) using the default algorithm. The NASDAQ Basic chain (0#UNIVERSE.NB) contains more than 8000 elements and may take more than 30 seconds to open with the default algorithm.

  • Step 6: Same as Step 5 but with the Name Guessing optimized algorithm. This time the chain takes 1 second or so to open.

  • Step 7: Opens the "NYSE Active Volume leaders" tile (.AV.O) with updates and displays any change that happens thanks to the ElementAddedFunction, ElementChangedFunction and ElementRemovedFunction functional interfaces. A Tile is a Chain that updates very frequently. .AV.O updates often when the US market is opened.

  • Step 8: Opens and displays the Japanese Equity Market recursive chain (0#JP-EQ).  This chain is a 3-level depth chain of chains.

  • Step 9: Same as Step 8 but with a maximum depth of 2 levels.

  • Step 10: Opens a chain that does not exit. Leverages the ChainErrorFunction functional interface to catch and display the error.

  1. The ChainExpander command line tool

This example application allows you to expand a flat chain from the command line. When the expansion is done, chain elements names are simply displayed on the output. The application accepts options and arguments that allow you to set the chain name, the service name and the DACS user name. You can also activate the optimization for long chains or even switch the application to a non verbose mode and redirect the output (the chain elements) to a file so that it can be processed by another application or script.

This example application allows you to expand a flat chain from the command line. When the expansion is done, chain elements names are simply displayed on the output either in text or JSON format. The application accepts options and arguments that allow you to set the chain name, the service name and the DACS user name. You can also activate the optimization for long chains or even switch the application to a non verbose mode and redirect the output (the chain elements) to a file so that it can be processed by another application or script.

Download the ChainExpander command line tool binary pack for Windows

Build and Run

Explanations for building and running the ChainExpander and the EmaChainToolkitExample are available in the README.md file within the Github project. Please refer to this file for more details. This readme file also describes the expected output and gives you some troubleshooting hints.

Using the EMA Chain Toolkit

This section explains how to use the EMA Chain Toolkit in case you would like to reuse it in other example applications.

The toolkit provides classes to handle two types of chains:

  • FlatChain objects: are mainly used to expand flat chains (chains that do not contain sub-chains). You can use a FlatChain object to open a recursive chain (a chain that contains sub-chains) but in that case only the first level of the recursive chain will be expanded.
  • RecursiveChain objects: are used to recursively expand recursive chains. That means that all elements of the recursive chain and all elements of its sub-chain will be expanded recursively. You can use a RecursiveChain object to open a flat chain (a chain that does not contain sub-chains). Obviously in that case, only the first level (and unique level) of the flat chain will be expanded.

What chain class to use? FlatChain or RecursiveChain?

It really depends on your use case: Do you want to open a chain that contains other chains in one go? RecursiveChain objects are very powerful for this purpose. However, they come with some drawbacks: 

  • By design RecursiveChain objects do not manage updates. This is a design choice made to avoid expensive structure changes that would come with updates of deep chains of chains.
  • RecursiveChain objects are less efficient and generate more requests than FlatChain objects. This is because recursive chains must subscribe to their elements to determine if they are sub-chains that need to be expanded or just simple instruments.
  • Because of their tree structure, RecursiveChain objects describe elements positions and elements names using lists of Longs and lists of Strings. This brings more complexity to you application compared to FlatChain objects.

If in doubt, FlatChain objects should be the preferred choice. For more details about recursive chains please refer to the Recursive chains section below.

Chain objects creation

Chain objects (either FlatChains or RecursiveChains) must be created using the related Builder class. Chains are built as if they were immutable objects. This means that, once they are built, you cannot directly change the fields' values (member's values) of these objects (they have no setter methods). Even if they are built the same way, chains cannot be considered as pure immutable objects as their states change when they receive data from EMA.

The following code snippet builds a FlatChain for the British FTSE 100 and a RecursiveChain for the Japanese Equity Markets:

OmmConsumer ommConsumer = ...;
      .
      .
      .
FlatChain fChain = new FlatChain.Builder()
            .withOmmConsumer(ommConsumer)
            .withServiceName("IDN_RDF")
            .withChainName("0#.FTSE")
            .build();

RecursiveChain rChain = new RecursiveChain.Builder()
            .withOmmConsumer(ommConsumer)
            .withChainName("0#JP-EQ")
            .withServiceName("IDN_RDF")
            .build();

To be noted that the build() method throws an IllegalStateException if you do not set the OmmConsumer or the chain name. The OmmConsumer must be properly initialized and connected to a Thomson Reuters Enterprise Platform or the Thomson Reuters Elektron platform. The service name can be omitted if the default value (“ELEKTRON_DD”) used by the Chain Toolkit matches your needs.

Opening and closing a chain

You simply open a chain by calling its open() method. Once opened, the chain starts subscribing to its underlying Chain Record instruments using the OmmConsumer passed to the builder.

To close a chain you just call its close() method. Doing this will automatically unsubscribe the underlying Chain Record instruments. 

Here is an example:

FlatChain theChain = new FlatChain.Builder()
            .withOmmConsumer(ommConsumer)
            .withServiceName("IDN_RDF")
            .withChainName("0#.FTSE")
            .build();

theChain.open();
      .
      .
      .
theChain.close();

When is the chain complete?

Once you created and opened the chain, you must wait for its completion before you can retrieve its elements list. If you do not, you may receive an incomplete list of elements.

You have two options to determine if a chain is complete. Either you call its isComplete() method that returns true if it is complete and false otherwise or you register a ChainCompleteFunction using the onComplete() builder method. This function will be called by the chain when it is complete. Here is an example that uses a lambda expression as a ChainCompleteFunction. This lambda gets the list of elements and displays it.

FlatChain theChain = new FlatChain.Builder()
            .withOmmConsumer(ommConsumer)
            .withServiceName("IDN_RDF")
            .withChainName("0#.FTSE")
            .onChainComplete(
                chain -> System.out.println(chain.getElements())
            )                  
            .build();

To be noted that the ChainCompleteFunction is only called once, after you opened the chain.

How to retrieve chain elements?

In the example above, we retrieved the chain’s elements by calling the getElements() method. This is handy as the elements are contained in a java collection: a Map with the elements positions as keys and the elements names as values. This map is sorted according to the natural ordering of the positions. The drawback of this method is that you must wait for the chain to be complete if you want the complete elements list.

Another way of doing it is to ask for notifications when the chain decodes new elements. To this aim you must register an ElementAddedFunction using the onElementAdded() builder method. This function will then be called each time the chain decodes a new element, giving you the element position in the chain and the element name. Here is an example that uses a lambda expression to do so. This lambda displays the chain’s name, the element’s position and name:

FlatChain theChain = new FlatChain.Builder()
            .withOmmConsumer(ommConsumer)
            .withServiceName("IDN_RDF")
            .withChainName("0#.FTSE")
            .onElementAdded(
                (position, name, chain) -> 
                    System.out.println(chain.getName() + "[" + position + "]= " +  name)
            )           
            .build();

What about chain updates?

As stated earlier, only the FlatChain class is able to manage updates. But note that the update management is deactivated by default. If you want to keep your chain up-to-date, you must call the withUpdate(true) method of the FlatChain builder. When this option is activated, the chain registers to EMA update messages and keep its elements list up-to-date so that you always get the latest elements list when you call getElements().

If you are interested in the updated details, you can register the ElementAddedFunction, ElementChangedFunction and ElementRemovedFunction functions by respectively calling the onElementAdded() on onElementChanged() and onElementRemoved() builder methods. Here is an example that uses lambda expressions as functions:

FlatChain theChain = new FlatChain.Builder()
            .withOmmConsumer(ommConsumer)
            .withServiceName("IDN_RDF")
            .withChainName(".AV.O")
            .withUpdates(true)
            .onElementAdded(
                (position, name, chain) -> 
                    System.out.println(chain.getName() + "[" + position + "]= " +  name)
            )           
            .onElementRemoved(
                (position, chain) -> 
                    System.out.println("Element removed at position " + position)
            )
            .onElementChanged(
                (position, previousName, newName, chain) -> 
                    System.out.println(previousName + " changed at position " + position + ". It's now named " + newName)
            )
            .build();

Note that all these functions may be called before or after the chain is complete.

The "Name Guessing" optimization

If your use case involves long chains that take several seconds to open, you may want to take advantage of the "Name Guessing" optimization. This optimization described in the first part of this article works around the subscription latency induced by the chains data structure. The idea is to guess names of the next Chain Records and to subscribe to these instruments in parallel.

To activate this optimized algorithm, you must call the withNameGuessingOptimization(int) method of the builder. The parameter indicates the number of Chain Record names to guess in advance. Here is an example:

FlatChain theChain = new FlatChain.Builder()
            .withOmmConsumer(ommConsumer)
            .withServiceName("IDN_RDF")
            .withChainName("0#UNIVERSE.NB")
            .withNameGuessingOptimization(50)
            .build();

This optimization is very effective for long chains. On the other hand it does not really make sense to use it with small chains and may be quite ineffective. Indeed, with small chains there's a higher risk that the algorithm subscribes to a big number of instruments that do not exist.

How to skip summary links?

Some chains start with summary links that are elements of the chain but not actual constituents of this chain. A good example is the Dow Jones chain (0#.DJI) that starts with the .DJI element that is not a Dow Jones constituent but a link to the Dow Jones index. If you’re not interested in these links, you can tell FlatChains and RecursiveChains to skip them. To do so, you must build an object that describes how many links to skip for the display templates used by your chains (read more about this in the 1st part of this article). Then, you just have to pass this object to the chains you build.

In the following example, we build 4 chains and tell them to skip:

  • 1 link for chains that use display template #187 (like 0#.DJI, the Dow Jones)
  • 2 links for chains that use display template #205 (like 0#.FTSE, the British FTSE 100)
  • 6 links for chains that use display template #1792 (like 0#.FTMIB, the Italian FTSE 100) 6 links for chains that use display template # 1098 (like 0#.FCHI, the French CAC40)
SummaryLinksToSkipByDisplayTemplate summaryLinksToSkip = 
    new SummaryLinksToSkipByDisplayTemplate.Builder()
            .forDisplayTemplate(187).skip(1)  // e.g. 0#.DJI
            .forDisplayTemplate(205).skip(2)  // e.g. 0#.FTSE
            .forDisplayTemplate(1792).skip(6) // e.g. 0#.FTMIB
            .forDisplayTemplate(1098).skip(6) // e.g. 0#.FCHI
            .build();

FlatChain chain1 = new FlatChain.Builder()
            .withOmmConsumer(ommConsumer)
            .withServiceName("IDN_RDF")
            .withChainName("0#.DJI")
            .withSummaryLinksToSkip(summaryLinksToSkip)
            .build();

FlatChain chain2 = new FlatChain.Builder()
            .withOmmConsumer(ommConsumer)
            .withServiceName("IDN_RDF")
            .withChainName("0#.FTSE")
            .withSummaryLinksToSkip(summaryLinksToSkip)
            .build();

FlatChain chain3 = new FlatChain.Builder()
            .withOmmConsumer(ommConsumer)
            .withServiceName("IDN_RDF")
            .withChainName("0#.FTMIB")
            .withSummaryLinksToSkip(summaryLinksToSkip)
            .build();

FlatChain chain4 = new FlatChain.Builder()
            .withOmmConsumer(ommConsumer)
            .withServiceName("IDN_RDF")
            .withChainName("0#.FCHI")
            .withSummaryLinksToSkip(summaryLinksToSkip)
            .build();

Recursive chains

Some chains contain elements that are also chains. You may want to open these chains of chains recursively in one go. To do so you must use a RecursiveChain object instead of a FlatChain. RecursiveChains are powerful tools that can expand deep recursive chains very simply. However, this recursive approach inevitably comes with some complexity when it comes to describe elements positions and elements names. With a RecursiveChain an element position is described as a list of integers (each integer represents the position of the element at a certain depth). An element name is described as a list of strings (these strings represent the path to navigate from the root of the chain to the element).

As an example, this is an extract of the "0#JP-EQ" recursive chain. At line 4 you see that ".MTHR" is the 3rd element of the ".TSEI" sub-chain (as the position numbering starts at 0 so .MTHR position in .TSEI is 2). You also see that ".TSEI" is the second element of 0#JP-EQ. The complete position description of ".TSEI" is a list of integers ("1, 2"). Its full path in the "0#JP-EQ" chain is a list of Strings (".TSEI", ".MTHR"). 

 1      0#JP-EQ[0] = [.TOPXC]
 2      0#JP-EQ[1, 0] = [.TSEI, .TOPX]
 3      0#JP-EQ[1, 1] = [.TSEI, .TSI2]
 4      0#JP-EQ[1, 2] = [.TSEI, .MTHR]
 5      0#JP-EQ[1, 3] = [.TSEI, .TSIL]
 6      0#JP-EQ[1, 4] = [.TSEI, .TSIM]
 7      0#JP-EQ[1, 5] = [.TSEI, .TSIS]
 8      0#JP-EQ[1, 6] = [.TSEI, .TOPXC]
 9      0#JP-EQ[1, 7] = [.TSEI, .TOPXL]
10           .
11           .
12           .
13      0#JP-EQ[5, 11, 27] = [0#JP-INDICES, .TSEK, .IBNKS.T]
14      0#JP-EQ[5, 11, 28] = [0#JP-INDICES, .TSEK, .ISECU.T]
15      0#JP-EQ[5, 11, 29] = [0#JP-INDICES, .TSEK, .IINSU.T]
16      0#JP-EQ[5, 11, 30] = [0#JP-INDICES, .TSEK, .IFINS.T]
17      0#JP-EQ[5, 11, 31] = [0#JP-INDICES, .TSEK, .IRLTY.T]
18      0#JP-EQ[5, 11, 32] = [0#JP-INDICES, .TSEK, .ISVCS.T]
19      0#JP-EQ[5, 12] = [0#JP-INDICES, .TSA1]
20      0#JP-EQ[5, 13] = [0#JP-INDICES, .TSA2]

If you do not want the RecursiveChain to drill down too deeply, you can limit the depth using the withMaxDepth(int)  method of the builder. As an example, here is how to recursively create a chain for the Japanese Equity Market with a maximum depth of 2 levels:

RecursiveChain theChain = new RecursiveChain.Builder()
            .withOmmConsumer(ommConsumer)
            .withServiceName("IDN_RDF")
            .withChainName("0#JP-EQ")
            .withMaxDepth(2)
            .build();

More about the Chain Toolkit

To learn more about the Chain Toolkit please refer to the EMA Chain Toolkit javadoc available for download below.