Difference between revisions of "JSON Integration Adapter"

From CDOT Wiki
Jump to: navigation, search
(Current Status)
(Code Review Notes)
Line 25: Line 25:
  
 
=== Code Review Notes  ===
 
=== Code Review Notes  ===
{| border="2"
+
 
 +
{| border="2" style="text-align:center"
 
! Issue
 
! Issue
 
! Description
 
! Description
 
! Status
 
! Status
 +
! Class
 
! Priority
 
! Priority
 
|-  
 
|-  
Line 34: Line 36:
 
| Complete exceptions marked by TODO
 
| Complete exceptions marked by TODO
 
| Incomplete
 
| Incomplete
 +
| All Classes
 
!  
 
!  
 
|-  
 
|-  
| Use Hashholder instead of HashSet
+
| Document methods fully in javadoc style
| Use HashHolder in the XMLJSONMessagePartLoader instead of HashSet and whenever appropriate replace HashSet with HashHolder
+
| Document every method fully in javadoc style. Also document parameters, and use full English sentences. Make sure special settings like root mode or mode are well documented enough to use
 
| Incomplete
 
| Incomplete
 +
| All classes
 
!  
 
!  
 
|-  
 
|-  
| Change 0x000 for byte
+
| Make sure exceptions output the correct message part.
| For root modes, change 0xXXX to a cast to byte. Also do not use leading zeroes.
+
| Make sure exceptions output the correct message part, for example the inner part, rather than always the root part. More specific is better.
 
| Incomplete
 
| Incomplete
 +
| All classes
 
!  
 
!  
 
|-  
 
|-  
| Document methods fully in javadoc style
+
| Remove spaces after new Object[]{(remove space here)}
| Document every method fully in javadoc style. Also document parameters, and use full English sentences. Make sure special settings like root mode or mode are well documented enough to use
+
| Remove spaces after the curly brackets for all exceptions.
 
| Incomplete
 
| Incomplete
 +
| All classes
 
!  
 
!  
 
|-  
 
|-  
| Remove/Use InvocationContextAware
+
| Add a newline after creating each object or after each decision construct.
| Do not implement the invocation context aware interface if not used by the class. Or, use the interface for formatting of primitives.
+
| Add newlines to conform to NexJ coding style.
 
| Incomplete
 
| Incomplete
 +
| All classes
 
!  
 
!  
 
|-  
 
|-  
| Name methods XXXPart, not XXXNode
+
| Space before break keyword.
| Rename methods to be XXXPart, for example as writeCompositeMessagePart, instead of writeCompositeMessageNode.
+
| Put a space before all break keywords.
 
| Incomplete
 
| Incomplete
 +
| All classes
 
!  
 
!  
 
|-  
 
|-  
| Format primitive values before outputting to writers.
+
| Use Hashholder instead of HashSet
| Use the StringFormatter to format primitive values before outputting to the writer. For example, timestamps must be formatted before outputting to the writer.
+
| Use HashHolder in the XMLJSONMessagePartLoader instead of HashSet and whenever appropriate replace HashSet with HashHolder
 
| Incomplete
 
| Incomplete
 +
| JSONMessagePartMapping
 
!  
 
!  
 
|-  
 
|-  
| Make sure exceptions output the correct message part.
+
| Change 0x000 for byte
| Make sure exceptions output the correct message part, for example the inner part, rather than always the root part. More specific is better.
+
| For root modes, change 0xXXX to a cast to byte. Also do not use leading zeroes.
 
| Incomplete
 
| Incomplete
 +
| JSONMessagePartMapping, RootJSONMessagePartMapping
 
!  
 
!  
 
|-  
 
|-  
| Wrap fail
+
| Remove StringParsre from JSONMessagePartMapping
| Wrap all instances of fail with IntegrationException.
+
| Either remove StringParser from the mapping, or use it in formatting primitives. Do not leave it in if it is not used.
 
| Incomplete
 
| Incomplete
 +
| JSONMessagePartMapping
 
!  
 
!  
 
|-  
 
|-  
| Change design document to note that MessageTable only supports one message.
+
| Create class called RootJSONMessagePartMapping
| Design document should reflect all special settings of JSON parser and formatter. Specifically, any normally standard features which are not implemented by the JSON parser/formatter should be noted.
+
| Create a class called RootJSONMessagePartMapping for each CompositeMessagePart. This root JSON mapping will handle the "mode" for each CompositeMessagePart.
 
| Incomplete
 
| Incomplete
 +
| JSONMessagePartMapping, RootJSONMessagePartMapping
 
!  
 
!  
 
|-  
 
|-  
| Override JSONParse
+
| Allow root modes for each CompositeMessagePart
| Override parse method of JSONParser in the JSONMessageParser class. Do this to avoid changing JSONParser.
+
| Rename root mode to mode, so that each CompositeMessagePart can use every type of mode.
 
| Incomplete
 
| Incomplete
 +
| JSONMessagePartMapping, RootJSONMessagePartMapping
 
!  
 
!  
 
|-  
 
|-  
| Use primitive formatter when reading.
+
| Rename m_sFormat
| When reading information in from a stream in the formatter, use primitive formatter to determine the correct format of a primitive.
+
| Rename the variable m_sFormat if not used (or use it and keep the name).
 
| Incomplete
 
| Incomplete
 +
| JSONMessagePartMapping
 
!  
 
!  
 
|-  
 
|-  
| Remove spaces after new Object[]{(remove space here)}
+
| Remove/Use InvocationContextAware
| Remove spaces after the curly brackets for all exceptions.
+
| Do not implement the invocation context aware interface if not used by the class. Or, use the interface for formatting of primitives.
 
| Incomplete
 
| Incomplete
 +
| JSONMessageParser
 
!  
 
!  
 
|-  
 
|-  
| Add a newline after creating each object or after each decision construct.
+
| Name methods XXXPart, not XXXNode
| Add newlines to conform to NexJ coding style.
+
| Rename methods to be XXXPart, for example as writeCompositeMessagePart, instead of writeCompositeMessageNode.
 
| Incomplete
 
| Incomplete
 +
| JSONMessageParser
 
!  
 
!  
 
|-  
 
|-  
| Remove StringParsre from JSONMessagePartMapping
+
| Wrap fail
| Either remove StringParser from the mapping, or use it in formatting primitives. Do not leave it in if it is not used.
+
| Wrap all instances of fail with IntegrationException.
 
| Incomplete
 
| Incomplete
 +
| JSONMessageParser
 
!  
 
!  
 
|-  
 
|-  
| Create class called RootJSONMapping
+
| Override JSONParse
| Create a class called RootJSONMapping for each CompositeMessagePart. This root JSON mapping will handle the "mode" for each CompositeMessagePart.
+
| Override parse method of JSONParser in the JSONMessageParser class. Do this to avoid changing JSONParser.
 
| Incomplete
 
| Incomplete
 +
| JSONMessageParser
 
!  
 
!  
 
|-  
 
|-  
| Allow root modes for each CompositeMessagePart
+
| Use primitive formatter when reading.
| Rename root mode to mode, so that each CompositeMessagePart can use every type of mode.
+
| When reading information in from a stream in the formatter, use primitive formatter to determine the correct format of a primitive.
 
| Incomplete
 
| Incomplete
 +
| JSONMessageParser
 
!  
 
!  
 
|-  
 
|-  
| Rename m_sFormat
+
| Format primitive values before outputting to writers.
| Rename the variable m_sFormat if not used (or use it and keep the name).
+
| Use the StringFormatter to format primitive values before outputting to the writer. For example, timestamps must be formatted before outputting to the writer.
 
| Incomplete
 
| Incomplete
 +
| JSONMessageFormatter
 
!  
 
!  
 
|-  
 
|-  
| Space before break keyword.
+
| Pass enumerations into formatMessageRoot
| Put a space before all break keywords.
+
| Do not use member/class variables for storing information unless absolutely necessary. Prefer to pass parameters around so that objects do not grow large as the parser/formatter iterates.
 
| Incomplete
 
| Incomplete
 +
| JSONMessageFormatter
 
!  
 
!  
 
|-  
 
|-  
| Pass enumerations into formatMessageRoot
+
| Change design document to note that MessageTable only supports one message.
| Do not use member/class variables for storing information unless absolutely necessary. Prefer to pass parameters around so that objects do not grow large as the parser/formatter iterates.
+
| Design document should reflect all special settings of JSON parser and formatter. Specifically, any normally standard features which are not implemented by the JSON parser/formatter should be noted.
 
| Incomplete
 
| Incomplete
 +
| Design Document
 
!  
 
!  
 
|}
 
|}

Revision as of 10:50, 10 August 2011

Project Goal

To create an adapter for the NexJ Core similar in methodology and design of existing message adapters (XML, Fixed, CSV etc.) but using the JSON format


Contributors


Current Status

Changes being implemented as requested from first code review.

Code Review 1 Results

Code review took place on August 9th, 2011 at NexJ with Grace Batumbya, Brian Lim and Andrew Borzenko in attendance.

Major changes include:

  1. Allowing modes for all message parts, not just the root.
  2. Refactoring methods to check for != right condition instead of checking for the wrong condition to throw an exception. (whitelist instead of blacklist)
  3. Formatting all primitives before writing or storing them, particularly timestamps and other format sensitive strings.
  4. Creating a RootJSONMessagePartMapping extending JSONMessagePartMapping

Code Review Notes

Issue Description Status Class Priority
Exception TODO Complete exceptions marked by TODO Incomplete All Classes
Document methods fully in javadoc style Document every method fully in javadoc style. Also document parameters, and use full English sentences. Make sure special settings like root mode or mode are well documented enough to use Incomplete All classes
Make sure exceptions output the correct message part. Make sure exceptions output the correct message part, for example the inner part, rather than always the root part. More specific is better. Incomplete All classes
Remove spaces after new Object[]{(remove space here)} Remove spaces after the curly brackets for all exceptions. Incomplete All classes
Add a newline after creating each object or after each decision construct. Add newlines to conform to NexJ coding style. Incomplete All classes
Space before break keyword. Put a space before all break keywords. Incomplete All classes
Use Hashholder instead of HashSet Use HashHolder in the XMLJSONMessagePartLoader instead of HashSet and whenever appropriate replace HashSet with HashHolder Incomplete JSONMessagePartMapping
Change 0x000 for byte For root modes, change 0xXXX to a cast to byte. Also do not use leading zeroes. Incomplete JSONMessagePartMapping, RootJSONMessagePartMapping
Remove StringParsre from JSONMessagePartMapping Either remove StringParser from the mapping, or use it in formatting primitives. Do not leave it in if it is not used. Incomplete JSONMessagePartMapping
Create class called RootJSONMessagePartMapping Create a class called RootJSONMessagePartMapping for each CompositeMessagePart. This root JSON mapping will handle the "mode" for each CompositeMessagePart. Incomplete JSONMessagePartMapping, RootJSONMessagePartMapping
Allow root modes for each CompositeMessagePart Rename root mode to mode, so that each CompositeMessagePart can use every type of mode. Incomplete JSONMessagePartMapping, RootJSONMessagePartMapping
Rename m_sFormat Rename the variable m_sFormat if not used (or use it and keep the name). Incomplete JSONMessagePartMapping
Remove/Use InvocationContextAware Do not implement the invocation context aware interface if not used by the class. Or, use the interface for formatting of primitives. Incomplete JSONMessageParser
Name methods XXXPart, not XXXNode Rename methods to be XXXPart, for example as writeCompositeMessagePart, instead of writeCompositeMessageNode. Incomplete JSONMessageParser
Wrap fail Wrap all instances of fail with IntegrationException. Incomplete JSONMessageParser
Override JSONParse Override parse method of JSONParser in the JSONMessageParser class. Do this to avoid changing JSONParser. Incomplete JSONMessageParser
Use primitive formatter when reading. When reading information in from a stream in the formatter, use primitive formatter to determine the correct format of a primitive. Incomplete JSONMessageParser
Format primitive values before outputting to writers. Use the StringFormatter to format primitive values before outputting to the writer. For example, timestamps must be formatted before outputting to the writer. Incomplete JSONMessageFormatter
Pass enumerations into formatMessageRoot Do not use member/class variables for storing information unless absolutely necessary. Prefer to pass parameters around so that objects do not grow large as the parser/formatter iterates. Incomplete JSONMessageFormatter
Change design document to note that MessageTable only supports one message. Design document should reflect all special settings of JSON parser and formatter. Specifically, any normally standard features which are not implemented by the JSON parser/formatter should be noted. Incomplete Design Document

Project Phases

JSON Integration Adapter Phases


Phase I. Research (DONE)

  1. Complete Fundamentals of NexJ Studio tutorial
  2. Complete NexJ Integration tutorial
  3. Install NexJ Studio Express from source


Phase II. Design Proposal (DONE)

  1. Receive general approval for project
  2. Receive approval for JSON encoding options
  3. Receive approval for JSON formatting options
  4. Final Project Proposal


Phase III. Message Formatter (DONE)

  1. Add JSON to .XSD base types
  2. Create sample JSON .message
  3. Parse sample JSON .message with JUnit Test
  4. Create first set of unit tests for Message Formatter
  5. Code Message Formatter first draft
  6. Resolve issues with first set of unit tests
  7. Create second set of unit tests
  8. Resolve issues with second set of unit tests
  9. Refactor and code message formatter


Phase IV. Message Parser (DONE)

  1. Discover how to connect JSON RPC parser with Message Parser
  2. Create first set of trivial unit tests for Message Parser
  3. Code Message Parser skeleton
  4. Pass first set of trivial unit tests
  5. Create second set of unit tests (non-trivial) for Message Parser
  6. Code Message Formatter first draft
  7. Resolve issues with second set of unit tests
  8. Refactor


Phase V. Integration and Stress Tests (IN PROGRESS)

  1. Test Message Formatter and Message Parser with integration and stress tests
  2. Test through NexJ Studio Express GUI (manual testing)


Phase VI. Internal Code Review

  1. Internally review code at CDOT


Phase VII. Optimization

  1. Find more code optimizations relevant to Enterprise environments


Phase VIII. First Code Review at NexJ

  1. Date TBA


Phase IX. Code Rewrite

  1. Rewrite code to conform with NexJ suggestions and standards


Phase X. Second Code Review at NexJ

  1. Date TBA


Phase XI. Project Completion

Project Repository

BitBucket : https://bitbucket.org/b_lim/nexj-express-json-integration-adapter/


Technical Notes

Definitions

Term Description
Formatter Turns data into JSON
Parser Turns JSON into data
Message The metadata to be used when formatting and parsing a message. The grammar. It is a tree-like structure.
MessageInstance The information to be parsed or formatted

Because the Integration layer allows a Model to interact with external applications asynchronously, a description / grammar of the data to be exchanged between the two systems is defined using a Message in the Model.

  • Formatting is turning internal NexJ representation of a MessageInstance into JSON. The formatter is given a Message (the grammar), MessageInstance (the data), and a pipe. This accomplished by walking the Message tree-like structure while validating the data in the MessageInstance and streaming JSON format to the pipe. If the data in the MessageInstance does not conform to the Message, the formatter throws an exception.
  • Parsing is turning JSON representation of a MessageInstance into an internal NexJ representation of a MessageInstance. The parser is given a Message (the grammar) and an input stream. First the input stream is parsed for JSON using a JSONParser which returns either an Object or an Array. The formatter walks the Message building the structure of the MessageInstance and adding validated values in the object returned by the JSONParser to the MessageInstance. If the data from the input stream does not match the expected input at any time, the parser throws an exception.


Message Fundamentals

Message - Messages can contain values or other messages. The red nodes are messages, the green nodes are values.

NexJ Express Messages

Internally, messages are Transfer Objects. To determine if a node is a message, use instanceof on CompositeMessagePartInstance after retrieving the MessagePart. For example,

public void format(TransferObject tobj, Message message, Output out) throws IntegrationException
{
...
CompositeMessagePart root = message.getRoot(); // Gets the root of the message.
Iterator it = root.getPartIterator();
while (it.hasNext())
{
  part = (MessagePart)it.next());
  if (part instanceof CompositeMessagePartInstance)
  {
    // This part is a message
  }
  else if (part instanceof PrimitiveMessagePart)
  {
    // This part is a value
  }
...
}
...

NOTE: THE MESSAGE PARAMETER DOES NOT CONTAIN THE MESSAGE VALUES. THE MESSAGE PARAMETER CONTAINS THE MESSAGE METADATA. THE TRANSFER OBJECT CONTAINS THE MESSAGE VALUES WHICH MUST BE VALIDATED AGAINST THE MESSAGE TO ENSURE CORRECT FORMAT.

MessagePart
Parts of a message. Messages can contain values or other messages. NexJ Express has two types of message parts, CompositeMessagePart and PrimitiveMessagePart.
CompositeMessagePart
implementation is CompositeMessagePartInstance . The relationship between CompositeMessagePartInstance and PrimitiveMessagePart with the above picture is as follows - CompositeMessagePartInstance are messages (the red nodes) and PrimitiveMessagePart are values (the green nodes). To determine multiplicity of MessageParts, use isCollection() method of MessagePart. Note that multiplicity of the above screenshot are all [0..1], that is zero or one instance. Possible node multiplicities:
[0..1] 
zero or one instance (i.e. an optional entry)
[1..1] 
exactly one instance (i.e. required)
[1..N] 
collection of at least one, at most N instances
[0..0] 
collection of zero to infinite instances (displayed as [*])
[1..0] 
collection of at least one, possibly unlimited instances
XMLJSONMessageMappingLoader
Used by the framework to autoload JSONMessagePartMapping for each of the message parts.
JSONMessagePartMapping
Each node in the above picture may has a corresponding JSONMessagePartMapping. Each node may have its own mapping, with its own values initialized in XMLJSONMessageMappingLoader. In order to get the mapping, first cast MessagePart to a concrete class such as CompositeMessagePartInstance or PrimitiveMessagePart, then use part.getMapping(). The purpose of the mapping is metadata for each node.
JSONMessageFormatter
Used to turn messages into JSON format.
JSONMessageParser
Used to turn JSON into a message.

Algorithm

format(TransferObject tobj, Message message, Output out) Algorithm for format method is as follows

  1. Retrieve the root of the message
  2. Iterate through each of the parts of the root
    1. If the part is a CompositeMessagePartInstance, it is a message node
    2. If the part is a PrimitiveMessagePart, it is a value node
    3. Validate the part against the MessagePartMapping. Different types of validation must be done if the part is a Composite versus if it is a Primitive.
    4. If the part is a CompositeMessagePartInstance, recursively call the format method with TransferObject set to the part. Suggested to overload the format method to format(TransferObject tobj, MessagePart message, Output out) and pass in the part, since retrieving a message root with getRoot() will always get the highest root of the message but what you want is the parent.
    5. If the part is a PrimitiveMessagePart, write the message part to the output.

Resources

JSON RFC : http://www.ietf.org/rfc/rfc4627.txt
Introduction to NexJ Studio Express : https://www.projects.openhealthtools.org/sf/go/doc1771?nav=1
NexJ Developer's Guide
NexJ Integration Fundamentals
Open Health Tools Integration Platform https://www.projects.openhealthtools.org/sf/projects/oht_aip/