NxsComment Class Reference

General notes on NexusTokenizing. More...

#include <nxstoken.h>

Detailed Description

General notes on NexusTokenizing.

File position information (pos, line and column) refer to the end of the token.

Note 1: the GetEmbeddedComments methods of ProcessedNxsToken and NxsToken can be tricky to use if detailed position location of the comment is required. A vector of "embedded comments" in the NCL context is a collection of all comments that were encountered during a GetNextToken operation. The behavior depends on whether the tokenizer can tell if a section of text is has the potential to span comment. Potentially comment-spanning tokens have to be read until a token-breaker is found. Thus they include trailing comments. Thus it is not always easy (or possible) for client code to determine whether a specifie comment belongs "with" a particular NEXUS token rather than the previous or next token. For example: Text Result as (token, {"embedded comment"}) pairs Explanation ============================ ============================================= ===================================== ;a[1]b; (;, {}), (ab, {1}), (;, {}) ab is a comment-spanning token ;[1]a[2]b; (;, {}), (ab, {1, 2}), (;, {}) tokenizer realizes that ; is always a single token so [1] is not encountered until the second GetNextToken() call. a[1];[2]b; (a, {1}), (;, {}) (b, {2}), (;, {}) First GetNextToken() call reads "a" token until ; (thus reading "[1]") ; is a single character token, so no comments are read, thus making [2] part of the third GetNextToken call().

In some cases the comment position information and token position information may reveal the exact location of the comments. Fortunately the relative order of comments is retained and the exact position is rarely needed.\

Note 2: Using the NxsToken class with the saveCommandComments LabileFlag causes [&comment text here] comments to be returned as tokens ONLY if they are not preceded by potentially comment-spanning tokens. This "feature" is new to NCL v2.1 and is the result of a bug-fix (previous versions of NCL incorrectly broke tokens at the start of any comment).

Text Result as in saveCommandComments mode Explanation ========================= ============================================= ===================================== =[&R](1, ("=",{}) ("&R", {}), ("(",{}), ("1",{}), (",",{}) [&R] is not in the middle of potentially-comment-spanning token. a[&R]b, ("ab",{"&R"}), (",",{}) [&R] is in the middle of comment-spanning token "ab" a[&R], ("a",{"&R"}), (",",{}) [&R] is in on the trailing end of a potentially-comment-spanning token "a" the tokenizer This wart makes it more tedious to deal with command comments. However it is tolerable becuase the only supported use of command comments in NCL is after single-character tokens (for example after = in a tree descpription).

The NHX command comments are not processed by NCL, but they occur in contexts in which it will be possible to determine the correct location of the comment (though it is necessary to check the embedded comments when processing NHX trees):

Text Result as NOT IN saveCommandComments mode Explanation ========================= =================================================== ===================================== ):3.5[&&NHXtext], (")",{}) (":", {}), ("3.5",{"&&NHXtext"}), (",",{}) "3.5" is potentially-comment-spanning, but the comment still is stored with other metadata for the same edge. )[&&NHXtext], (")",{}) (",", {"&&NHXtext"}) NHX comment is parsed with the second token, but because , is NOT potentially-comment-spanning know that [&&NHXtext] must have preceded the comma (the token and comment column numbers would also make this clear.

New in 2.1

See Note 2 above (bug-fix, but wart introduced). This could lead to loss of backward compatibility if client code relies of saveCommandComments in contexts in which command comments occur within potentially comment-spanning tokens.
NxsComment class.

Comments are stored in tokenization (the GetNextToken() call will trash the previous comments, so client code must store comments if they are needed permanently).

NxsToken::SetEOFAllowed method added and SetEOFAllowed(false) is called when entering a block. This means that when parsing block contents the NxsToken tokenizer will raise an NxsException if it runs out of file (thus Block reader code no longer needs to check for atEOF() constantly to issue an appropriate error).
ProcessedNxsToken class, NxsToken:ProcessAsSimpleKeyValuePairs, and NxsToken:ProcessAsCommand methods. This makes it easier to parse commands (by allowing random access to the tokens in a command). These methods are not appropriate for very long commands (such as MATRIX) or commands that require fiddling with the tokenizing rules (such as disabling the hyphen as a token breaker)
lazy NxsToken::GetFilePosition() and low level io operations dramatically speed up tokenization (~10-20 times faster).
some other utility functions were added, and some refactoring (delegation to NxsString) was done to clean up

Storage for a comment text and (end of the comment) file position information

Definition at line 110 of file nxstoken.h.

The documentation for this class was generated from the following file:

nxstoken.h