NxsComment Class Reference

General notes on NexusTokenizing. More...

#include <nxstoken.h>

List of all members.

Detailed Description

General notes on NexusTokenizing.

File position information (pos, line and column) refer to the end of the token.

Note 1: the GetEmbeddedComments methods of ProcessedNxsToken and NxsToken can be tricky to use if detailed position location of the comment is required. A vector of "embedded comments" in the NCL context is a collection of all comments that were encountered during a GetNextToken operation. The behavior depends on whether the tokenizer can tell if a section of text is has the potential to span comment. Potentially comment-spanning tokens have to be read until a token-breaker is found. Thus they include trailing comments. Thus it is not always easy (or possible) for client code to determine whether a specifie comment belongs "with" a particular NEXUS token rather than the previous or next token. For example: Text Result as (token, {"embedded comment"}) pairs Explanation ============================ ============================================= ===================================== ;a[1]b; (;, {}), (ab, {1}), (;, {}) ab is a comment-spanning token ;[1]a[2]b; (;, {}), (ab, {1, 2}), (;, {}) tokenizer realizes that ; is always a single token so [1] is not encountered until the second GetNextToken() call. a[1];[2]b; (a, {1}), (;, {}) (b, {2}), (;, {}) First GetNextToken() call reads "a" token until ; (thus reading "[1]") ; is a single character token, so no comments are read, thus making [2] part of the third GetNextToken call().

In some cases the comment position information and token position information may reveal the exact location of the comments. Fortunately the relative order of comments is retained and the exact position is rarely needed.\

Note 2: Using the NxsToken class with the saveCommandComments LabileFlag causes [&comment text here] comments to be returned as tokens ONLY if they are not preceded by potentially comment-spanning tokens. This "feature" is new to NCL v2.1 and is the result of a bug-fix (previous versions of NCL incorrectly broke tokens at the start of any comment).

Text Result as in saveCommandComments mode Explanation ========================= ============================================= ===================================== =[&R](1, ("=",{}) ("&R", {}), ("(",{}), ("1",{}), (",",{}) [&R] is not in the middle of potentially-comment-spanning token. a[&R]b, ("ab",{"&R"}), (",",{}) [&R] is in the middle of comment-spanning token "ab" a[&R], ("a",{"&R"}), (",",{}) [&R] is in on the trailing end of a potentially-comment-spanning token "a" the tokenizer This wart makes it more tedious to deal with command comments. However it is tolerable becuase the only supported use of command comments in NCL is after single-character tokens (for example after = in a tree descpription).

The NHX command comments are not processed by NCL, but they occur in contexts in which it will be possible to determine the correct location of the comment (though it is necessary to check the embedded comments when processing NHX trees):

Text Result as NOT IN saveCommandComments mode Explanation ========================= =================================================== ===================================== ):3.5[&&NHXtext], (")",{}) (":", {}), ("3.5",{"&&NHXtext"}), (",",{}) "3.5" is potentially-comment-spanning, but the comment still is stored with other metadata for the same edge. )[&&NHXtext], (")",{}) (",", {"&&NHXtext"}) NHX comment is parsed with the second token, but because , is NOT potentially-comment-spanning know that [&&NHXtext] must have preceded the comma (the token and comment column numbers would also make this clear.

New in 2.1

Comments are stored in tokenization (the GetNextToken() call will trash the previous comments, so client code must store comments if they are needed permanently).

Storage for a comment text and (end of the comment) file position information

Definition at line 110 of file nxstoken.h.

The documentation for this class was generated from the following file:
 All Classes Functions Variables Enumerations Enumerator Friends
Generated on Mon Mar 29 16:37:12 2010 for NCL by  doxygen 1.6.3