Branch: refs/heads/master
Revision: 799aa6f
Author: mppf
Log Message:
Merge pull request #17588 from mppf/new-asts-parser
responsive compiler prototype: parser to produce new AST classes
This PR contains a separate parser for the compiler revamp effort.
Split out from PR #17583, which is a start at the compiler revamp effort
described in the
1.24 release notes ongoing efforts.
The flex/bison files started out as copies of the parser from the main
compiler. I removed all semantic actions and adjusted these to not use
global variables. Additionally, instead of storing comments in bison
locations, it stores a vector of comments not yet consumed in the
ParserContext structure. Along with removing the semantic actions I have
adjusted the types of these rules and made a couple of minor changes to
get the comments processing correctly. As far as I know, this parser no
longer uses any global state. Several rules that used to return a
BlockStmt (because they are a "statement) now return CommentsAndStmt. The
idea here is that, during parsing, when we create a statement, we gather
comments that appeared before it. We include those in the result of
parsing that statement. Additionally there is some logic to clear the
comments when we reach the end of a statement.
While working on the parser, I moved lots of the C++ support code that
used to be in chapel.flex/chapel.bison to other files that are
#included in these. That way, we don't have to regenerate the parser if
they change.
Note that in the lexer, I've changed the handling of comments to include
recording the entire comment (rather than leaving out the introduction or
dealing with other formatting). That way, tools like chpldoc can
separately decide which comment introductions require further processing.
Additionally, in the lexer, I've changed the parsing of string literals
to record string data in the unescaped form. Previously, the compiler
stored string literals with C escapes. Storing the escapes has led to a
variety of bugs over time (because things like string.length for param
strings needs to handle the escapes). But more importantly, as the
default backend switches to LLVM, these C escapes don't make sense
anymore in the default configuration. Instead, we will need to add the
necessary escapes when generating C code.
The parser in this PR has the ability to parse a simple subset of the
language (including { } blocks, trivial variable declarations like var a;, and identifier statements like a;). Other language elements still
have parser rules but their semantic actions just create error sentinel
nodes.
The parser currently stores the results from the rules in a union. As a
result, we can't have types that have default constructors. (This is the
reason for the PODUniqueString type and also a reason that the parser
code has its own kind of expression list type, etc). Bison has some
support for C++ variants but I did not take on the task of updating our
parser to use those. As it is today, the nested rules from a parser rule
generally return a pointer that needs to be stored somewhere or deleted.
This is manageable, although it makes the parser code less elegant than
it could be.
Lastly, I created several helper #defines that can be used in the
semantic actions. This was important because for now most of the semantic
actions are "TODO" and so I needed a short way of putting that. But,
these macros also help with things like handling comments appropriately
when parsing a Statement. These macros like TODOSTMT LOC STR etc
are defined in parser-help.h.
- [x] full local testing
Reviewed by @lydia-duncan - thanks!
Modified Files:
A compiler/next/include/chpl/Frontend/Parser.h
A compiler/next/lib/Frontend/Parser.cpp
A compiler/next/lib/Frontend/Parser/ParserContext.h
A compiler/next/lib/Frontend/Parser/ParserContextImpl.h
A compiler/next/lib/Frontend/Parser/ParserDependencies.h
A compiler/next/lib/Frontend/Parser/README
A compiler/next/lib/Frontend/Parser/chapel.lex
A compiler/next/lib/Frontend/Parser/chapel.ypp
A compiler/next/lib/Frontend/Parser/lexer-help.h
A compiler/next/lib/Frontend/Parser/parser-help.h
Compare: https://github.com/chapel-lang/chapel/compare/be4f2f7ca112...799aa6f4b2c7