I messed with chpl.lex and did
make parser
All good.
I updated my bison to bison 3.7.6 because of Chapel's requirements. I did
make compiler
and the sky fell in.
See attached error log. Any ideas?
errors.txt (16.6 KB)
Thanks - Damian
Hi Damian —
Doesn't look familiar on a sleepy Saturday morning, but if the order of operations above is correct, you should do make parser again after upgrading your version of bison.
-Brad
Thanks for interrupting your weekend.
The order was incorrect.
After I upgraded bison, I ran mark parser and then tried make compiler
Do I need to upgrade flex also? I have 2.5.37?
Thanks again.
That could be the issue. frontend/lib/parsing/README says "Flex 2.6 or later"
-Brad
Did you run a make parser step after upgrading flex but before running the make whose output you attached in that last message?
-Brad
Yes. The make parser works without errors.
Hi Damian —
In frontend/lib/parsing/flex-chpl-lib.h, do you see lines like the following within the first 20 lines or so, indicating which version of flex is being used:
#define YY_FLEX_MAJOR_VERSION 2
#define YY_FLEX_MINOR_VERSION 6
#define YY_FLEX_SUBMINOR_VERSION 4
And then in the first line of bison-chpl-lib.h in that same directory, do you see a line like:
/* A Bison parser, made by GNU Bison 3.7.6. */
?
Thanks,
-Brad
Yes to both questions. Thanks for helping me on your Sunday.
What C++ compiler and version are you using?
-Brad
I build flex and bison with gcc 11.2.1.
Both g++ 11.2.1 and clang15 are in my search path.
I have CHPL_LLVM=bundled so I assume that the Makefile is using clang15.
I cannot even understand what the error messages are telling me.
I built everything from scratch.
I have flex 2.6.4 and bison 3.8.
Standard 1.33.0 installs just fine.
make parser
Works fine. Typing
make compiler
It looks like it wants a Clang_DIR and none is defined.
Making the compiler - same problem:(logs attached)
make[1]: Nothing to be done for 'llvm'.
make[1]: Nothing to be done for 'jemalloc'.
***** ./ *****
-- Configuring Chapel version: 1.33.0
-- CMAKE_INSTALL_PREFIX is /usr/local
CHPL_DEVELOPER is not set, using OFF
-- No CHPL_LLVM_GCC_PREFIX env var or value given from command line.
-- Using Python: python3
-- Could NOT find Doxygen: Found unsuitable version "1.8.5", but required is at least "1.8.17" (found /usr/bin/doxygen, found components: doxygen)
-- Could NOT find Clang (missing: Clang_DIR)
-- Using libclang from Clang_DIR-NOTFOUND
-- Configuring done
-- Generating done
dgm0-parser.txt (1.7 KB)
dgm0-compiler.txt (28.0 KB)
On the system with flex 2.6.4 and bison 3.8,m I downloaded the latest chapel-main and built the compiler and tested it. It works. With the chpl.lex in that release, I then did
make parser
make compiler
Same errors.
I'm not sure what's going wrong, but I might be able to explain some of the errors:
error: 'yyget_extra' was not declared in this scope; did you mean 'yyextra'?-- I think this means that flex isn't generating a re-entrant scanner with bison-bridge. These are requested with these lines in chpl.lex:
%option reentrant
%option bison-bridge
error: invalid conversion from 'void*' to 'ParserContext*'-- I think this means that the "extra type" is not being set. It should be set toParserContext*with the following line in chpl.lex:
%option extra-type="ParserContext*"
It's possible that there's some bit of an earlier experiment that is lying around and messing things up. You could try manually deleting the generated parser files and then running make parser again to see if you get a different result. (You should be able to see these are modified with git status if you are in a git clone, but they are:
frontend/lib/parsing/bison-chpl-lib.cpp
frontend/lib/parsing/bison-chpl-lib.h
frontend/lib/parsing/flex-chpl-lib.cpp
frontend/lib/parsing/flex-chpl-lib.h
Lastly, part of the reason that we include these generated files in the repository is that it can be tricky to get a working version of flex and bison. However, that approach does not help if you want to adjust the parser.
Regarding the bison version:
I updated my
bisontobison 3.7.6because of Chapel's requirements. I did
What gave you the idea that you should use bison 3.7.6? If it was in documentation, it is probably out of date. We have recently been using bison 3.8.2 and flex 2.6.4 when generating the parser when we update it. These versions correspond to the current versions in Homebrew.
I upgrsaded bison,to 3.8.2, I deleted those four files and then run:
make -s parser
\Making the parser...
-- Configuring Chapel version: 1.33.0
-- CMAKE_INSTALL_PREFIX is /usr/local
CHPL_DEVELOPER is not set, using OFF
-- No CHPL_LLVM_GCC_PREFIX env var or value given from command line.
-- Using Python: python3
-- Could NOT find Doxygen: Found unsuitable version "1.8.5", but required is at least "1.8.17" (found /usr/bin/doxygen, found components: doxygen)
-- Could NOT find Clang (missing: Clang_DIR)
-- Using libclang from Clang_DIR-NOTFOUND
-- Configuring done
CMake Error at frontend/lib/CMakeLists.txt:50 (add_library):
Cannot find source file:
/pub/lang/chapel/chapel-1.33.0/frontend/lib/parsing/bison-chpl-lib.cpp
Tried extensions .c .C .c++ .cc .cpp .cxx .cu .mpp .m .M .mm .ixx .cppm .h
.hh .h++ .hm .hpp .hxx .in .txx .f .F .for .f77 .f90 .f95 .f03 .hip .ispc
-- Generating done
CMake Generate step failed. Build files cannot be regenerated correctly.
make[1]: *** [Makefile:198: parser] Error 1
make: *** [Makefile:92: parser] Error 2
When I originally upgraded flexI figured I needed a newer version of bison and randomly chose 3.7.6 rather than the latest. I now have 3.8.2 but it causes grief.
I went into the latest chapel-main and noticed that it too was using flex 2.6.4. I did NOT delete those files and did NOT modify the original chpl.lex file. I ran
touch frontend/lib/parsing/chpl.lex
make parser
make compiler
and while the newly created bison files are the same, the newly created flex files disagree with the original. The delta is attached.
delta.txt (138.8 KB)
That is odd. What OS are you using? And did you say you built flex from source? I suspect we will need to reproduce the problem to be able to make progress on it.
Regarding deleting the files, yeah I see that did not work, but that has nothing to do with the bison version. It's just I led you astray. I should have said that you needed to git checkout those files to avoid the potential for stray information.
But, all that is potentially not important. You were trying to try something by modifying the chpl.lex file, right? Maybe there is a simpler way we can help you meet your original goals. Would a more reproducible environment to rebuild the parser help? E.g. what if we create a apptainer or Docker image in which you could rebuild the parser and then copy it out once that is done?
I am running CentOS 7.9. I built both flex and bison from source. I am simply trying to modify chpl.lex and rebuild the compiler. I have done this before but long ago. This is not rocket science. Also, I should be able to remove those files and rebuild things. Very odd.
I do not need a Docker image or a container.
I am happy to give you access to the system.
I cannot see why I cannot download a compile of chapel-main, simply do a touch of chpl.lex and still manage to break a rebuild.
Can you send me the git sequence please. It is ages since I did that with a Chapel release. Tjanks.
I did a git clone. I captured the flex-* and bison-* files from frontend/parsing/lib.
I built the compiler. I did not touch any file. I then did
make parser
The bison-* files are identical. The flex-* files are totally different. The original flex-* files were built with 2.6.4. The new files were built with the same version.
I used Ubuntu.
- With the standard
flexpackage 2.6.4 installed, the generated files match the release. - Installing
flexfrom source, and using that version, the generated files do NOT match.
I was using the latest flex sources. Well the .zip version
In frustration, I went fishing. On the Github project page for Flex and found `flex-2.6.4.tar.gz'.
I went back to CentOS. I built flex from the TAR file and it gives different results to the ZIP.
What is interesting, the results it gives now match that of the Chapel release.
Really scary stuff. Thanks for the feedback which gave me enough hints to figure out the solution.
1 Like