• Help
  • Help! For the life of me, I cannot run the C# tests locally

vsajip Try removing the netstandard2.1; and rebuilding and seeing if the error persists.

Well, now there is a different error. (By the way, that also involved commenting out the two lines here: https://github.com/javacc21/javacc21/blob/master/ptest.py#L90-L91 since the dlls array is now empty.

But after that, I get:

Failed: Failed to import org.parsers.json at part org
Failed: name 'd' is not defined.

And I can see where that comes from. The message is here: https://github.com/javacc21/javacc21/blob/master/ptest.py#L110 but the problematic line is actually 105, I guess. The import is failing. But I have to admit that I feel pretty helpless in terms of figuring this out.

I'll try on Linux shortly and report back. I'm realizing how absolutely abysmal my knowledge of Dotnet is. Now, I always thought it was basically like Java, so the CLR is basically like the JVM. In Java land, I can package everything up into a self-contained jarfile and there's no shared library impedance sort of issue because everything is self-contained and the only classes it's using outside that jar are the ones in the standard library. Surely we ought to be able to build something like that for Dotnet that "just works". You know, it really seems to me that the whole "DLL hell" problem should not really exist so much in a world where a megabyte of RAM costs less than a penny. (And disk space is far cheaper than that even.)

    revusky since the dlls array is now empty

    It shouldn't be, because the net5.0 version should have been built, even if the netstandard2.1 hasn't been. If there is no DLL, then imports will of course fail because we're trying to import stuff from the DLLs.

    revusky Surely we ought to be able to build something like that for Dotnet that "just works"

    That should be possible, I think, but you will typically have a generated parser as a separate "assembly" which is not quite analogous to a .jar because you can't combine them together - they are independent, executable binary constructs (.dll and .exe) and not as nicely organised/composable as Java classes are. (As an aside, Python can also be zipped up and applications/libraries shipped as .zip files).

    Of course, the "DLL hell" problem is one of packaging / distribution. An application might be able to bundle all its dependencies neatly, but with libraries it's harder to achieve that, especially when dependencies keep evolving and their using software needs to keep abreast of changes.

    vsajip

    I now realize why I thought that we were using Mono on mac/linux. I just typed ipy on the command line on Ubuntu and it gives me:

    IronPython 2.7.12 (2.7.12.1000)
    [.NETFramework,Version=v4.5 on Mono 6.8.0.105 (Debian 6.8.0.105+dfsg-3.2 Wed Jun 30 05:34:49 UTC 2021) 
    (64-bit)]
    Type "help", "copyright", "credits" or "license" for more information.
    >>>

    It's launching /usr/bin/ipy which is a simple little script that is:

    #!/bin/sh
    /usr/bin/mono /usr/share/ironpython2.7/ipy.exe $*

    So, clearly, I installed IronPython at some point in the past and it put in a launch script that used Mono (which I had also previously installed.)

    I also do have Dotnet installed. If I type dotnet --version on the command line, I get back:

    6.0.302

    However, if I try to launch IronPython with dotnet instead of mono, i.e.

    dotnet /usr/share/ironpython2.7/ipy.exe 

    I get:

    Cannot use file stream for [/usr/share/ironpython2.7/ipy.deps.json]: No such file or directory
    A fatal error was encountered. The library 'libhostpolicy.so' required to execute the application was not found 
    in '/usr/share/ironpython2.7/'.
    Failed to run as a self-contained app.
      - The application was run as a self-contained app because '/usr/share/ironpython2.7/ipy.runtimeconfig.json' 
    was not found.
      - If this should be a framework-dependent app, add the '/usr/share/ironpython2.7/ipy.runtimeconfig.json' file 
    and specify the appropriate framework.

    Well, I guess I need to have that ipy.runtimeconfig.json in the same directory as the ipy.exe. So, what should that file look like? Actually, it also looks like I need ipy.deps.json and libhostpolicy.so

    All of this give me a rather uneasy feeling because I have no recollection of ever installing Mono on Ubuntu. Am I in early stage Alzheimer's, or maybe I installed some other app that had Mono as a dependency and thus, it got installed... Well, actually, I think I did install Mono at some point, because I was curious and wanted to muck with it. (Though I never did...) Must be, it's the kind of thing I'd do, but I can't remember doing it...

      revusky I have no recollection of ever installing Mono on Ubuntu. Am I in early stage Alzheimer's

      No, it's that IronPython was built using Mono and hasn't been rebuilt using .NET Core (don't know why, but it's a volunteer project, so you can't expect anything).

      It's a bit of Yak shaving at this point, so unless you want to go down this rabbit hole, it might be easier to do what I do - develop on Linux and let the GitHub runner do the heavy lifting on Windows and macOS. Further down the line, we will of course probably need to dig into the causes of this kind of issue.

        vsajip

        Well, I spent a little more time trying to get this working on Linux (Ubuntu 22.04) and I have to admit that I'm totally confused. I went into the synaptic package manager and I uninstalled mono. A forced removal in which I uninstalled mono-runtime and everything else associated with it, including IronPython. I figured I would just try to go through the installation steps here: https://github.com/javacc21/javacc21/blob/master/.github/workflows/csharp-tests.yml

        So then I did sudo apt install dotnet6 and then I figured I could install IronPython and it would use dotnet. I try to do the IronPython install as per the above YAML spec, i.e. https://github.com/javacc21/javacc21/blob/master/.github/workflows/csharp-tests.yml#L59-L60 and I get:

        revusky@gram ~ % sudo dpkg -i ironpython_2.7.11.deb
        Selecting previously unselected package ironpython.
        (Reading database ... 300477 files and directories currently installed.)
        Preparing to unpack ironpython_2.7.11.deb ...
        Unpacking ironpython (2.7.11) ...
        dpkg: dependency problems prevent configuration of ironpython:
         ironpython depends on mono-runtime (>= 5.12); however:
          Package mono-runtime is not installed.
         ironpython depends on libmono-posix4.0-cil (>= 5.12); however:
          Package libmono-posix4.0-cil is not installed.
        
        dpkg: error processing package ironpython (--install):
         dependency problems - leaving unconfigured
        Errors were encountered while processing:
         ironpython

        This Debian package "thinks" that mono-runtime is a dependency of ironpython and won't install the latter if Mono is not there. So, what to do? I re-install mono-runtime and then the above command works and I've got ironpython 2.7.11 installed and then when I type ipy on the command line, I get:

        IronPython 2.7.11 (2.7.11.1000)
        [.NETFramework,Version=v4.5 on Mono 6.8.0.105 (Debian 6.8.0.105+dfsg-3.2 Wed Jun 30 05:34:49 
        UTC 2021) (64-bit)]
        Type "help", "copyright", "credits" or "license" for more information.
        >>> 

        So it's running ironpython on Mono. And, as best I understand, the mono runtime is installed on ubuntu, at least this latest version, by default -- which would be why I have no recollection of ever installing it!

        So, here's a question: when you type ipy on your development box, does it tell you that your IronPython is running on top of Mono, or somehow is it using Dotnet?

        Of course, this leads finally to other questions, like: shouldn't our tests really work indistinctly with mono or dotnet anyway?

        You see, this is what I don't grasp. With the Jython, you have this jython.jar and it really should (and I think does) work on just about any JVM. So, if you grabbed the IBM Java implementation instead of the Oracle one, I'm pretty sure that Jython doesn't care. And, actually, since Jython is a rather stagnant project, I think you'd have to have a very ancient JVM (at least 15 years old) to be unable to run it. And, since Oracle has been pretty good about maintaining backward compatibility, any JVM will will do, from JDK 5 (most likely , though I'm just guessing...) up to the just-released JDK 19. You type java -jar jython.jar and it will just work. (Though our build/test requires JDK 8 or higher.) But anyway, the tests that use jython do just work as long as you have the jython.jar in $HOME/bin.

        I mean, if the C# code we are generating is not very bleeding edge at all, it ought to build and run on Mono as well as any Dotnet that's not really really old. But I am reasoning more or less by analogy with how things work with Java, which, for all its imperfections, really does fullfill the write once, run anywhere promise pretty well. (Well, at least for non-GUI, I daresay...) And backward compatibiity is very very good. I mean, one can test on different Java versions for thoroughness sake, but it is very very rare that you would write code that runs on JDK version n and it doesn't also run on JDK version n+1 or n+2 or whatever.

        Well, anyway, looking on the bright side, I am (gradually) learning some stuff as a result of fighting with this, but really, I think this should be a lot smoother. I cannot fathom how the automated test works on the server but I can't get it working on my local box (on any system, Windows, Linux or Mac!) to save my life seemingly.

        On my system:

        ~  $ ipy
        IronPython 2.7.11 (2.7.11.1000)
        [.NETFramework,Version=v4.5 on Mono 6.12.0.182 (tarball Tue Jun 14 22:35:00 UTC 2022) (64-bit)]
        Type "help", "copyright", "credits" or "license" for more information.

        I also (at some point) did:

        $ sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 3FA7E0328081BFF6A14DA29AA6A19B38D3D831EF
        $ echo "deb https://download.mono-project.com/repo/ubuntu stable-focal main" | sudo tee /etc/apt/sources.list.d/mono-official-stable.list
        $ sudo apt update
        $ sudo apt install mono-runtime
        $ sudo apt-install mono-devel

        The above are instructions from the Mono site. I'd forgotten about doing that, but checked my shell history.
        Note that I'm on the focal version of Ubuntu, so you may need to do something slightly different.

        Additional data point: on a fresh install of Ubuntu 22.04, I had to install mono-devel from the Mono site (following the above commands) to get everything upgraded to 6.12.x and following that, ant test-csharp completed successfully. I had to use their focal repository as they don't seem to have a version for jammy. I didn't need to edit any files from our repo.

          BTW I notice that the Python and C# tests have been broken for about two months following changes ... starting with this commit, I think.

            vsajip

            Well, I see that the problem there was with this #scan thing. In the commit that broke things, I tweaked it so it wouldn't generate a Node subtype called "scan", which it shouldn't. You see, #scan is a special tree-building annotation that does not actually mean that we build a node called scan, but rather that the production only exists for lookahead purposes AND consequently, it (quite obviously) does not participate in tree-building. Or, alternatively, maybe a better way of expressing things is that it's like #void except it also has the additional meaning that we never actually create a parsing routine, only lookahead. (Of course, you could never call a node subtype "node" or "abstract" for that matter, since these are reserved words in Java... but "scan" is not.)

            Well, I guess something in the python/C# code was not adjusted for that somehow. Maybe you could look into that. It surely is that. See: https://github.com/javacc21/javacc21/actions/runs/3037528432/jobs/4890083076#step:16:140

            So we're getting a bunch of errors like:

             [exec] /tmp/javacc-csharp-test-t3giquo9/java/cs-javaparser/Parser.cs(1859,13): error CS0246: 
             The type or namespace name 'scan' could not be found (are you missing a using directive or 
             an assembly reference?) 
             [/tmp/javacc-csharp-test-t3giquo9/java/cs-javaparser/org.parsers.java.csproj]

            Well, I'm sure that's really easy to fix, but much more so for you!

              revusky you could never call a node subtype "node" or "abstract" for that matter, since these are reserved words in Java

              Is node a reserved word in Java? I'd thought not. Is that something new? Of course, I know abstract is.

                vsajip

                Okay, thanks. I'll give it a try pretty soon. I was rather absorbed with some other things today.

                Even on Ubuntu 22.x, the mono that is installed by default, is 6.8, not 6.12. So, it needs 6.12 then? (Or something after 6.8 presumably...) But hold on, which mono is it running on the server for github workflow tests? If it's just installing ubuntu with its default mono package, wouldn't that be 6.8 at most?

                But I have to admit I find all this confusing. I just looked here: https://www.mono-project.com/docs/about-mono/releases/ and mono 6.8 is not that old, released 15 January 2020. Just offhand, that seems like it should be new enough for our purposes.

                Also, I realize, from that page, that Mono was putting out new releases as recently as June of this year, which is when 6.12 was release. This led me to wonder whether Microsoft's dotnet for Linux is really just a rebranded Mono anyway? (Is that the case?) Maybe dotnet for linux/mac has the same relationship with Mono that the Google Chrome browser has with the open-source Chromium.... i.e. it's kinda the same thing... (N.B. I'm just thinking aloud. I don't really know...)

                I also learned (in a curiosity killed the cat diversion) from his Wiki page that Miguel de Icaza was working for Microsoft from 2016 to early 2022. Funny diversion from that diversion was to learn that already, back in 2009, none other than Richard M. Stallman had labelled de Icaza as "A Traitor to the Free Software Community".

                And you know what we do to traitors! (Evil laugh.)

                Some possible answers:

                1. We knee-cap them.
                2. We waterboard them.
                3. We "whack" them like the Joe Pesci character in Good Fellas
                4. We don't do anything at all to them. (Except maybe call them nasty names and that's only RMS...)

                (I wonder who can guess the correct answer...)

                Well, that's just a funny (maybe) aside. It really seems like we ought to generate fairly low common denominator C# code that can run on any dotnet/mono that is not totally ancient. As I said, the Java code that we generate builds/runs on JDK 8 or later. JDK 8 is March 2014. JDK 8 introduced some very nice things, like lambdas and method references and the stream API. I can't really see much that came after that that is really must-have (though I could change my mind on that, not sure...) so it seems like having any generated code being runnable on JDK 8 is a reasonable thing to keep to for the indefinite future.

                Just looking at the version history of C# I think a similar sweet spot could be about C# 6.0 (which I think was released at some point in 2015). So, I reckon that if we stuck to that, any C# code we generate really should be able to build/run on any mono/dotnet that's not really ancient.

                But I think we really do need to go through the extra 9 yards that the whole thing is really seamless from a newcomer's viewpoint. Actually, I have an idea related to that, that I'm going to write up separately in the next day or so. Not just about C# specifically, just a general idea really.

                  vsajip

                  Sorry, I misspoke. I must have been thinking about #interface. And actually, #node isn't used specially in that spot. So, yeah, we currently have #void, #abstract, #interface, and #scan. And of those, only "scan" is not a reserved word in Java. I wrote about #abstract and #interfacehere.

                  The #scan thing doesn't even have its own blog article. It's just described here under the heading "Some other Odds and Ends".

                  So I wrote that as a little addendum in the blog article that describes assertions!

                  revusky that seems like it should be new enough for our purposes

                  The .NET world moves a lot faster than Java. As a consequence, things sometimes break.

                  revusky But hold on, which mono is it running on the server for github workflow tests?

                  I updated the test to print the ipy version, which also prints the Mono version it's built with. Looks like 6.12.0.182.

                  The GitHub runners might well have different packages to "stock" Ubuntu. I myself use Linux Mint, and there are slight differences from Canonical's Ubuntu (for example, Mint is not wedded to Snap as a distribution format, AFAICT).

                  Thanks for the pointer re. scan.

                  Here is a bit more info. On a mac (my mac anyway) IronPython seems to be quite determined to run on Mono.

                  I have Mono installed (6.12.0.113) and it appears that this has an IronPython bundled with it already. If you run ipy on the command line, it is running:

                  /Library/Frameworks/Mono.framework/Versions/Current/Commands/ipy

                  which, as you can see, seems to have been included with Mono.

                  That script, which I don't think I ever edited myself, is:

                  export IRONPYTHONPATH=/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/
                  exec /Library/Frameworks/Mono.framework/Versions/6.12.0/bin/mono  
                  /Library/Frameworks/Mono.framework/Versions/6.12.0/lib/ironpython/ipy.exe "$@"

                  So, it runs the version of IronPython that is included with Mono itself. But here is an interesting little fact. If you run that ipy script, you get:

                  IronPython 3.0 (3.0.0.0) on .NET 4.0.30319.42000
                  Type "help", "copyright", "credits" or "license" for more information.

                  So, it claims that it is running an IronPython labelled 3.0.0.0 (that is the one included in the Mono installation) and says it is running on a .NET 4.0.x, but it is clearly running on Mono, no? But it is reported as a .NET 4.0.x.

                  But if you try to run that IronPython that comes with mono using the official dotnet, it gives you:

                  Cannot use file stream for [/Library/Frameworks/Mono.framework/Versions/6.12.0/lib/ironpython/ipy.deps.json]: No such file or directory
                  A fatal error was encountered. The library 'libhostpolicy.dylib' required to execute the application was not found in '/Library/Frameworks/Mono.framework/Versions/6.12.0/lib/ironpython/'.
                  Failed to run as a self-contained app.
                  - The application was run as a self-contained app because '/Library/Frameworks/Mono.framework/Versions/6.12.0/lib/ironpython/ipy.runtimeconfig.json' was not found.
                  - If this should be a framework-dependent app, add the '/Library/Frameworks/Mono.framework/Versions/6.12.0/lib/ironpython/ipy.runtimeconfig.json' file and specify the appropriate framework.

                  I have to think that if one does address the problem of putting in the files it is asking for, then it would probably work. Haven't tried though...

                  And then I have a separate IronPython installation (not being used above) that is at:

                   /Library/Frameworks/IronPython.framework

                  I can run the IronPython version there, via:

                   mono /Library/Frameworks/IronPython.framework/Versions/2.7.12/bin/ipy.exe

                  And I get this:

                  IronPython 2.7.12 (2.7.12.1000)
                  [.NETFramework,Version=v4.5 on Mono 6.12.0.113 (2020-02/4fdfb5b1fd5 Mon Dec  7 12:18:53 EST 2020) (64-bit)]
                  Type "help", "copyright", "credits" or "license" for more information.
                  >>> 

                  Here it tells me that it is running Mono and what version. The IronPython that is bundled with Mono just says it is running on Dotnet version 4.0.x. not mentioning Mono!

                  I find the whole situation rather shoddy, to be honest. It seems to me that, by analogy with Jython, IronPython should run out-of-the-box with any Dotnet version, whether it is Mono or the one from Microsoft. And this business that you do a separate IronPython install (which I must have done at some point) but it still runs the one that was bundled with Mono... They really don't seem to have done the extra incremental work to make this work sensibly out-of-the-box.


                  4 months later

                  vsajip
                  Okay, well, here I am resuming this conversation after a 3 month hiatus.

                  This whole situation with the tests for python and csharp (but particularly for csharp) has certainly caused me a fair bit of pain/discomfort over the last few months. And I've decided that this is a problem that must be resolved. And I see little option but to resolve it by taking a rather autocratic sort of "executive decision".

                  Effective, as soon as possible, we will NOT be using either Jython or IronPython, or Mono, for that matter, in this project.

                  Any use of Python should be the standard or canonical CPython only, and then only for the Python generating branch of the project.

                  But, I mean, look, as regards the Csharp tests, I took at least a couple of runs on getting it working on my own box and just never succeeded -- not on Mac or LInux, but not even on Windows. A couple of weeks ago, I recruited the help of an online acquaintance to try to help me get the Csharp tests working locally. This guy has quite a bit of experience with Dotnet. Well, I figured that surely I was just missing something obvious and he could figure it out quickly.

                  Result: This guy also could not get the tests working locally.

                  But there was a key moment in that whole interaction. After at least an hour of mucking with this thing (I'm sure it was over an hour, but you know, one loses one's notion of time when mucking with all these magical incantations on the command-line...) but anyway, after various attempts to get this all working, I had a sort of epiphany because the guy asked me the following question: "What is this ipy thing?"

                  And then, of course, I answered. I told him: "Oh, that's IronPython, which is a version of Python that runs on dotnet... analagous to Jython, which is a version of Python that runs on the JVM...."

                  Come to think of it, in retrospect, I don't think he knew what Jython was either. Not sure about that, but he definitely did not know what ipy was. So, hopefully, you get my point. I don't know exactly how to put it, but I would say that for most people, these are very sort of weird, exotic things. I mean, aside from the fact that we need these tests for our own purposes, we also need to provide very clear examples of usage. I mean, so finally, I just decided, for various reasons, that our test harness for running a Csharp example MUST be written in Csharp. Our test harness for running a parser generated in Java MUST be in Java -- not in Jython (or JRuby or Kotlin or.... The test harness for a parser generated in Python MUST be in Python -- but again, we only use the canonical CPython.

                  Really, it doesn't matter that you can express whatever test harness code a bit more tersely and elegantly in some alternative language for the JVM, or Dotnet... I'm not saying that what I did was perfect or anything, but I think we should have something for Python and CSharp that is much more similar to the the little test harnesses I wrote to invoke the parsers generated in Java -- JParse.java, CSParse.java, or PyTest.java respectively. Granted, those files are a bit repetitive and it's in the back of my mind to fix it up a bit, BUT... the point here is that at least those things provide a fairly clear example (I think) of how to invoke the parser. Another thing, by the way, is that they provide a main() method that you can use as the entry point to fire up the thing in a debugger. I'm not saying, by the way, that it's impossible to get the code working in a debugger if it is invoked from Jython or IronPython. I'm sure it's possible, but it's far less obvious!

                  Well, I guess I've said what I meant to say. I would just add that I feel that this is part of a larger whole, which is that we do need to get much more... I guess the word might be purposeful... about setting certain objectives and systematically getting there. Well, I guess the broad objective I'm thinking about mostly is just getting this out there into these communities -- Python, Dotnet/CSharp... -- to announce the tool and have a reasonable expectation that somebody with some curiosity could get in there and get the examples working in short order. And I would have to say that, for me, it is really quite frustrating that we have had the Python and CSharp code generation working basically for over a year and we have, as far as I know, ZERO users. Well, it's not that we have widespread adoption on the Java side either, and we need to address that, but I'm sure there are some users. I recall vaguely that, back in the FreeMarker days, I estimated that, at most 1 in 30 end-users ever showed up in our community and said boo. But, well, I guess different people are motivated by different things. You put a very considerable work into getting the CSharp and Python code generation working. To do that and then not put in the incremental work to get this out there and get some people actually using it.... Well, whatever, but speaking for myself, I really want to get to that point reasonably quickly. Now that we have the machinery done for a polyglot project, and it looks feasible to get INJECT working for Python and CSharp pretty soon, I really want to get to the point where we can scream this from the rooftops. Maybe by the end of the summer at latest. Hopefully sooner...

                  But, look, I don't want to argue about this IronPython/Jython question. I've decided that we're not going to use those things in this project moving forward. At least not for now... So I'd really like you to address this ASAP.