Help! For the life of me, I cannot run the C# tests locally

revusky

(I put this out there for the general public but I know the only person who is likely to help me on this is Vinay.)

I simply cannot get the C# tests to work locally. Not on Windows (where it should be easiest) or on Linux or Mac. I must be missing something or other. I have the latest dotnet installed on Windows. I have mono installed on Mac and Linux. I have Python3, Jython 2.7, and IronPython installed (on each respective box) and I think the script can find them.

On Windows, I get:

   [exec] Traceback (most recent call last):
   [exec]   File "ptest.py", line 95, in <module>
   [exec]     clr.AddReferenceToFile(dll)
   [exec] IOError: [Errno 2] Could not load file or assembly 'netstandard, Version=2.1.0.0, Culture=neutral, 
   PublicKeyToken=cc7b13ffcd2ddd51' or one of its dependencies. The system cannot find the file specified.

I must be missing something. On mac, I get:

 [exec] Traceback (most recent call last):
 [exec] Failed: C# lexer test run failed. 
 [exec] SystemError: Operation is not supported on this platform.Traceback (most recent call last): 
 [exec] SystemError: Operation is not supported on this platform.

That's a crappy error message, of course. No line location info. What operation is not supported on this platform?

mono --version returns this:

Mono JIT compiler version 6.12.0.113 (2020-02/4fdfb5b1fd5 Mon Dec  7 12:18:53 EST 2020)                                 
Copyright (C) 2002-2014 Novell, Inc, Xamarin Inc and Contributors. www.mono-project.com                                         
TLS:
SIGSEGV:       altstack                                                                                                 
Notification:  kqueue
Architecture:  amd64                                                                                                    
Disabled:      none
Misc:          softdebug                                                                                                
Interpreter:   yes
LLVM:          yes(610)
Suspend:       hybrid 
GC:            sgen (concurrent by default)

BTW, I got the python tests working by putting a jython.jar in my $HOME/bin directory and also creating a $HOME/logs directory (I didn't have one) and that seems to do the trick. But I can't figure out how to get the C# tests working.

vsajip

Hmmm ... there are a lot of different .NET versions around and lots of weird incompatibilities between them. I'll see if I can find out what the problem is, but note that the tests in GitHub actions run OK, and they start with a new VM and do all the steps needed. Weirdly, you can get more problems on Windows because there are two families of .NET which might be found on there - the original .NET which was Windows-only, and the newer stuff which is .NET Core, intended to be cross-platform across Windows, Linux and macOS. I would look at the contents of

https://github.com/javacc21/javacc21/blob/master/.github/workflows/csharp-tests.yml

and try and replicate those steps in your environment. You shouldn't have or need Mono as far as I know, but it's worth checking if it's installed by default on GitHub runners. (I didn't check; apart from a different procedure on macOS for installing IronPython, Linux and macOS test steps should be the same.)

Also note that the latest .NET version is 7.x, but the tests are for 5.0.x and 6.0.x only. You might want to clean out all .NET related stuff, and try with .NET 6 (which is what I run on my Linux dev box - I don't run tests on Windows or macOS other than via GitHub Actions).

revusky

vsajip

Hi, thanks for the prompt response. Also, I'm sorry that I've been so quiet for the last couple of months. I was actually traveling for over a month and just got back on Thursday. I actually believed that I was going to get some more work on this done while I was on the road, but it was hard to concentrate for various reasons.

You know, I have to admit that your reference to this was kind of a "Duh" moment for me, because I have to admit that I was unaware of that! I don't know whether I noticed that there was this .github folder or if I did notice it, curiosity did not kill the cat, as I never looked into it. But anyway, I mean I did not even know that this is where those tests were specified!

And, as for mono, I actually somehow believed that the tests on Windoze used the dotnet from MS, but the ones on Mac/Linux used mono. Don't ask me where I got that idea from! Actually, as regards mono, what is up with that anyway? I guess that project (originator, as I recall, was MIguel de Icaza, a pretty estimable guy, the main force behind Gnome) may be kind of abandoned. I mean, if MS themselves are supporting an up-to-date dotnet on Windows/Mac/Linux, what is really the point of Mono? Well, I guess if you're utterly obsessed with everything you're using being open-source, then... though, for all I know, dotnet is under some sort of open-source license nowadays.

So, I'm going to try again, I guess.

Oh, I did install dotnet 7.0 on my Windows partition. But Dotnet 7 is very recent, released in the last week even, but surely any tests that work with versions 5 or 6 should also work using 7, no? Could you check that?

Oh, and as a final point, I am loath to tell anybody how to do anything when they're the ones doing the work, but is it really worth it to have this test harness in Python -- I mean especially this rather baroque disposition, where the entry point is regular Python and it then spawns off (depending on the case) Jython or IronPython? I understand that you have a strong preference for coding in Python and I can even see why, but still, if the test harness was just written in pure Java, maybe the whole thing would be simpler. I mean, even taking into account the greater readability/terseness of Python, it's not like we really should be spending that much time with this test harness code.

Meanwhile, I'm just going to get rid of Mono on all my machines and just put in dotnet. My preference would be to just install the latest version. It really seems to me that the latest dotnet (v7) should work for this, so could you check that?

vsajip

revusky Actually, as regards mono, what is up with that anyway?

A bit of a chequered history. It started off as a FOSS alternative implementation for .NET but has been overtaken by MS' own .NET Core initiative. Meanwhile, a lot of FOSS stuff was built using Mono and may not have been repackaged for use with .NET Core. I use .NET Core pretty much exclusively except for legacy projects. There are various "standards" which MS has come up with, but they keep falling by the wayside (for example, see this page for information about .netstandard, which was referenced in your error message).

vsajip

revusky but surely any tests that work with versions 5 or 6 should also work using 7, no? Could you check that?

Will do in due course, but I would expect it to work without any changes (for example, 5.0.x -> 6.0.x didn't require any changes IIRC).

vsajip

revusky but is it really worth it to have this test harness in Python -- I mean especially this rather baroque disposition, where the entry point is regular Python and it then spawns off (depending on the case) Jython or IronPython?

It's worth it to me, as Python is a lot more expressive than e.g. Bash, PowerShell or Java/Kotlin/Groovy (and I've used all those). I'd rather spend a lot less time writing test harness code! I can definitely iterate a lot faster than with any other framework.

revusky

vsajip

I still really don't know very much about C# and Dotnet. Probably even less than what I know about Python! (Though I'm trying to remedy both those things...)

What I find confusing is that there is, as you say .NET Core and so on, but there's also just the basic fact that there are separate version numberings for the C# language itself, as opposed to the Dotnet package, so, like, if you look at the Wikipedia page, there is, let's say, for C# 6.0, this corresponds to Dotnet Framework 4.6, Dotnet Core 1.0 (or 1.1) and their IDE support for that language begins with Visual Studio 15 (which is another numbering!). (At least in Java land, we just talk about JDK 8 or 11 or whatever and that's it!)

That wiki page still refers to C# 11 as a preview, but that is no longer true, as of some days ago. C# 10 is still pretty bleeding edge really. It was released 1 year ago, November 2021, is supported in Dotnet 6.0 (and 6.01) and their IDE support begins in Visual Studio 2022, a.k.a. version 17, which was released in April of this year, so their IDE support lagged the availability of the core package by 6 months apparently. (Does the latest version of Visual Studio support coding in C# 11.0, I wonder?)

Now, in terms of what we're doing, it seems to me that there are various distinctions to keep clear. In terms of the C# code that the tool generates, it probably should generate code that is not particularly bleeding edge, i.e. the parser/lexer we generate should not use any C# language features beyond... well, I would say C# 6.0, I guess... (Something like that, I'm not sure, but we should probably make an "official" decision on that. So, in that case, going by the chart I pointed to, we would be saying that any parser generated by Congo in C# should run on Dotnet Core 1.0 or later, say. (Or alternatively Dotnet Framework 4.6.))

BUT... in terms of the C# parser that we are bundling with the tool, that should be able to consume pretty much the latest version of the language. By analogy with the situation with Java, the Java code we generate should (and does) build and run on JDK 8 or later. And I don't see much reason for that to change. However, the Java parser built in consumes pretty much the latest Java. I just looked at the last two JDK releases, 18 and 19, and the latest language feature, matching "patterns" in a switch statement is still a "preview" feature, as it was in JDK 17, and the current parser does not handle that construct. Most likely, the next cycle, it will be considered "stable", so I'll feel an onus to get that working. I was pretty confident that we had C# 10 working but now I updated the Roslyn project from Github and it's failing to parse 14 source files (out of 11,120 files). So maybe they have started using some new things in C# 11 internally in that project. It's probably pretty easy to get those 14 source files parsing. (Hint, hint.)

I'm still pretty confident that the Java, Python, and C# parsers we have can parse the overwhelming majority (like at the 99.999% level at least!) of the code out there in the wild. Of course, if the project was neglected for some years, that percentage would gradually slip downwards. But, for now, we're still pretty good. (The frustrating aspect of all this, of course, is that, unfortunately, not many people know about this!)

revusky

vsajip It's worth it to me, as Python is a lot more expressive than e.g. Bash, PowerShell or Java/Kotlin/Groovy (and I've used all those).

Well, I get that. I'd probably feel the same way if I knew Python half as well as you do! The countervailing problem, though, is that, if you look at this from the POV of a newbie just coming into this project and wanting to get involved (and I know that doesn't really happen too often) you're raising the bar somewhat. When the only target language was Java, all a full build/test required was the JDK and Ant and that's it. And it's really pretty hard to imagine that any Java developer wouldn't already have that, so they could just check out the code, type:

 ant test

And it would just build everything (the tool itself and then the various examples) and test it. Well, that's still the case, except that that it only builds and tests the parsers in Java. The parsers that we can build in C# or Python are not tested. But, you know, really, if we wanted the three languages to be equal first-class citizens (which I consider a goal for CongoCC anyway) then we would have:

 ant test-java
 ant test-csharp
 ant test-python

And I suppose a plain ant test would just run all three of the above. And I would say that we really want this to be hassle free.

I'd rather spend a lot less time writing test harness code!

Well, again, that's understandable, but I do feel I have to ask: how much time do you anticipate spending on that anyway?

Well, look, here is an idea. Why don't you work up something along the lines of a Configure script that just goes through a checklist of what you need to run the tests -- regular python (above a certain version number), dotnet (above a certain version number), Jython, IronPython. Have the script check whether it can find all these things locally, and when it can't, just point to where/how to grab the thing (based on the platform it's running on, of course.) In principle, if the script goes through the checklist and verifies that you have what is needed, then the tests should run.

revusky

vsajip
Okay, well, here I am resuming this conversation after a 3 month hiatus.

This whole situation with the tests for python and csharp (but particularly for csharp) has certainly caused me a fair bit of pain/discomfort over the last few months. And I've decided that this is a problem that must be resolved. And I see little option but to resolve it by taking a rather autocratic sort of "executive decision".

Effective, as soon as possible, we will NOT be using either Jython or IronPython, or Mono, for that matter, in this project.

Any use of Python should be the standard or canonical CPython only, and then only for the Python generating branch of the project.

But, I mean, look, as regards the Csharp tests, I took at least a couple of runs on getting it working on my own box and just never succeeded -- not on Mac or LInux, but not even on Windows. A couple of weeks ago, I recruited the help of an online acquaintance to try to help me get the Csharp tests working locally. This guy has quite a bit of experience with Dotnet. Well, I figured that surely I was just missing something obvious and he could figure it out quickly.

Result: This guy also could not get the tests working locally.

But there was a key moment in that whole interaction. After at least an hour of mucking with this thing (I'm sure it was over an hour, but you know, one loses one's notion of time when mucking with all these magical incantations on the command-line...) but anyway, after various attempts to get this all working, I had a sort of epiphany because the guy asked me the following question: "What is this ipy thing?"

And then, of course, I answered. I told him: "Oh, that's IronPython, which is a version of Python that runs on dotnet... analagous to Jython, which is a version of Python that runs on the JVM...."

Come to think of it, in retrospect, I don't think he knew what Jython was either. Not sure about that, but he definitely did not know what ipy was. So, hopefully, you get my point. I don't know exactly how to put it, but I would say that for most people, these are very sort of weird, exotic things. I mean, aside from the fact that we need these tests for our own purposes, we also need to provide very clear examples of usage. I mean, so finally, I just decided, for various reasons, that our test harness for running a Csharp example MUST be written in Csharp. Our test harness for running a parser generated in Java MUST be in Java -- not in Jython (or JRuby or Kotlin or.... The test harness for a parser generated in Python MUST be in Python -- but again, we only use the canonical CPython.

Really, it doesn't matter that you can express whatever test harness code a bit more tersely and elegantly in some alternative language for the JVM, or Dotnet... I'm not saying that what I did was perfect or anything, but I think we should have something for Python and CSharp that is much more similar to the the little test harnesses I wrote to invoke the parsers generated in Java -- JParse.java, CSParse.java, or PyTest.java respectively. Granted, those files are a bit repetitive and it's in the back of my mind to fix it up a bit, BUT... the point here is that at least those things provide a fairly clear example (I think) of how to invoke the parser. Another thing, by the way, is that they provide a main() method that you can use as the entry point to fire up the thing in a debugger. I'm not saying, by the way, that it's impossible to get the code working in a debugger if it is invoked from Jython or IronPython. I'm sure it's possible, but it's far less obvious!

Well, I guess I've said what I meant to say. I would just add that I feel that this is part of a larger whole, which is that we do need to get much more... I guess the word might be purposeful... about setting certain objectives and systematically getting there. Well, I guess the broad objective I'm thinking about mostly is just getting this out there into these communities -- Python, Dotnet/CSharp... -- to announce the tool and have a reasonable expectation that somebody with some curiosity could get in there and get the examples working in short order. And I would have to say that, for me, it is really quite frustrating that we have had the Python and CSharp code generation working basically for over a year and we have, as far as I know, ZERO users. Well, it's not that we have widespread adoption on the Java side either, and we need to address that, but I'm sure there are some users. I recall vaguely that, back in the FreeMarker days, I estimated that, at most 1 in 30 end-users ever showed up in our community and said boo. But, well, I guess different people are motivated by different things. You put a very considerable work into getting the CSharp and Python code generation working. To do that and then not put in the incremental work to get this out there and get some people actually using it.... Well, whatever, but speaking for myself, I really want to get to that point reasonably quickly. Now that we have the machinery done for a polyglot project, and it looks feasible to get INJECT working for Python and CSharp pretty soon, I really want to get to the point where we can scream this from the rooftops. Maybe by the end of the summer at latest. Hopefully sooner...

But, look, I don't want to argue about this IronPython/Jython question. I've decided that we're not going to use those things in this project moving forward. At least not for now... So I'd really like you to address this ASAP.

vsajip

revusky then we would have:

ant test-java
ant test-csharp
ant test-python

I'm not sure I get your point - we already have those, right? for C# and Python, for example. Right now we don't need to run all the tests in one command (though of course you can add a test-everything (or whatever name) target that will run all the tests) because when something is pushed, all the tests are run on a GitHub server, and I get emailed if there are any failures. I don't need to wait for all the tests to run locally - e.g. if I'm working on C# templates I don't really need to run the Python tests.

If developers want to engage with the project as contributors, they'd need to understand Python and C# to be able to contribute to those parts - else just use the tool in Java, in which case they don't need to bother with those. As far as I'm concerned, the bar seems high enough anyway because of a general shortage of developers with the level of experience and interest to contribute meaningfully. I think lack of documentation is probably the biggest problem at the moment in terms of getting new contributors on board, not the fact that parts of the test harness need Python/Jython/IronPython. (AFAICT developers with just Java knowledge can contribute to the core tool just fine, they don't need to download or install the Python tools unless they want to - just ant test will do fine for them.)

revusky

vsajip As far as I'm concerned, the bar seems high enough anyway because of a general shortage of developers with the level of experience and interest to contribute meaningfully. I think lack of documentation is probably the biggest problem at the moment in terms of getting new contributors on board, not the fact that parts of the test harness need Python/Jython/IronPython.

Well, okay, you're right that, in terms of attracting developers, this has not been the issue. No, of course not. As for there being a shortage of people with the interest and skills... well, it's hard to say really, because regardless of that problem, even if there were enough people who would be interested, it doesn't matter if they don't even know about the project. For example, consider what I wrote towards the end of this blog post. In particular:

I will make one final point about all this now. People in this space really do need to understand that my work on JavaCC, the resuscitated JavaCC 21 project, is the only game in town. (By town, I mean JavaCC broadly speaking. If you want to get involved with ANTLR or something else entirely different, that is another matter, of course.) But if you like JavaCC, its basic approach, you find it more usable, and in particular, if you want to make the intellectual investment of understanding how the code works, so that you feel some sort of mastery over the system.... in that case, this refactored version of the code is the only feasible basis to build on.

I stand by what I say above. But, I mean to say, the whole question of whether there is truly such a shortage of people who would be interested in getting involved in something like this... well... I don't know... it's hard to say. I actually think that there are quite a few people out there who could be interested in getting involved in an open source project like this one. Occasionally somebody does seem interested in the (legacy) JavaCC project, shows up and starts asking questions that would seem to suggest they could be interested in doing something. Of course, once they check out the code and start looking at it, then maybe they change their minds! Or, just a couple of days ago, somebody was on the FreeMarker list talking about wanting to get involved in development. Of course, "Apache FreeMarker" is now a classic nothingburger project and anybody trying to get involved in that is bound to be wasting their time. Well, granted, I don't know how serious that guy is either...

But the thing is that there are a lot of people out there, students, I guess, or not students, like people who feel totally unchallenged in their day jobs, like they sit in some cubicle somewhere writing glue code, and feel they could do more, would like to sink their teeth into something more meaty. I think that exists. The problem is that for such people, it is not so clear where to turn.

But, getting back to the question at hand, well, as for the question of the bar being high, well, sure it is, but there is no need to make the bar higher by making it a bear to even get basic stuff running out-of-the-box. So I have to admit that I still have not managed to run the C# tests locally, on Windows or elsewhere.

I run into this (on Windows):

Traceback (most recent call last):
  File "ptest.py", line 95, in <module>
    clr.AddReferenceToFile(dll)
IOError: [Errno 2] Could not load file or assembly 'netstandard, Version=2.1.0.0, Culture=neutral, 
PublicKeyToken=cc7b13ffcd2ddd51' or one of its dependencies. The system cannot find the file specified.
Traceback (most recent call last):
  File "ptest.py", line 95, in <module>
IOError: [Errno 2] Could not load file or assembly 'netstandard, Version=2.1.0.0, Culture=neutral, 
PublicKeyToken=cc7b13ffcd2ddd51' or one of its dependencies. The system cannot find the file specified.

It feels like this must be something pretty trivial. But the thing is that something similar in Java, I suppose it would be one of these classpath sorts of things and I'm used to that and I would resolve it very quickly, but I just don't have the "culture" in dotnet. So, well, I don't really want to argue too much about this stuff. After all, you're the only person who did show up and make the intellectual investment of learning the code and so on, but I would put it to you basically... well, do things as you wish, but you really ought to go the extra 9 yards so that this works out-of-the-box quite hassle free.

vsajip

revusky well, do things as you wish, but you really ought to go the extra 9 yards so that this works out-of-the-box quite hassle free.

It's not always easy to debug these sort of problems ... this is one variant of the "DLL hell" problem that besets Windows. There are just too many possible combinations of installed third-party packages (or even Microsoft packages) that could lead to this type of error. Plus, Microsoft often breaks their own stuff when Windows updates itself, and that happens with monotonous regularity. I will take a look when I get the time, but can I ask - have you tried installing dotnet on Linux? If so, how did that go?

One thing you could try is to change the C# template file project.csproj.ftl:

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <TargetFrameworks>netstandard2.1;net5.0</TargetFrameworks>
  </PropertyGroup>
</Project>

Try removing the netstandard2.1; and rebuilding and seeing if the error persists.

revusky

vsajip Try removing the netstandard2.1; and rebuilding and seeing if the error persists.

Well, now there is a different error. (By the way, that also involved commenting out the two lines here: https://github.com/javacc21/javacc21/blob/master/ptest.py#L90-L91 since the dlls array is now empty.

But after that, I get:

Failed: Failed to import org.parsers.json at part org
Failed: name 'd' is not defined.

And I can see where that comes from. The message is here: https://github.com/javacc21/javacc21/blob/master/ptest.py#L110 but the problematic line is actually 105, I guess. The import is failing. But I have to admit that I feel pretty helpless in terms of figuring this out.

I'll try on Linux shortly and report back. I'm realizing how absolutely abysmal my knowledge of Dotnet is. Now, I always thought it was basically like Java, so the CLR is basically like the JVM. In Java land, I can package everything up into a self-contained jarfile and there's no shared library impedance sort of issue because everything is self-contained and the only classes it's using outside that jar are the ones in the standard library. Surely we ought to be able to build something like that for Dotnet that "just works". You know, it really seems to me that the whole "DLL hell" problem should not really exist so much in a world where a megabyte of RAM costs less than a penny. (And disk space is far cheaper than that even.)

revusky

vsajip

I now realize why I thought that we were using Mono on mac/linux. I just typed ipy on the command line on Ubuntu and it gives me:

IronPython 2.7.12 (2.7.12.1000)
[.NETFramework,Version=v4.5 on Mono 6.8.0.105 (Debian 6.8.0.105+dfsg-3.2 Wed Jun 30 05:34:49 UTC 2021) 
(64-bit)]
Type "help", "copyright", "credits" or "license" for more information.
>>>

It's launching /usr/bin/ipy which is a simple little script that is:

#!/bin/sh
/usr/bin/mono /usr/share/ironpython2.7/ipy.exe $*

So, clearly, I installed IronPython at some point in the past and it put in a launch script that used Mono (which I had also previously installed.)

I also do have Dotnet installed. If I type dotnet --version on the command line, I get back:

6.0.302

However, if I try to launch IronPython with dotnet instead of mono, i.e.

dotnet /usr/share/ironpython2.7/ipy.exe

I get:

Cannot use file stream for [/usr/share/ironpython2.7/ipy.deps.json]: No such file or directory
A fatal error was encountered. The library 'libhostpolicy.so' required to execute the application was not found 
in '/usr/share/ironpython2.7/'.
Failed to run as a self-contained app.
  - The application was run as a self-contained app because '/usr/share/ironpython2.7/ipy.runtimeconfig.json' 
was not found.
  - If this should be a framework-dependent app, add the '/usr/share/ironpython2.7/ipy.runtimeconfig.json' file 
and specify the appropriate framework.

Well, I guess I need to have that ipy.runtimeconfig.json in the same directory as the ipy.exe. So, what should that file look like? Actually, it also looks like I need ipy.deps.json and libhostpolicy.so

All of this give me a rather uneasy feeling because I have no recollection of ever installing Mono on Ubuntu. Am I in early stage Alzheimer's, or maybe I installed some other app that had Mono as a dependency and thus, it got installed... Well, actually, I think I did install Mono at some point, because I was curious and wanted to muck with it. (Though I never did...) Must be, it's the kind of thing I'd do, but I can't remember doing it...

vsajip

revusky since the dlls array is now empty

It shouldn't be, because the net5.0 version should have been built, even if the netstandard2.1 hasn't been. If there is no DLL, then imports will of course fail because we're trying to import stuff from the DLLs.

revusky Surely we ought to be able to build something like that for Dotnet that "just works"

That should be possible, I think, but you will typically have a generated parser as a separate "assembly" which is not quite analogous to a .jar because you can't combine them together - they are independent, executable binary constructs (.dll and .exe) and not as nicely organised/composable as Java classes are. (As an aside, Python can also be zipped up and applications/libraries shipped as .zip files).

Of course, the "DLL hell" problem is one of packaging / distribution. An application might be able to bundle all its dependencies neatly, but with libraries it's harder to achieve that, especially when dependencies keep evolving and their using software needs to keep abreast of changes.

vsajip

revusky I have no recollection of ever installing Mono on Ubuntu. Am I in early stage Alzheimer's

No, it's that IronPython was built using Mono and hasn't been rebuilt using .NET Core (don't know why, but it's a volunteer project, so you can't expect anything).

It's a bit of Yak shaving at this point, so unless you want to go down this rabbit hole, it might be easier to do what I do - develop on Linux and let the GitHub runner do the heavy lifting on Windows and macOS. Further down the line, we will of course probably need to dig into the causes of this kind of issue.

revusky

vsajip

Well, I spent a little more time trying to get this working on Linux (Ubuntu 22.04) and I have to admit that I'm totally confused. I went into the synaptic package manager and I uninstalled mono. A forced removal in which I uninstalled mono-runtime and everything else associated with it, including IronPython. I figured I would just try to go through the installation steps here: https://github.com/javacc21/javacc21/blob/master/.github/workflows/csharp-tests.yml

So then I did sudo apt install dotnet6 and then I figured I could install IronPython and it would use dotnet. I try to do the IronPython install as per the above YAML spec, i.e. https://github.com/javacc21/javacc21/blob/master/.github/workflows/csharp-tests.yml#L59-L60 and I get:

revusky@gram ~ % sudo dpkg -i ironpython_2.7.11.deb
Selecting previously unselected package ironpython.
(Reading database ... 300477 files and directories currently installed.)
Preparing to unpack ironpython_2.7.11.deb ...
Unpacking ironpython (2.7.11) ...
dpkg: dependency problems prevent configuration of ironpython:
 ironpython depends on mono-runtime (>= 5.12); however:
  Package mono-runtime is not installed.
 ironpython depends on libmono-posix4.0-cil (>= 5.12); however:
  Package libmono-posix4.0-cil is not installed.

dpkg: error processing package ironpython (--install):
 dependency problems - leaving unconfigured
Errors were encountered while processing:
 ironpython

This Debian package "thinks" that mono-runtime is a dependency of ironpython and won't install the latter if Mono is not there. So, what to do? I re-install mono-runtime and then the above command works and I've got ironpython 2.7.11 installed and then when I type ipy on the command line, I get:

IronPython 2.7.11 (2.7.11.1000)
[.NETFramework,Version=v4.5 on Mono 6.8.0.105 (Debian 6.8.0.105+dfsg-3.2 Wed Jun 30 05:34:49 
UTC 2021) (64-bit)]
Type "help", "copyright", "credits" or "license" for more information.
>>>

So it's running ironpython on Mono. And, as best I understand, the mono runtime is installed on ubuntu, at least this latest version, by default -- which would be why I have no recollection of ever installing it!

So, here's a question: when you type ipy on your development box, does it tell you that your IronPython is running on top of Mono, or somehow is it using Dotnet?

Of course, this leads finally to other questions, like: shouldn't our tests really work indistinctly with mono or dotnet anyway?

You see, this is what I don't grasp. With the Jython, you have this jython.jar and it really should (and I think does) work on just about any JVM. So, if you grabbed the IBM Java implementation instead of the Oracle one, I'm pretty sure that Jython doesn't care. And, actually, since Jython is a rather stagnant project, I think you'd have to have a very ancient JVM (at least 15 years old) to be unable to run it. And, since Oracle has been pretty good about maintaining backward compatibility, any JVM will will do, from JDK 5 (most likely , though I'm just guessing...) up to the just-released JDK 19. You type java -jar jython.jar and it will just work. (Though our build/test requires JDK 8 or higher.) But anyway, the tests that use jython do just work as long as you have the jython.jar in $HOME/bin.

I mean, if the C# code we are generating is not very bleeding edge at all, it ought to build and run on Mono as well as any Dotnet that's not really really old. But I am reasoning more or less by analogy with how things work with Java, which, for all its imperfections, really does fullfill the write once, run anywhere promise pretty well. (Well, at least for non-GUI, I daresay...) And backward compatibiity is very very good. I mean, one can test on different Java versions for thoroughness sake, but it is very very rare that you would write code that runs on JDK version n and it doesn't also run on JDK version n+1 or n+2 or whatever.

Well, anyway, looking on the bright side, I am (gradually) learning some stuff as a result of fighting with this, but really, I think this should be a lot smoother. I cannot fathom how the automated test works on the server but I can't get it working on my local box (on any system, Windows, Linux or Mac!) to save my life seemingly.

vsajip

On my system:

~  $ ipy
IronPython 2.7.11 (2.7.11.1000)
[.NETFramework,Version=v4.5 on Mono 6.12.0.182 (tarball Tue Jun 14 22:35:00 UTC 2022) (64-bit)]
Type "help", "copyright", "credits" or "license" for more information.

I also (at some point) did:

$ sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 3FA7E0328081BFF6A14DA29AA6A19B38D3D831EF
$ echo "deb https://download.mono-project.com/repo/ubuntu stable-focal main" | sudo tee /etc/apt/sources.list.d/mono-official-stable.list
$ sudo apt update
$ sudo apt install mono-runtime
$ sudo apt-install mono-devel

The above are instructions from the Mono site. I'd forgotten about doing that, but checked my shell history.
Note that I'm on the focal version of Ubuntu, so you may need to do something slightly different.

vsajip

Additional data point: on a fresh install of Ubuntu 22.04, I had to install mono-devel from the Mono site (following the above commands) to get everything upgraded to 6.12.x and following that, ant test-csharp completed successfully. I had to use their focal repository as they don't seem to have a version for jammy. I didn't need to edit any files from our repo.

revusky

vsajip

Okay, thanks. I'll give it a try pretty soon. I was rather absorbed with some other things today.

Even on Ubuntu 22.x, the mono that is installed by default, is 6.8, not 6.12. So, it needs 6.12 then? (Or something after 6.8 presumably...) But hold on, which mono is it running on the server for github workflow tests? If it's just installing ubuntu with its default mono package, wouldn't that be 6.8 at most?

But I have to admit I find all this confusing. I just looked here: https://www.mono-project.com/docs/about-mono/releases/ and mono 6.8 is not that old, released 15 January 2020. Just offhand, that seems like it should be new enough for our purposes.

Also, I realize, from that page, that Mono was putting out new releases as recently as June of this year, which is when 6.12 was release. This led me to wonder whether Microsoft's dotnet for Linux is really just a rebranded Mono anyway? (Is that the case?) Maybe dotnet for linux/mac has the same relationship with Mono that the Google Chrome browser has with the open-source Chromium.... i.e. it's kinda the same thing... (N.B. I'm just thinking aloud. I don't really know...)

I also learned (in a curiosity killed the cat diversion) from his Wiki page that Miguel de Icaza was working for Microsoft from 2016 to early 2022. Funny diversion from that diversion was to learn that already, back in 2009, none other than Richard M. Stallman had labelled de Icaza as "A Traitor to the Free Software Community".

And you know what we do to traitors! (Evil laugh.)

Some possible answers:

We knee-cap them.
We waterboard them.
We "whack" them like the Joe Pesci character in Good Fellas
We don't do anything at all to them. (Except maybe call them nasty names and that's only RMS...)

(I wonder who can guess the correct answer...)

Well, that's just a funny (maybe) aside. It really seems like we ought to generate fairly low common denominator C# code that can run on any dotnet/mono that is not totally ancient. As I said, the Java code that we generate builds/runs on JDK 8 or later. JDK 8 is March 2014. JDK 8 introduced some very nice things, like lambdas and method references and the stream API. I can't really see much that came after that that is really must-have (though I could change my mind on that, not sure...) so it seems like having any generated code being runnable on JDK 8 is a reasonable thing to keep to for the indefinite future.

Just looking at the version history of C# I think a similar sweet spot could be about C# 6.0 (which I think was released at some point in 2015). So, I reckon that if we stuck to that, any C# code we generate really should be able to build/run on any mono/dotnet that's not really ancient.

But I think we really do need to go through the extra 9 yards that the whole thing is really seamless from a newcomer's viewpoint. Actually, I have an idea related to that, that I'm going to write up separately in the next day or so. Not just about C# specifically, just a general idea really.

vsajip

BTW I notice that the Python and C# tests have been broken for about two months following changes ... starting with this commit, I think.

revusky

vsajip

Well, I see that the problem there was with this #scan thing. In the commit that broke things, I tweaked it so it wouldn't generate a Node subtype called "scan", which it shouldn't. You see, #scan is a special tree-building annotation that does not actually mean that we build a node called scan, but rather that the production only exists for lookahead purposes AND consequently, it (quite obviously) does not participate in tree-building. Or, alternatively, maybe a better way of expressing things is that it's like #void except it also has the additional meaning that we never actually create a parsing routine, only lookahead. (Of course, you could never call a node subtype "node" or "abstract" for that matter, since these are reserved words in Java... but "scan" is not.)

Well, I guess something in the python/C# code was not adjusted for that somehow. Maybe you could look into that. It surely is that. See: https://github.com/javacc21/javacc21/actions/runs/3037528432/jobs/4890083076#step:16:140

So we're getting a bunch of errors like:

 [exec] /tmp/javacc-csharp-test-t3giquo9/java/cs-javaparser/Parser.cs(1859,13): error CS0246: 
 The type or namespace name 'scan' could not be found (are you missing a using directive or 
 an assembly reference?) 
 [/tmp/javacc-csharp-test-t3giquo9/java/cs-javaparser/org.parsers.java.csproj]

Well, I'm sure that's really easy to fix, but much more so for you!

vsajip

revusky you could never call a node subtype "node" or "abstract" for that matter, since these are reserved words in Java

Is node a reserved word in Java? I'd thought not. Is that something new? Of course, I know abstract is.