Calling inline C routine

Users
41

I half solved it. When there is an extern block, and C has a pointer to data, it wants a c_ptrTo(D) where D is the data. So, I can half document it fully. In particular I can write my own routine

proc c_data_ptr_to(ref x : ?T) : c_ptr(T) return c_ptrTo(x);

The problem is what happens when the argument is a pointer to a function.

Firstly, I can find no Chapel example where a proc is passed as an argument to another proc. What am I missing in my search. Surley it exists, it is such a fundamental requirement in a language.

I only have half the answer. I need to be able to write a routine like

proc c_proc_ptr_to(p : proc(...)) : c_fn_ptr return c_ptrTo(p);

which would let me investigate what is going on. But that won't compile. No will

proc c_proc_ptr_to(p) : c_fn_ptr return c_ptrTo(p);

but that won't compile either. I cannot figure out what creates a c_fn_ptr because I know that
the routine c_ptrTo(P) does not. Its definition is just

inline proc c_ptrTo(x : c_fn_ptr) : return x;

which has me stumped.

Any pointers?

42

I am guessing there is some implicit conversion happeing. But I cannot see it.

43

Going back (near the start ofg this thread) to the need for static inline within an extern block. If I remove the inline, there is no need for static.

So, something is still missing/inconsistent in the rules. in the way you declare and spell ouit a routine in the said block.

44

Going back (near the start ofg this thread) to the need for static inline within an extern block. If I remove the inline, there is no need for static.

static and inline in this context are C business. You code will compile with Chapel IFF it compiles without Chapel. Generally speaking.

So, something is still missing/inconsistent in the rules

I am open to that. May I suggest creating a github issue asking to resolve the inconsistency?

45

Sorry for misleading you. My new understanding is that ref i:int and i:c_ptr(int) in the argument list of an extern function are equivalent in the C world. If you pass Chapel integer variables to your extern, you want to use the former. If you are working with C pointers, the latter.

I understand that this is inconsistent in your view. However I do not see us sacrificing one or the other for the sake of consistency.

[Added] Indeed, as you observe, a pointer formal argument within an extern block translates into a c_ptrTo in the Chapel land. However you can override that with an extern proc declaration outside of the extern block: 
use CTypes;

extern {
  static void printme(int* p) { printf("p -> %d\n", *p); }
}

extern proc printme(ref arg: c_int);

var xxx: c_int;

printme(xxx);           // works only with the above 'extern proc printme'
printme(c_ptrTo(xxx));  // works only w/o the above 'extern proc printme'
46

Overnight, one of the things you mentioned

static and inline in this context are C business. You code will compile with Chapel IFF it compiles without Chapel. Generally speaking.

I have been using C for a long time so I might have gotten into bad habits. And I work with too many versions of the language (including C23) that I sometimes do not know whether I am coming or going. So your suggestion that my C was not strictly correct could be valid. I could have succumbed to some bad habits. So, I took my C code from the block verbatim, fed it to CLANG and boom, it works with or without

static inline

So naked inline is perfectly legal C. However, within the extern block, you have to use a static to get it to compile. Something is at odds here. I have no problem with this situation but there is a need to spell out how to make things work in the documentation (or fix the underlying cause of the problem).

All I was trying to do at the start of this thread was get my extern block code working. Reading the Technotes as they stand is insufficient. Certainly in my case, without your help suggesting I use static, I would still be floundering around.

I want to try and get the Technotes to be clear, concise and complete.

47

All I was trying to do at the start of this thread was get my extern block code working.

My understanding is that you have accomplished that. Otherwise please discuss further.

I want to try and get the Technotes to be clear, concise and complete.

Thank you. It will be valuable for everyone! Given that interop is still a moving target, we will accept revisions that do not achieve completeness. :slight_smile:

So naked inline is perfectly legal C. However, within the extern block, you have to use a static to get it to compile. Something is at odds here.

I believe the odds here are the additional flags we pass to clang behind the scenes for stricter checking of extern blocks than what you normally get. See   https://github.com/chapel-lang/chapel/blob/main/compiler/llvm/clangUtil.cpp#L2388-L2394   — soon to be adjusted by Stop tying Clang flags for extern blocks to the --cc-warnings flag by dlongnecke-cray · Pull Request #20904 · chapel-lang/chapel · GitHub

Also, one C compiler may accept some code that another compiler will flag as an error.

Perhaps the technote could mention this additional strictness that we impose on the extern blocks. A similar consideration probably applies to .c and .h files, where the errors may depend on C vs. LLVM backend and whether chpl --cc-warnings is enabled.

48

Thanks Vass.

I am not worried about the inconsistency. I am not advocating sacrificing one or the other. However, in my humble opinion, to avoid confusion, you should only use and refer to the consistent way in the main body of the documentation . You can document the inconsistent one in the Appendix.

What you are saying is that IFF you ADD an extra definition

extern proc printme(ref arg: c_int)

can you use the simpler form

printme(xxx);

But the documentation explicitly says that no such extra definition is needed. Very very confusing.

In the new Technotes, I have avoided all mention of the ref in the main body of the documentation. That also keeps it consistent with the use of function pointers. I have then added an Appendix where I talk about adding the extra Chapel prototype as you have done to let you use a ref argument. That Appendix was already looking very similar to your explanation above. Almost word for word. Scarily so. Before I release the new version, I want to augment it with some stuff with regards to function pointers but will post some notes about that later on this weekend. Function pointers still have me a bit stumped at the moment, both technically and how to best to describe it clearly. I think they are being magically coerced in the background but I cannot find where or how.

Regards - Damian

49

I can find no Chapel example where a proc is passed as an argument to another proc. What am I missing in my search. Surley it exists, it is such a fundamental requirement in a language.

Your intuition is good! Thank you for trusting Chapel. Let me know if you have seen the technote on first-class functions:   Chapel Documentation — Chapel Documentation 1.33

var f = proc(x:int, y:int) { return x + y; }; // lambda a.k.a. anonymous function 
writeln(f(1,2));  // outputs: 3

proc multiplyFn(x:int, y:int) { return x * y; };

proc callMeGeneric(arg) {
  writeln("in callMeGeneric: ", arg(3,4));
}

proc callMeConcrete(arg: proc(x:int, y:int): int) {
  writeln("in callMeConcrete: ", arg(3,4));
}

callMeConcrete(f);
callMeConcrete(multiplyFn);

callMeGeneric(f);
callMeGeneric(multiplyFn);

I need to be able to write a routine like
proc c_proc_ptr_to(p : proc(...)) : c_fn_ptr return c_ptrTo(p);

Thanks to David L, who is leading the FSF effort, for helping me with this.

Chapel function types are not compatible with C function pointers represented by c_fn_ptr. To get the latter, please use c_ptrTo(Chapel function name) . The Chapel function must be concrete and not overloaded. The type c_fn_ptr does not specify argument/return types. We plan to support more robust C function types in the future.

extern {
  #include <stdio.h>  // works without it, too
  static void makeCall(int (*fn)(int)) { printf("%d\n", fn(4)); }
}

extern proc makeCall(fn: c_fn_ptr): void;   // works without this, too

proc foo(x: int): int { return x*x; }

proc main() {
//makeCall(foo);  // error: the actual argument type is
                  // 'proc(x: int): int', need 'c_fn_ptr'
  makeCall(c_ptrTo(foo)); // OK

  const fooPtr = c_ptrTo(foo);
  makeCall(fooPtr);  // works, too
}

[proc c_proc_ptr_to] won't compile

These look to me like limitations of the current implementation. If the ability to write one of these is important for your Chapel projects, kindly please open an issue to request this functionality.

50

In the new Technotes, I have avoided all mention of the ref in the main body of the documentation. That also keeps it consistent with the use of function pointers. I have then added an Appendix where I talk about adding the extra Chapel prototype as you have done to let you use a ref argument.

I like your idea of separating out ref. Looking forward to seeing your revision! I suspect that I will want ref to be a sub/section in the main body, not an appendix, as a somewhat first-class feature here, however we will see.

That Appendix was already looking very similar to your explanation above. Almost word for word. Scarily so.

Scary indeed!! (Halloween is coming...) In my defense, I probably copied my words from somewhere. :slight_smile:

I think they are being magically coerced in the background but I cannot find where or how.

When you pass a Chapel proc to c_ptrTo, the compiler handles that internally via PRIM_CAPTURE_FN_FOR_C, rather than calling into one of the overloads of c_ptrTo in CTypes.chpl. Edit: incorrect, the c_ptrTo(ref x: ?t) overload gets invoked. So, like your said, it is most likely that the Chapel proc types get converted to c_fn_ptr implicitly somewhere. I am not motivated enough to dig deeper into it.

This is of course an implementation detail. However, please let us know or reflect in Technotes if it may affect a Chapel user. One important note is that the Chapel function passed to a Chapel proc or to C via c_fn_ptr must not use outer variables that are not globals a.k.a. module-level variables.

51

I just saw your two most recent postings and have not digested them. But I will throw this document into the mix for now in case you are still awake.

I appreciate that a lot of this point stuff evolved over time and there is backward compatibility to consider. I apologize if what I am saying is controversial. Also my words have the benefit of hindsight.

In Interoperability, there are the special types:

c_fn_ptr
c_void_ptr
c_ptr(T) - indexable
c_array(T, n) - indexable
c_string - indexable (to be deprecated in favour of c_ptr(int(8))

There is a support routine c_ptrTo(D).

Some points:

  •   As this is Chapel, not Rust, 'fn' should really be 'proc'

  •   there is too much vaguery with c_ptrTo(Z) being used for both a proc
  and data. There really needs to be separate routines, even if at the
  low level, they are one and the same routine. It is highly confusing
  and makes no sense to me. It appears to be reliant on some casting
  magic which is scary.

  •   c_ptrTo(D) has a mix of nomenclature styles. As everything else uses
  underscores and no capitals, using ALL underscores makes more sense.

So it seems to this user that these types are should be called:

c_proc_ptr
c_void_ptr
c_data_ptr(T) - indexable
c_array(T, n) - indexable
c_string - indexable (to be deprecated in favour of c_data_ptr(int(8))

with the creation of a proc and data pointer from a proc P and data D, being done respectively with c_proc_ptr_to(p) and c_data_ptr_to(D)

This is consistent and clean and can be described easily and clearly.

I can create the c_data_ptr_to(D) routine but cannot figure out Chapel code for c_proc_ptr_to(P).

As the extern block would generate a

c_ptr (or which can be treated as my new c_data_ptr)

type for arguments passed by address, I suggest that the recommendation to use a 'ref' type be omitted from the main body of the Technote. It is inconsistent at best and confusing at worst.

In an Appendix, there can be some explanation of how to use a ref with an extern block, or without it, in both scenarios demanding an extra Chapel prototype but for slightly different reasons.

52

Vass, you mentioned

the `c_ptrTo(ref x: ?t)` overload gets invoked. So, like you said,
it is most likely that the Chapel `proc` types get converted to a 
c_fn_ptr implicitly somewhere. I am not motivated enough to dig
deeper into it.

This is of course an implementation detail.

Actually, if I can crack that one, I believe the whole C pointer variable issue becomes so, so much easier to explain and document. I think. I will try and wrap up the rewrite of the Technotes in the next few days. The crux of it is that I assume that I can say that where you need to pass bar, a Chapel function pointer, to a C routine foo, you just say

foo(c_proc_ptr_to(bar));

It is so easy to remember that if you want a pointer/ptr to a proc, you call c_proc_ptr_to.

Similarly,if you are passing data D of type T to a C routine foo by address, you just say

foo(c_data_ptr_to(D, T));

Again, it is easy to remember that if you want a pointer/ptr to data, just call c_data_ptr_to.

So simple - I think. And I know this second case works for an extern block, having already tested it.

You also mentioned

One important note is that the Chapel function passed to a Chapel
proc or to C via c_fn_ptr must not use outer variables that are not
globals a.k.a. module-level variables.

I think that this is a reasonable level of complexity to say is too hard to handle.

53

When I said Appendix, I was imprecise.

The concept of an export block appears quite late in the technotes. As was going to introduce a section Declaring Pass-by-Reference Arguments at this point. So it is in main body. Just not early on. I have not quite figured out all the words for that section yet.

54

Hi Damian,

Thank you for clarifying your point about "Appendix" for ref args. This sounds good to me.

You certainly have good suggestions about Chapel's interop. I suggest you open GitHub issues with those, to make sure your ideas are reflected well.

I'd like to argue with a couple of your ideas:

  • c_fn_ptr --> c_proc_ptr : this depends on your point of view. This type exists explicitly to describe things callable in the C world. Those things are "fns", not "procs". By analogy, "c_void_ptr" is gibberish in the Chapel world, as in Chapel there are no values of void type and no pointers at all. So "fn" in the name makes more sense to me.

  • c_data_ptr_to() and c_proc_ptr_to() : to me this is like you are proposing different names for the plus operator on ints vs. complex vs. strings. It is common in Chapel to overload a single name to mean similar operations on different argument types. The case of c_ptrTo or c_ptr_to falls squarely into this category in my view.

  • "c_ptrTo relies on some casting magic" -- indeed it does. Likewise "+" on integers relies on some bitwise magic, doesn't it? Regardless, let us say that we have implemented c_ptrTo() that does not rely on any magic. How would you document that function?

55

Thank you for clarifying your point about "Appendix" for ref args. This
sounds good to me.

Sorry for the original misunderstanding.

You certainly have good suggestions about Chapel's interop. I suggest you
open GitHub issues with those, to make sure your ideas are reflected well.

The ideas need refinement. I am only trying to document things at the moment.

That's for the feedback. My notes just came from some discussions with a few colleagues and some minor refinement.

I will respond with their feedback. I have rewritten this stuff using groff with parameterized names so I can push it back to the old names if they are cast in stone.

It seemed like there were 3 kinds of C pointers that had to be handled

  1. function pointers,
  2. void pointers, and
  3. data pointers.

When it came to the c_?????_ptr names, I was asked over several cups of coffee

Why not just use the first 4 letters of each to minimize abbreviations?

and I said "why not"? Hence, from a C perspective, these became

	c_func_ptr
	c_void_ptr
	c_data_ptr

When we changed to those names in the documentation, things were a lot more obvious and delineated. And everything had 4 letters so it read a lot better which is key to any documentation.

I unilaterally changed our consensus first name to proc - silly me. It is now reverted to func

We also could not understand the mix of CAPs and underscores in the name in

	c_ptrTo

As we figured that the c_ was not negotiable, we changed it to c_ptr_to. But then nothing coerces to a c_void_ptr, so we came up with

	c_func_ptr_to
and
	c_data_ptr_to

for two reasons. Firstly, it read just soooo much better. Also, we thought the former might need to be special later if it has to handle types for both argument the function's argument(s) and return value. In hindsight, that may not be a good idea.

I'd like to argue with a couple of your ideas:

Go for it.

c_fn_ptr --> c_proc_ptr : this depends on your point of view. This
exists explicitly to describe things callable in the C world. Those
things are "fns", not "procs". By analogy, "c_void_ptr" is gibberish in
the Chapel world, as in Chapel there are no values of void type and no
pointers at all. So "fn" in the name makes more sense to me.

Agreed, they are func[tions]. While first-letter/last-letter of a name is a valid way to abbreviate the name, we used the first 4 letters of the words data and void, so we choose the first four from function. Also, func is a lot more obvious in meaning to a first time reader who has not seen Rust.

c_data_ptr_to() and c_proc_ptr_to() : to me this is like you are
proposing different names for the plus operator on ints vs. complex
vs. strings. It is common in Chapel to overload a single name to
mean similar operations on different argument types.

Agreed - In hindsight, I agree that these two should not be different.

I have been overloading operators and functions for 25 years before I ever saw Chapel. Well longer than that if you add Algol68 experience with operators.

The case of c_ptrTo or c_ptr_to falls squarely into this category in my view.

Maybe. But really c_ptrTo seems like it was chosen by somebody who cannot decide between underscores and camelCase and wants a bet each way. We did a version of the document with the original name and then another with

c_ptr_to

The latter reads infinitely easier on the eyes and does not clutter up the interline space in the same way that a camelCase name does. We then screwed around with those two variants of that but as I said, we can forgot those.

"c_ptrTo relies on some casting magic" -- indeed it does. Likewise
"+" on integers relies on some bitwise magic, doesn't it? Regardless,
let us say that we have implemented c_ptrTo() that does not rely on any
magic. How would you document that function?

I am happy with casting magic, but not when I cannot understand it.

In my current rewritten 'Interoperability' Technotes, I have changed the name back to c_ptr_to and am happy to keep the single name. That means I can say

  c_ptr_to( <Chapel-function-name> )

coerces/evaluates to a C pointer to that Chapel-function-name, and similarly

  c_ptr_to( <Chapel-variable-name> )

coerces/evaluates to a C pointer to that Chapel-variable-name.

As for the document itself, I still need to rearrange some bits and add more examples. The latter includes your own examples of C function prototypes and declarations with a few extra words, in both extern { } block style and require style. and my own extern {} block style which has some embedded assembler and uses a less common but very well accepted #include file.

I will then do a PDF so people can read it like a real document and once I get something approximating consensus, I will then emasculate it back to Markdown and people can hack it to their heart's content.

But my day job takes precedence this week.

56

Hi Damian,

Good explanations, thank you.

I have not been replying to your arguments about the name c_ptrTo because we have something cooking. However your arguments are good and are worth a github issue. Unless we are about to switch to all-underscores already.

Re c_ptr_to(fn or data) -- a good way to document it is to explain what happens when I dereference the C pointer that it produces. For c_ptr_to(fn), I get a function I can call from C assuming I pass it appropriate arguments. For c_ptr_to(data), I get the contents of that Chapel variable. So I do not share your thirst for understanding the casting magic.

P.S. It is cool that your colleagues are joining forces to improve the documentation!

57

Keep me posted on what's cooking with c_ptrTo.

58

@damianmoz please see Proposed / potential changes to `c_ptrTo()` · Issue #18016 · chapel-lang/chapel · GitHub

Also somewhat related: Should / could `c_void_ptr` be replaced with `c_ptr(void)` · Issue #18011 · chapel-lang/chapel · GitHub

59

Thanks Vass. I am still working on my rewrite.

60

@damianmoz fyi a recent development on #18016: Proposed / potential changes to `c_ptrTo()` · Issue #18016 · chapel-lang/chapel · GitHub