ometasharp Forum Rss Feed

New Post: Blowing my mind...

justinc — Wed, 11 Nov 2009 03:38:39 GMT

Wow, thanks for the response. You've given me a lot to read and think about. I'll have to read about it more and decide whether or not to go the route of creating a new parser. Your words are encouraging however, which is dangerous because I think I'm starting to talk myself into it despite the time I know it will take.

I will have to look at your source code more closely as well.

New Post: Blowing my mind...

jeffmoser — Wed, 11 Nov 2009 03:10:39 GMT

Hi Justin,

When MGrammar was released a few weeks or so after I started OMeta#, I decided to stop work on it and I moved on to other things. The reason is that the language is the easy part and tool support becomes the hard part. This was just too much to do well on my own in my spare cycles. My hope was that MGrammar would get mainstream support for DSLs. I was willing to let some of OMeta's features go in exchange for Microsoft backing something similiar.

It's sort of sad to hear where Oslo is going. MGrammar had far greater potential than just being SQL Server specific.

I haven't been in the OMeta# code for over a year, but I'd like to point out a few things based off your questions:

The DLR and C# 4.0 wasn't realistically available for use when I wrote OMeta# initially. The use of these (especially C# 4.0's "dynamic" feature) would make the code simpler. It'd probably be best to mostly start over from scratch. You could be up and running in not too much time.
As mentioned in my initial blog post, OMeta handles streams of *anything* and does pattern matching on it. Streams of characters is just one kind. Note that optimizer is written in OMeta#. In that case, the input and output are a simplified AST. in this case, "ometa OMetaOptimizer<HostExpression> : Parser<HostExpression>" means that it is taking in (and producing) HostExpression(s) which you can think of as LISP cons cells (more details here). You could easily (in theory) write generators for any language from these. I picked C# and wrote a HostExpression/AST -> C# translator as an example. Note that this is done without needing explicit "visit"s as would be the case in a Visitor pattern implementation. See Alex's thesis for more details on why OMeta eliminates the need for a specialized Visitor pattern and does everything with a super pattern-matching concept.
As for your "sum" question, check out the Calculator example. In that case I do it two different ways. The first evaluates it directly:
AddExpr = AddExpr:x '+' MulExpr:y -> { x + y }

Here you see the result of matching AddExpr will be an Int32 and it'll be bound to "x" (the right hand side of a -> is what the pattern matches. The beauty of OMeta is that you can then use it directly in the semantic action. In fact, the OMeta parser itself is written in OMeta (that takes characters and produces an AST)

This is the case if you're taking in text and producing a value. The other approach is to create a AST like you are doing in your M example. I do this in the Calculator Parser example:
AddExpr = AddExpr:x '+' MulExpr:y -> { "add", x, y }

This is more or less exactly what you're doing in your example (the result is just a tuple/list/cons cell)
Grammar re-use is a huge concept in OMeta. You can extend other grammars or use the "foreign" keyword to "lend" your stream to another grammar. This makes it easy to build language mashups.

I put OMeta# under a very permissive license. If you read the my posts and Alex's thesis, you can go a long way and quickly surpass what I did here. One approach you could take is to rewrite the core of OMeta in C# 4.0 and use that as a basis for your MetaSharp. Altenatively, I can give you commit access here and you could have MetaSharp reference it.

Does that help clarify things?

New Post: Blowing my mind...

justinc — Tue, 10 Nov 2009 21:30:22 GMT

Ok I have to be honest. I was just reading Douglas Purdy's latest blog post (http://www.douglaspurdy.com/2009/11/10/from-oslo-to-sql-server-modeling/) about project "Oslo" and feeling very afraid that MGrammar wasn't going to turn out like I hoped. Probably mired in licensing or lost in some sql server modeling behemoth. That's ok I guess but now I am out looking for a replacement and I was doing so with low expectations. But here I find OMeta#, it looks different enough from MGrammar but also strikingly similar... :)

I'll try to give you a little background on where I'm coming from and then ask some questions, some MGrammar comparisons some just for my own understanding of OMeta#.

First off, I'm working on a related project here on codeplex called MetaSharp (http://metasharp.codeplex.com). Sorry for the similar name, it was an accident but don't be threatened! It's actually very complimentary rather than competitive. So far I've been using MGrammar to do all of my parsing and grammar authoring then everything that MetaSharp is concerned with is the before and after the actual parsing. In short it's a transformation engine. Transforming text into an AST is just one of those steps. This is what I would like to get out of OMeta#. I would like a way to parse arbitrary text and result in a tree of Nodes. The theory is, with standard AST nodes for common programming constructs you can build transformers to convert that AST into anything else (such as CodeDom objects or Linq Expressions or text etc.). The goal here is to enable arbitrary DSLs and scripting languages and, frankly, any type of language oriented tool.

So I have a few questions, I was poking around in the source code here (for example http://ometasharp.codeplex.com/SourceControl/changeset/view/17706#361393) and I see that your right hand projections appear to be arbitrary code (C# even?). So it seems that OMeta# merely generates code that matches patterns based on the left hand side DSL then generates the code on the right hand side? Plus the "ometa" keyword generates some type of specialized class? So what I see here is that you're generating an html string, so I'm assuming the Parser<char> indicates that the output of this parser is a stream of chars, otherwise known as a string? So If I changed this to Parser<Node> could I potentially create a sytnax such as:

Example -> { Push(new ExampleNode()) }

Additionally I see some things in your code such as:

InputDecl -> { CodeGen.Create()
                .Write("(")
                .Write("OMetaStream<", Get<string>("CodeType"), "> ")
                .WriteLeveledVar("inputStream")
                .Write(", out OMetaList<HostExpression> ")
                .WriteLeveledVar("result")
                .Write(", out OMetaStream <", Get<string>("CodeType"), "> ")
                .WriteLeveledVar("modifiedStream")
                .WriteLineEnd(")")
               },

Which actually concerns me a little, it goes a little counter to my reasoning. So for this question I just want to tell you what I feel is wrong with this, how I have tried to solve it with MetaSharp and I would basically just like to hear what you think about it.

What looks like is happening here is that you're doing a textual transformation directly inside of your grammar. To me the problem with this is that you end up with a grammar that is only good for one thing (and is kind of ugly). Ideally, I think you should generate AST nodes then use visitors to generate this text instead. The reason why I suggest this is because, to me, that makes the grammar more "pure" for one thing (it would be akin to separating a view from logic) and also it would allow you to write multiple transformers, so you could generate C# today and VB tomorrow for example. You could also push it through multiple transformation steps and you could also allow plugins to modify the AST (aspects?) etc. To me what would be much more valuable instead of arbitrary code on the right hand side is an AST declaration syntax, this was one thing that MGrammar did very well. I could be satisfied with the Parser<Node> described above though.

Next, suppose I wanted to define code to parse "sum(x, y)". How would I do this in OMeta#? Here it is in MGrammar you should be able to understand what I'm saying here.

sytnax sum = "sum(" a:Join(Identifier, ",") ")" => MethodInvocationExpression { Name => "sum", Arguments => a };

syntax Join(a, b) = a => [a] | a b c:Join(a) => [a, valuesof(c)];

[...] implies a collection of results, "valuesof" in this context simply flattens the collection so you don't have a bunch of nested items.

Also, what sort of support do you have for grammar reuse? In addtion to AST reuse I want to promote Grammar reuse. Specifically, in an external DSL, it's really common to write something very declarative but then add some support for BinaryExpressions or other common general purpose language constructs. And it turns out that BinaryExpressions are really hard! I would love it if, when writing a new DSL, I could start from a place where I could pull in common things such as binary expressions or if / else statements etc. without having to use everything else too (opt in). Also, are there any namespacing/import grouping mechanisms for your grammars?

I have more but I'll try to pace myself for now. This is exciting!

New Post: Brain dump

jeffmoser — Sun, 28 Sep 2008 19:39:21 GMT

Thanks for the update Nick! (as well as the praise for OMeta. Alex and friends have made a very interesting language)

I agree that negative matches are confusing at first. The trick to realize is that "~SomeRule" does not, as commonsense might first lead you to believe, match any rule but "SomeRule." To understand the core of language on this point, you need to look at what "~~SomeRule" means. A "~~" or "not not" means a "lookahead" assertion (see the original OMeta paper for more on this). Thus, "~~SomeRule" means "verify that the SomeRule is what follows, but don't actually consume the input. Likewise, "~SomeRule" means "verify that SomeRule is not what follows", but don't actually consume any input.

You've keyed in on a big design issue when you discuss lists and strings. It is quite confusing. The pain you experience is my possibly poor decision to make a list of characters and a string to be roughly equivalent in OMeta# just like they are in OMeta/JS. At first, I forced grammar writers to say "Sugar.Implode(listName)" to take a character list and convert it to a string, but then I decided this was the common case, so I made it automatic. I can revisit this decision if it's causing problems that can't be worked around. Let me know specifics if you get stuck.

I don't think that moving to XML would necessarily make things easier. OMetaList is fairly flexible as a recursive data structure if you need it. It's the data structure that the optimizer uses to optimize parsed OMeta/C# code.

Let me know what you would find helpful in a "quick reference" or ideas for future example programs and I'd be happy to work on those if that'd be helpful.

Keep up the great work and questions!

Jeff

New Post: Brain dump

nblumhardt — Sun, 28 Sep 2008 18:19:29 GMT

Some thoughts after spending a bit more recreational time with OMeta#...$0$0$0$0It is easier to build a parser with OMeta than anything I've attempted to use before - unambiguous, contextual matching is so nice!$0$0$0$0$0OMeta# is getting easier to use. It will be a lot more approachable with some better support for error messages, and a syntax "quick reference" but these things take work I know... All in good time :)$0$0$0$0$0The biggest points of confusion for me so far have been:$0$0$0$0Negative matches - ~blah just doesn't seem to work for me :($0$0Confusion between 'strings' and 'lists' - i.e. Foo+:f ends up spitting out a list, rather than a string of n Foos - yet trying to manipulate f in the transformation section to create a string gives me errors because it is cast in the generated code to a string :) - the result is output that looks like [h, e, l, l, o]$0$0$0I think perhaps looking back I would have been better off not specifying that the Markup grammar produces a string, and instead perhaps transformed to a nested XmlNode structure of some kind? Need to experiment some more - living and learning.$0$0$0$0$0Sure I'll work my way through some of this. Just thought I'd post an update. Cheers!$0$0

New Post: Newbie syntax questions

nblumhardt — Tue, 23 Sep 2008 00:48:52 GMT

Jeff - thanks for clearing that up! Answers everything perfectly.

I will let you know how I go extending the sample.

Nick

On Mon, Sep 22, 2008 at 5:11 PM, jeffmoser <notifications@codeplex.com> wrote:

From: jeffmoser

It's great to see you "getting your hands dirty" with OMeta#!

One thing that's subtle with OMeta's syntax is that "sample" in double quotes applies the **Token** rule that is defined in runtime's Parser with the given argument of whatever is inside the quotes (see the SCharacters rule in OMetaParser.ometacs). The token rule allows for an arbitrary number of Space(s) to precede it. If you look at the Space rule that is defined in OMeta.cs, you'll see that it will match whatever Char.IsWhitespace deems as whitespace (which includes newlines '\n'). This is one thing that was causing you grief. '\n' matches the literal new line rather than the token newline. The difference is that with the default Space rule, "\n" will never match because it already matches an arbitrary number of whitespace (that includes '\n') before it.

Also, note that ":h" by itself is an abbreviation for "Anything:h" which consumes one item of input from the input stream, which in this case is a single character. This doesn't appear to be what you wanted.

Other than these small issues, everything else looked pretty good. I took advantage of OMeta's quantifying operators (* and + which mean "zero or more" and "one or more" respectively) to simplify things down a bit. I also changed your HDelim rule to be slightly more general.

The result is a new "Markup" example project in the latest check-in. The grammar file is located here.

Does this answer your question?

Read the full discussion online.

To add a post to this discussion, reply to this email (ometasharp@discussions.codeplex.com)

To start a new discussion for this project, email ometasharp@discussions.codeplex.com

You are receiving this email because you subscribed to this discussion on CodePlex. You can unsubscribe or change your settings on codePlex.com.

Please note: Images and attachments will be removed from emails. Any posts to this discussion will also be available online at codeplex.com

New Post: Newbie syntax questions

jeffmoser — Tue, 23 Sep 2008 00:10:55 GMT

It's great to see you "getting your hands dirty" with OMeta#!

One thing that's subtle with OMeta's syntax is that "sample" in double quotes applies the **Token** rule that is defined in runtime's Parser with the given argument of whatever is inside the quotes (see the SCharacters rule in OMetaParser.ometacs). The token rule allows for an arbitrary number of Space(s) to precede it. If you look at the Space rule that is defined in OMeta.cs, you'll see that it will match whatever Char.IsWhitespace deems as whitespace (which includes newlines '\n'). This is one thing that was causing you grief. '\n' matches the literal new line rather than the token newline. The difference is that with the default Space rule, "\n" will never match because it already matches an arbitrary number of whitespace (that includes '\n') before it.

Also, note that ":h" by itself is an abbreviation for "Anything:h" which consumes one item of input from the input stream, which in this case is a single character. This doesn't appear to be what you wanted.

Other than these small issues, everything else looked pretty good. I took advantage of OMeta's quantifying operators (* and + which mean "zero or more" and "one or more" respectively) to simplify things down a bit. I also changed your HDelim rule to be slightly more general.

The result is a new "Markup" example project in the latest check-in. The grammar file is located here.

Does this answer your question?

New Post: Newbie syntax questions

nblumhardt — Mon, 22 Sep 2008 00:47:29 GMT

.. this time hopefully not corrupted:

ometa OMetaSharp.Examples.LittleTypeChecker<char, string> : Parser<char> {

TypeCheck = Body:t End -> { "<html><body>" + t + "</body></html>" },

Body = Block:p Body:b -> { p + b }

| Block,

Block = Heading | Para,

Heading = HDelim:h Text:t :h NewLine

-> { "<" + h + ">" + t + "</" + h + ">" },

HDelim = "===" -> { "h3" }

| "==" -> { "h2" }

| "=" -> { "h1" },

Para = Text:t -> { "<p>" + t + "</p>" },

Text = Item:i Text:t -> { i + t }

| Item,

Item = LetterOrDigit

| SpaceOrMore -> { " " },

SpaceOrMore = Space Spaces,

NewLine = "\n"

}

New Post: Newbie syntax questions

nblumhardt — Mon, 22 Sep 2008 00:46:20 GMT

Hi,$0$0$0$0I've been trying to do a simple text transformation along the lines of the Google Code wiki markup.$0$0$0$0$0The small chunk I've been implementing matches text, which are transformed to HTML paragraphs, and headings in the form:$0$0$0$0$0= This is a Heading =$0$0$0$0$0...which are transformed into HTML headings. The number of '=' signs denote a heading level H1 to H3.$0$0$0$0$0The complete (broken) grammar is below. The two points I'm having trouble with are:$0$0$0$0$01. Matching newlines$0$0$0$0$0No matter where I put it, "\n" causes an error when running the generated parser.$0$0$0$0$02. Named matches emit as raw text, not transformed, when appearing more than once in the same rule$0$0$0$0$0I.e. see the HDelim rule below, that I believe should transform the equals signs into an appropriate HTML tag name. When :h appears later in the rule, the raw source text is emitted.$0$0$0$0$0You can paste the grammar into the Little Typechecker example if you want to give it a run - grammar and expression names are unchanged :)$0$0$0$0$0Any advice on where I'm going wrong greatly appreciated.$0$0$0$0$0Nick$0$0$0$0$0--$0$0$0$0$0$0ometa OMetaSharp.Examples.LittleTypeChecker<char, string> : Parser<char> {$0$0$0$0$0 TypeCheck = Body:t End -> { "<html><body>" + t + "</body></html>" },$0$0 $0$0 Body = Block:p Body:b -> { p + b }$0$0 | Block,$0$0 $0$0 Block = Heading | Para,$0$0 $0$0 Heading = HDelim:h Text:t :h NewLine$0$0 -> { "<" + h + ">" + t + "</" + h + ">" },$0$0 $0$0 HDelim = "===" -> { "h3" }$0$0 | "==" -> { "h2" }$0$0 | "=" -> { "h1" },$0$0 $0$0 Para = Text:t -> { "<p>" + t + "</p>" },$0$0 $0$0 Text = Item:i Text:t -> { i + t }$0$0 | Item,$0$0$0$0$0 Item = LetterOrDigit$0$0 | SpaceOrMore -> { " " },$0$0 $0$0 SpaceOrMore = Space Spaces,$0$0 $0$0 NewLine = "\n"$0$0}$0$0$0$0$0

New Post: Regenerating parsers

nblumhardt — Sun, 21 Sep 2008 22:00:42 GMT

That did the trick. Thanks!

On Sun, Sep 21, 2008 at 2:53 PM, jeffmoser <notifications@codeplex.com> wrote:

From: jeffmoser

You should be able to set the Calculator project as your startup project and then change the parameter you pass to the run to be this:

OMetaConsoleProgram.Run<Program>(OMetaConsoleOptions.CompileGrammars);

This tells the code that Program inherits from that you want to compile the Calculator.ometacs fine to Calculator.cs.

Read the full discussion online.

To add a post to this discussion, reply to this email (ometasharp@discussions.codeplex.com)

To start a new discussion for this project, email ometasharp@discussions.codeplex.com

You are receiving this email because you subscribed to this discussion on CodePlex. You can unsubscribe or change your settings on codePlex.com.

Please note: Images and attachments will be removed from emails. Any posts to this discussion will also be available online at codeplex.com

New Post: Regenerating parsers

jeffmoser — Sun, 21 Sep 2008 21:53:23 GMT

You should be able to set the Calculator project as your startup project and then change the parameter you pass to the run to be this:

OMetaConsoleProgram.Run<Program>(OMetaConsoleOptions.CompileGrammars);

This tells the code that Program inherits from that you want to compile the Calculator.ometacs fine to Calculator.cs.

New Post: Regenerating parsers

nblumhardt — Sun, 21 Sep 2008 20:23:12 GMT

Hi Jeff!$0$0$0$0New solution organisation looks great.$0$0$0$0$0I've deleted the generated 'Calculator.cs' file and can't seem to get it back, either by running the bootstrapper or README - any suggestions?$0$0$0$0$0Cheers,$0$0Nick$0