JavaScript’s lack of macros is a bit troubling for me, especially given the recent tendency for JavaScript programmers to deal in lots of nested asynchronous callbacks: “Unindented functions to clean callback mess.” “Asynchronous callbacks in JavaScript”

I haven’t felt the pain of the asynchronous callback nesting problem firsthand, but I know a horror close to that one: Monadic programming in Groovy. Observe the indentation.

In fact, that indentation-heavy part of the Blade code is what originally got me fed up with using Groovy and interested in making Penknife as an intermediate project on the way to Blade. Well, my work on Penknife has been so focused on modularity and hygiene that it’s slack on what really matters: Syntax. Oh, syntax, how you really really matter so! :-p

Blade and Penwhatsit?

Unless you’ve seen my hobby-horsing rants on the Arc Forum, you probably have no clue what I mean by Blade or Penknife. They’re projects I’ve been working on for quite some time now, but I never did get around to blogging about them until now. In summary, they’re both language ideas which take customizable syntax to the limit, approaching the vision I described in my last post.

What’s the difference between Penknife and Blade, you ask? Well, they both center around extensible/customizable syntax, but Blade projects are finite collections of declarations, and Penknife projects are discrete, potentially ongoing sequences of commands. Blade is the kind of language that could have good IDE integration, but Penknife is the kind of language that could have good REPL integration. I started working on Penknife after those indentation difficulties with Blade (as I just mentioned), and Penknife is now much farther along in development.

Penknife and Blade are based on square bracket syntaxes which are designed to work seamlessly inside of most other-language code. In most existing programming languages, square brackets are somewhat less common than parens and curly braces, and when they do appear, they’re usually balanced. (Sure, the half-open mathematical range syntax (0,4] isn’t balanced, but what programming language uses that? :-p ) So the premise of Penknife and Blade’s syntax extension is pretty simple: Parse out the square brackets, and compile the outermost ones according to some rule or other that splits off an operator and applies it to the rest of the (unparsed) body. Chops uses this kind of syntax too.

Chops, with flashy examples

Going back to my roots a bit, I’ve put together chops.js (and lumped it in with some other JavaScript tools here in Lathe/js). It makes no attempt at being a programming language. It’s merely a text preprocessor. In Chops, each outermost square bracket is compiled by splitting off all the non-whitespace characters at the beginning, and then invoking a particular JavaScript function (parseInlineChop) which takes three parameters: that string, the current Chops environment, and the “rest” of the form as an array of strings and arrays.

One of the first things I put together was a little tool to help out with those indentation issues I was talking about. Instead of writing this:

thisWill( function ( err, a ) {
    return becomeSome( function ( err, b ) {
        return horriblyNested( function ( err ) {
            return code( a, b );
        } );
    } );
} )

You can write this:

[> [just thisWill( [next] ); ]
   [just function ( err, a ) { return becomeSome( [next] ); } ]
   [just function ( err, b ) { return horriblyNested( [next] ); } ]
   [just function ( err ) { return code( a, b ); } ]]

The “>” macro parses the last whitespace-delimited word in its body, then parses the one before it with that result provided as a local “next” macro, and so on. The “just” macro does nothing; its only purpose is to keep the code inside from counting as multiple words.

On Thursday, I took that up a notch. I have a Haskell-style monad notation now:

[>-> g1 bar
     g2 baz
     [just foo( g1, g2 ) ]]

This expands into something equivalent to the following (modulo meaningless parentheses and whitespace):

monadBind( bar, function ( g1 ) {
    return monadBind( baz, function ( g2 ) {
        return foo( g1, g2 );
    } );
} )

I even have a crazy monad notation of my own devising:

[>- foo( [<- bar ], [<- baz ] ) ]

You can actually try out these syntaxes at http://rocketnia.github.com/lathe/chops.html, though that’s mostly a kind of unit testing sandbox page, and my REPL interface there is extremely unpolished.

>>> rocketnia.chops.parseChopcode( choppaScript, "[>- foo( [<- bar ], [<- baz ] ) ]" )
--> (monadBind( ( bar ), function ( gs8352752204_2 ) { return ((monadBind( ( baz ), function ( gs8352752204_3 ) { return ( foo( gs8352752204_2, gs8352752204_3 ) ); } ))); } ))

Here’s that output prettied up, with some unnecessary parentheses removed:

monadBind( bar, function ( gs8352752204_2 ) {
    return monadBind( baz, function ( gs8352752204_3 ) {
        return foo( gs8352752204_2, gs8352752204_3 );
    } );
} )

There’s also a “” macro, so that any opening square bracket followed immediately by whitespace is treated as though it were regular text:

>>> rocketnia.chops.parseChopcode( choppaScript, "[> foo[ [next] ] bar]" )
--> foo[ bar ]

Chops, with flashy hand-waving

Chops doesn’t parse JavaScript. This is totally a text-level translation (unless you go and write an ambitious macro that does the parsing itself), and any kind of malformed or browser-specific JavaScript is just fine. Output languages other than JavaScript are fair game too, including Arc and Haskell.

In fact, the output language doesn’t have to be textual. I personally intend to use Chops as yet another markup language for generating HTML, and I intend to use an intermediate AST representation. I was building my website in Penknife this way already, but I think Chops’ll be a bit more straightforward in the short term.

At this point, Chops is a JavaScript library, but there’s no good way to develop code that uses Chops as a preprocessor. That’s the next step, as I see it. Node.js is an option, but I’d mainly like to able to build in the browser, especially in a way that uses some existing online storage API. (Any storage recommendations? I’ve been trying the GitHub Gist API, but it uses unusual HTTP actions like PUT, which apparently can’t be sent by typical browsers. ^^; Now that I’ve seen JSPaste, I’m kinda thinking about RDBHost….)

There’s way more to this concept than than Arc tackles, though.

Programmers in the future will learn from the mistakes we make designing languages today, and they’ll design better ones. For there to be a single language that persists despite future innovations, snowballing its own library support along the way, it ironically needs to be all the various languages people want it to be. It needs to be a customizable language. (And it needs to target platforms other than its runtime.)

Foreign Affairs

Any two of those languages will easily target each other’s runtimes, so if things happen the way I expect, we’re likely to find a landscape full of competing languages that all abstract over each other with no clear winner.

To limit that effect, it’s important for those languages to try to have no obvious defects from the outset, so that they don’t build up audiences that feel they need to spin off into various camps to pursue slightly better languages.

For this reason, I think it’s a mistake for any language that takes itself half-seriously to impose an arbitrary limitation on its users. A limitation that makes other things easier is fine, but the point of a tool is always to make people more productive, not less. Good programmers will police themselves as necessary.

But for the same reason, I think the strongest position in a winnerless landscape of customizable languages is a minimalistic one. If a customizable language’s runtime has only the features it needs for convenient syntax customization, the designer has fewer opportunities to mess up and introduce flaws. In that sense, the language should be severely limited, but for a non-arbitrary cause.

Civil Unrest

Meanwhile, each of these languages will itself host a lot of competing syntaxes and abstraction layers. Not all of those will follow the language’s own philosophy, ’cause they’ll be their own things. But many will be general-purpose tools for people to use when customizing the language’s syntax, and their presence will effectively recolor the language for people who want to take advantage of their added features. I believe the language should have a strong philosophy encouraging consistency between these libraries and the main language runtime API, so that the experience doesn’t fray into separate and competing sub-experiences.

But a language’s library landscape is anything but minimalistic, so it can’t share that aspect with the core. Libraries will inevitably introduce extra concepts with flawed designs and buggy implementations. Instead of trying to avoid this, library programmers should assume the programmers who use their code know better than they do, by nature of those programmers being in the future. Thus, when in doubt, they should leave their implementation details exposed rather than hiding them. They should maximize their users’ freedom, just like the language itself does.

A naive take on this would be challenging for libraries in active development. They tend to change their implementation details from release to release in incompatible ways, so if they feel pressured not to break others’ code, that limits their freedom! To solve this, I believe library users should be capable of using unstable features, but that the language culture should encourage library writers to make it clear what’s stable and what isn’t, perhaps in such a way that even if it’s easy for library users to use the unstable parts, it’s also easy for them to refrain from doing that.

Conclusion

So altogether, I believe the best kind of language is a minimalistic, customizable-syntax, cross-targeting build language which has a philosophy encouraging consistently unstable library APIs, makes nothing intentionally hard, and most of all prepares for future innovation.

Not only is this my kind of language, I think it’s imperative to make these languages extremely well from the beginning so that it we don’t let suboptimal versions gain the advantage of entrenchment.

Platforms stack, vaguely. The LAMP stack is a classic example: The server is Linux running Apache, content-specific sub-platforms are developed on top of PHP or Perl for Apache to use, data sub-platforms are developed on top of MySQL for the PHP/Perl code to use, and the machine code programs (the OS included) rely on a hardware platform of some sort or another.

Platforms also target other platforms and hide them away. For instance, a PHP coder hardly cares about the instruction set in use on the machine, ’cause the interpreter takes care of that. It’s easy to write a simple PHP script that doesn’t really care what the hardware, the OS, the server, etc. look like. In this case, PHP itself may be all we’re thinking about. In that case I’d say that PHP is our platform.

The reality today is that platforms and languages are developed symbiotically. Innovative general-purpose platform concepts spawn new languages suited for them, like Erlang. Languages which intend to support multiple platforms just end up defining their own. Most platforms can’t be perfectly translated into each other, so ports of platform-defining languages can only be sparse or imperfect, and when it comes down to it, one’s choice of platform determines one’s language options.

Many programmers, including myself, prefer to choose languages not based on the platforms they target but based on their smoothly adapting syntax as project-specific needs come up. Languages which support extensible syntax, like most lisps, are ideal for this purpose, since they hold an implicit infinity promise that they’ll eventually be every bit as beautiful and convenient as every other language, if people only bother to develop the necessary syntax extensions.

But customizable languages fall short today. People are used to developing programs that target just one platform—the runtime of the language they’re writing in—and that makes customizable languages only as flexible as the one runtime they’re limited to.

For a language to have an inflexible runtime is important for code reuse, but if we’re going to have a convenience-and-aesthetics language that’s capable of continuing to build momentum over the next, say, 100 years (Arc’s shtick), we need a language that will target 100 years of platforms we haven’t invented yet. Therefore, the platform the language code runs in and the platform it targets (usually) won’t be the same.

Instead, the language needs to encourage a development process that’s like writing a compiler or build script, where the code produces project artifacts like binaries and other-language scripts, and those are all that’s deployed.

AST Processing

If we start with a “compiler” that just outputs what it’s told to output, then writing code generators in the language will already be at least as convenient as writing the code manually, so the infinity promise will already fulfilled, albeit unimpressively. Templating languages are a step beyond this even, but flat string concatenation is an ineffective way to go forward. Instead, I believe the customizable language’s core focuses should be on AST manipulation, parsing, dependency resolution, and other compiler duties.

This is the direction I’ve been taking Blade and Penknife, my own programming languages. Platform innovation is well and good, but I believe truly general-purpose languages are nothing if not homes for syntax innovation.

It’s on now.

After about a month of me submitting bugs to the Arc forum and Jarc’s creator, JD Brennan, fixing them, Jarc is now compatible enough with Arc 3.1 that my Lathe (the library I introduced last time) now wholeheartedly supports the Jarc implementation of Arc. This means it works on four Arc setups: Jarc, official Arc 3.1 (which itself needs a Windows fix), Anarki (which needs the same fix), and Rainbow. Most of you reading this are probably from the Arc forum, in which case you knew all that already. :-p

Those are all the setups I want Lathe to support for now, but I might consider arc3f, arc2c, or another Arc implementation if I realize it’s actually active. Another complication is that quite a lot of Arc users modify one of the PLT Scheme versions of Arc (Arc or Anarki) to suit their needs. In those cases, I figure the burden of Lathe compatibility is on them if they need it.

Still, there is a bit of a need to add new features to Arc. To this end, Anarki, Jarc, and Rainbow have provided ways to call from Arc into the host platform (PLT Scheme, the JVM, and the JVM respectively).

In fact, if you know Jarc, the above example isn’t all that impressive. You can already do this:

$ cd /path/to/jarc

$ java -server -cp \
>   "jarc.jar;$GROOVY_HOME/embeddable/groovy-all-1.7.2.jar" jarc.Jarc

Jarc> (groovy.util.Eval.me "2 + 2")
4

On Rainbow, you can accomplish the same thing in a couple of ways:

$ cd /path/to/rainbow/src/arc

$ java -server \
>   -cp "rainbow.jar;$GROOVY_HOME/embeddable/groovy-all-1.7.2.jar" \
>   rainbow.Console

arc> (java-static-invoke "groovy.util.Eval" 'me "2 + 2")
4
arc> (java-import groovy.util.Eval)
[Rainbow displays the source of the resulting macro.]
arc> (Eval me "2 + 2")
4

So what does Lathe get you here? Not much, I guess, but it actually does give you the ability to assign to JVM fields:

arc> (jv.jvm!height= (jv.jvm!java-awt-Rectangle-new 1 2 3 4) 5)
5

That’s a feature that’s lacking in Jarc and in Rainbow. Lathe’s jvm.arc accomplishes it through reflection calls on java.lang.reflect.Field objects, and it has to do one of the worst kinds of JVM reflection: Branching on primitive types. I’m glad that’s behind me!

Lathe also gives sort of a nice syntax, in my opinion. The syntax prioritizes (my) intuition over consistency, in ways that unfortunately make for a monstrous implementation that takes just as long to fully explain. Here’s the important part of the documentation, which assumes you’ve put jv.jvm into the global variable jvm through something like (= jvm jv.jvm):

; [blah, blah, blah, paragraphs and paragraphs of text]
;
; At long last, here are some examples:
;
; getting a static field
; jvm!javax-swing-JFrame-EXIT_ON_CLOSE
; jvm!EXIT_ON_CLOSE.jframe-instance
;
; setting a static field
; jvm!com-example-SomeClass-staticField=.new-value
; jvm!staticField=.someclass-instance.new-value
; (jvm!staticField= someclass-instance new-value)
;
; calling a static method
; jvm!java-util-Arrays-asList.array
;
; getting a top-level class as a Class instance
; jvm!java-lang-Object-class
;
; getting a static nested class as a Class instance
; jvm!java-util-HashMap-Entry-class
;
; calling a constructor
; (jvm!java-util-ArrayList-new)
;
; calling an instance method (same as a static one, but shortenable)
; (jvm!java-util-HashMap-get map key)
; (jvm!get map key)
;
; getting an instance field
; jvm!java-awt-Rectangle-x.rect
; jvm!x.rect
;
; setting an instance field
; (jvm!java-awt-Rectangle-x= rect newx)
; jvm!java-awt-Rectangle-x=.rect.newx
; (jvm!x= rect newx)
; jvm!x=.rect.newx
;
; accessing an inner class (a non-static nested class) as a Class
;   instance
; jvm!com-example-OuterClass-InnerClass-class
; jvm!InnerClass.outer!class
;
; constructing an instance of an inner class
; jvm!com-example-OuterClass-InnerClass-new.outer
; (jvm!InnerClass.outer!new)
;
; some makeshift importing
; (= jutil jvm!java-util
;    weakhash jutil!WeakHashMap-new)
; (def jweak (x)
;   (if (ajava x jutil!Map-class)
;     weakhash.x
;       (isa x 'table)
;     (let result (weakhash)
;       (each (k v) x
;         (jvm!put result k v))
;       result)
;     (err "A non-map was passed to 'jweak.")))

But what this inconsistent interface gets us most of all is a different form of consistency: Greater consistency between Jarc and Rainbow. Now that the jvm function exists, if I want to add some other feature to Lathe that requires me to make JVM calls, I may be able to get away with writing the same code on both platforms.

It’s not quite as easy as that. Both platforms do certain things for you automatically, namely converting between Arc types and JVM types and deciding which JVM method/constructor best suits the Arc parameters you’ve given. An Arc library can’t easily overcome that; it can’t get its hands on certain kinds of JVM value, since they’re converted before Arc even sees them. Therefore, rather than putting abstraction inversion complexity on top of jv.jvm‘s other complexity, I’ve had it go with the flow for now.

But if you happen to have Groovy in the classpath, you can drop to the JVM and rise back up to Groovy by calling groovy.util.Eval.me(), and you can use Groovy as a more Arc-agnostic basis for manipulating JVM objects. You can also do additional things like extending JVM classes (not just interfaces) at runtime, which otherwise you would probably have to do by fiddling with bytecode and ClassLoaders. Time will tell if Lathe will provide extra features when Groovy is available, but it’s pretty likely considering how much I like Groovy. ^_^

That was a lot of focus on the JVM side of Arc. What about PLT Scheme? Well, thanks to a bug I discovered in Arc, Anarki isn’t the only setup that can drop to PLT Scheme. So Lathe’s sniff.arc, which is a home for platform detection flags, now includes four lines that implement a plt macro. It treats its argument as a PLT Scheme expression and returns its result, regardless of whether official Arc or Anarki is in use.

Both plt and jvm are set to nil when on an incompatible platform, and I expect that to allow for some nicely idiomatic cross-platform library code.

Well, it will once I discover a library I actually want to write with these things! My two ideas are a desktop GUI library, which I’m not sure I need, and a dynamic binding library, something I’m really interested in which seems to be a tough nut to crack on Rainbow.

P.S.: There’s another bit of progress on Lathe worth mentioning. In order to get multivals working on Jarc, which doesn’t support reinvokable continuations, I ported over an iteration library I had originally written in Groovy. It’s a pretty substantial library for building iterators when continuations aren’t available, and when they are, it also supports the yield coroutine style used in Python, C#, and so forth. It’s Artistic License 2.0 like the rest of Lathe. Check it out.

That Arc multimethod system is one of those utilities, but first…

Modules

I’m not very bold when it comes to polluting the global namespace, and that’s discouraged me from releasing any Arc code. Even if I pick names no one else does, I’m still limiting my own ability to give those same names different meanings in the future. If I use any specific global names, then multiple versions of my code will not be able to coexist in the same application, and that bugs me.

So that I could avoid this trouble, the first thing I contributed to Lathe was a module system. I have a few basic goals in mind for this system, which I’ve so far achieved:

• Hiding code is okay but not Arclike. An Arc module system should free code from accidental naming conflicts, not protect it from intentional ones.
• One library should be able to declare a dependency on another library without specifying how to satisfy that dependency. For example, two libraries should be able to depend on a third library without that third library being loaded (or downloaded, compiled, etc.) twice.
• Even if it isn’t possible to treat existing Arc libraries as modules, code written for the module system should be easy to rewrite in a more traditional way. I’m sharing Arc hacks, not committing people to a framework.
• Some Arc code may rely on modifications to the Arc implementation, but the module system itself should not. For instance, none of my code currently drops to Scheme; in fact, it works just as well on Rainbow as it does on Arc and Anarki.

Another basic property I’d like to fulfill in this module system is the ability for multiple libraries to depend on each other, which can be useful when a big project needs to be split up into more manageable chunks. I have a plan in mind for this, but I’m going to put off this plan until I work a bit more on multimethods, especially since multimethods and module systems might influence each other in subtle ways.

Here’s an example of the module system in action (which may also give you a bit of insight into the multimethod system):

; multirule.arc

(packed:using-rels-as mt "multival.arc"
                      oc "order-contribs.arc"
                      ru "../rules.arc"

; A basic-rulebook-reducer multival takes contributions that are rules
; (which are extracted here from the 'val entries of contribution
; detail tables), it sorts those contributions using order-contribs,
; and it ultimately becomes a function that calls those sorted rules
; as a basic rulebook.
(def my.basic-rulebook-reducer (contribs)
  (let rulebook (map !val (apply join oc.order-contribs.contribs))
    (obj val (fn args
               (apply ru.call-basic-rulebook rulebook args))
         cares `(,oc!order-contribs))))

(mac my.rule (name parms . body)
  (zap expand name)
  (let (label . actualbody) body
    (zap expand label)
    (unless (and actualbody anormalsym.label)
      (= actualbody (cons label actualbody) label (uniq)))
    `(do (,mt!defmultifn-stub ,name)
         (,mt!contribute ',name ',label ,my!basic-rulebook-reducer
           (,ru!ru ,parms ,@actualbody)))))

)

All the definitions are surrounded by a (packed ...) form, which does a couple of things: It defines the “my” namespace anaphorically, it executes its body at the top level, and it returns a package value that exposes that namespace.

The using-rels-as form here resolves the three dependencies '(rel "mutlival.arc"), '(rel "order-contribs.arc"), and '(rel "../rules.arc"), gets the package values for them, and binds 'mt, 'ot, and 'ru to those packages’ namespaces for the rest of the form, which it executes at the top level.

A dependency of the form '(rel "relative/path/to/code.arc") uses relative addressing. There’s a little bit of bookkeeping done behind the scenes to make this work. (Currently, this is actually the only form of dependency supported. It’s possible to extend the system to support things like installing libraries from the Web, requiring certain Arc implementations to be in use, requiring certain implementation patches to have been applied, or doing version number comparison, but I don’t yet have a plan in mind for those things.)

I’ve mentioned that 'my, 'mt, 'oc, and 'ru are bound to namespaces here. A namespace is just a macro that associates symbols to gensyms.

• If you pass it a symbol (“my.foo“), it expands into a gensym (“gs1234-foo“). This means you can use my.foo in most places you can use a variable name, except that it has to be a place where macros are expanded.
• If you pass it a quoted symbol (“my!foo“), it expands into a quoted gensym (“'gs1234-foo“). This is useful for code generation, as you can see in multirule.arc above.
• Finally, if you pass it a form other than a 'quote form (“(my:foo a b c)“), if the car of that form is a symbol, then it expands into the same form except with a gensym at the car (“(gs1234-foo a b c)“). This is useful if my.foo is a macro; if instead you just say (my.foo a b c), the Arc compiler will expand the my.foo ssyntax, then determine that (my foo) is not a symbol globally bound to a macro (since it’s a list), then macro-expand (my foo), and you’ll end up getting a runtime error when the macro is invoked as a procedure.

In order to facilitate the use of namespaces, the module system redefines 'def, 'mac, and 'safeset so that they expand the names given to them, and Lathe code continues with this policy everywhere it makes sense to do so. You can see this effect in multirule.arc above, where 'rule explicitly expands both its name and its label.

Multivals, a generalization of multimethods

For the time being, Lathe’s multimethod API is relatively abstract. In fact, the term “multimethod” itself indicates a method that dispatches based on the types of multiple parameters, and since Arc has no obvious and extensible type system to dispatch on, I haven’t attempted multimethods in this proper sense. But inasmuch as a multimethod is a method defined in multiple pieces, it’s a special case of what I’ll call a multival.

Conceptually, a multival is anything which can be defined in multiple pieces (its contributions). When a multival is needed, the accumulated contributions are sent to a single reducer associated with the multival, and the reducer’s result is used.

When something is defined in multiple places, it’s tough to say in general which of those definitions should have precedence or what order they should be combined in, especially when the definitions are far apart or in separate libraries (when code order becomes obscure or vague). This is a problem many multiple-definition systems like CSS and Inform 7 solve with complicated orders of precedence (like a notion of how specific a definition is) alongside patched-on ways to escape those conventions (like CSS’s !important and Inform 7′s procedural rules). For the purposes of solving this problem, Lathe’s multival system provides a single multival, 'order-contribs, whose purpose is to sort multival contributions.

That’s right, 'order-contribs is a multival. True to its intent to sort the contributions of every multival, it sorts its own contributions. It does this the hard way, by brute-forcing every permutation of its contributions (with quite a bit of pruning) until it finds an order that’s consistent with itself. If this order can’t be found, 'order-contribs signals an error. This should cleanly model the behavior desired by Andrew Plotkin for a rule-based rule precedence system. And hey, if you can find a smarter way to do this order-finding, you can write that function and contribute it to 'order-contribs as the only contribution. The built-in brute-force search won’t hurt you, since the factorial of one is one.

Finally, for those who really don’t care about 'order-contribs, it’s opt-in. Most multival reducers Lathe defines will call 'order-contribs right away, but there’s nothing preventing someone from defining a reducer that does no such thing. One reason for doing this is when you need to use a strange library that contributes things to 'order-contribs that mess it up for everybody else. (When a rogue rule asserts its own precedence over all other rules, in principle it’s impossible to overcome that with another rule unless you get clever/lucky regarding the order in which the permutations are checked.)

Multivals in practice

I could go into the internals of defining your own reducers and contributions in Lathe, but there’s nothing particularly clever about it. There’s just a global store of multimethod contributions and reducers, along with an API for contributing to that store. And then there’s 'order-contribs, which has similar idiosyncracies if you want to use it directly. So instead I’ll show you how this infrastructure looks on the surface, using some contrived examples:

; multirule-demo.arc

(prn "starting multirule-demo")

(using-rels-as mr "../multival/multirule.arc"
               oc "../multival/order-contribs.arc"

wipe.fail

(mr:rule factorial (n)
  (unless (< 0 n) (fail "The parameter isn't positive."))
  (* n (factorial:- n 1)))

(mr:rule factorial (n)
  (unless (is n 0) (fail "The parameter isn't zero."))
  1)

(mr:rule expbysquare (base exp) one
  (unless (is exp 1) (fail "The exponent isn't one."))
  (list base 0))

(mr:rule expbysquare (base exp)
  (unless even.exp (fail "The exponent isn't even."))
  (let (prevresult mults) (expbysquare base (/ exp 2))
    (list (* prevresult prevresult) (+ mults 1))))

(mr:rule expbysquare (base exp)
  ; This condition subsumes the condition for the 'one contribution on
  ; purpose, so that we can test 'prec. Since this contribution comes
  ; later in the code, it ends up farther toward the beginning of the
  ; contribution stack, and ultimately this rule would be tried before
  ; the 'one rule (catastrophically) if not for the presence of
  ; [iso (map _ '(name label)) '(expbysquare one)] in the 'prec line
  ; below.
  (unless odd.exp (fail "The exponent isn't odd."))
  (let (prevresult mults) (expbysquare base (- exp 1))
    (list (* base prevresult) (+ mults 1))))

(mr:rule expbysquare (base exp) diabolical
  (err:+ "Somehow the 'diabolical contribution to 'expbysquare was "
         "used."))

(oc.prec [iso (map _ '(name label)) '(expbysquare one)]
         [~iso (map _ '(name label)) '(expbysquare diabolical)])

(for i 0 10
  (prn "The factorial of " i " is " factorial.i "."))

(for i 1 10
  (let (result mults) (expbysquare 2 i)
    (prn "Two to the " i " gives " result " after "
         (plural mults "multiplication") ".")))

(prn "finishing multirule-demo")

nil

)

First, note that this file is not set up a module, and it doesn’t hesitate to pollute the global namespace. But although it isn’t careful about what comes out of it, it is careful about what goes in; the “wipe.fail” line toward the beginning makes sure that 'fail is not a macro and therefore doesn’t get macro-expanded in the following code.

There are only three new operators this code uses: 'rule, 'fail, and 'prec.

The implementation of the 'rule macro was shown further above. Essentially, it works like a normal function definition, except that multiple such definitions can coexist.

In order to make sure the appropriate rule is used for the given situation, any rule that doesn’t apply should call 'fail right away. The 'fail function is defined anaphorically within the context of a rule, and it exits the rule immediately, making its use very similar to that of 'err. However, 'fail and 'err have different meanings; 'fail signifies that this rule doesn’t apply but some other rule might, and 'err signifies that something is positively wrong.

Another feature 'rule provides is to label a particular rule with a given symbol, by writing that symbol unquoted after the argument list. As you can see, this is optional; the 'factorial rules have no labels, but two of the 'expbysquare rules do. Labeled rules have two advantages: One, they’re easier to refer to specifically from elsewhere in the code, and two, if you define multiple rules with the same name and the same label, the older ones are overwritten. The overwriting feature is especially useful at a REPL, and it was inspired by an early version of Andrew Wilcox’s 'extend macro.

Finally, there’s 'prec, a function that takes any number of tests and institutes a precedence order on contributions so that those which satisfy the tests come first. When there are multiple tests given, the earlier tests in the argument list have more powerful influence than the later ones.

A 'prec call just works by registering a contribution to 'order-contribs. It’s possible for this contribution to be given a label too, using the 'label-prec macro like so:

(oc:label-prec expbysquare-precedence
  [iso (map _ '(name label)) '(expbysquare one)]
  [~iso (map _ '(name label)) '(expbysquare diabolical)])

If you label your precedence contributions like this, you can define precedences over precedences.

Furthermore, with the right setup, you should be able to store arbitrary metadata about rules at the same time as you define them (by associating the name and label of the rule to a metadata container via a global table or something) and then have much more expressive precedence rules based on that medatada, making it less necessary to write special-case precedence rules like these. This is the path I’m going to take, ’cause there’s no point in having multiple definitions of something if the definitions are going to be manipulated individually.

And much, much more (but not that much)

Even if the module system and multival system are too monolithic or too specific for your needs, Lathe still has quite a few bite-size hacks worth looking at. Here are a few of them:

The 'global function, defined in Lathe’s arc/modules/modulemisc.arc, takes a symbol and returns the value of that symbol in the global environment, or nil if the symbol is unbound. That’s nothing exciting; (global x) is just (bound&eval x) with some error checking. What’s exciting is that 'global has a 'setforms entry, allowing deeply nested code to modify arbitrary global bindings. This is slightly harder than it sounds; to do it without using 'global requires code similar to (eval `(= ,x ,something)), but that 'something can’t refer to any part of the lexical context, since it’s being evaluated in the global one, so a single temporary global variable is needed to ferry the value over. The 'global 'setforms entry takes care of all of that.

The 'parse-magic-withlike function, which is also defined in modulemisc.arc, is a utility for macros to use. It takes the body of a form that’s similar to Arc’s 'with, and it extracts all bindings from the beginning of it and pairs them up in a list. If the first element of the body is a list, it behaves just like Arc’s 'with and treats that list as a list of places and values to bind. However, the parentheses around those bindings can be omitted if all the places being bound are non-ssyntax symbols. Since a non-ssyntax symbol doesn’t make a lot of sense at the beginning of a 'do body (unless it’s the only expression in that body), it shouldn’t be misinterpreted as a place to be bound. And the macro can still support destructuring or setforms for bindings, since the user can always wrap the bindings in a list the old-fashioned way when that feature is needed.

Somewhat randomly, I wrote an independent implementation of Andrew Wilcox’s 'extend macro, which I put in Lathe’s arc/extend.arc. In addition to the 'extend macro itself, this also provides 'label-extend, which does the same thing but associates a unique label with the extension so that another extension with the same label will overwrite it. There was an interesting hurdle to overcome in this implementation to handle corner cases in the way Andrew Wilcox’s 'extend binds 'it in the body to the value of the test; I end up having to store the body as a function that takes 'it and returns a function that takes the regular parameters, since 'it is bound this way in the parameter list too. I haven’t needed this at all, but it was a nice exercise.

These snippets are great candidates for committing to Anarki. However, I don’t know where I should draw the line, I’m not bold enough to stomp into a community project carrying a module system, and because of its Rainbow support, Lathe needs a home apart from Anarki anyway. If my hacks do find their way into other projects, that’s fine, but I think their first stop will continue to be Lathe.

And all in all, With the comfortable structure the Lathe repo gives me, I’m hoping to have a outlet for my language design pursuits while at the same time making the results more available to the rest of the world.

Earlier, my approach to MVTron was to wrap AviSynth in the hopes that it could take advantage of all the fast AviSynth plugins people have written in C. I managed to get a scene detector working that processed my test video in about twice the video’s duration. That struck me as a pretty poor performance, but I was still hopeful, and I was planning to get my plan in motion from there.

Well, it’s kind of embarrassing to say this, but I completely forgot about that AviSynth-Groovy success story until just now when I read my own blog. That said, this time around I duplicated that success in less than a day without interfacing to AviSynth at all, using Xuggle’s (relatively) new MediaTools API. And now I’m usually looking at processing times only 170% the test video’s duration. Thanks, Xuggle!

Random-Access Pass

It’s probably not worth mentioning this, but before I turned to MediaTools, I first tried to improve the Xuggler code I was already using. I have a random-access video class that wraps a Xuggler container and uses seek to go backwards. I’m well aware that video isn’t usually encoded for fast random access, so the idea of this interface is that, as long as you’re asking for frames in the right order, you get them as efficiently as usual, but when you need to back up, that’s still easy to do. It does a great job of abstracting away the usual packet-pushing Xuggler API… if you’re lucky.

The problem was that the first few frames after a seek weren’t necessarily the right ones, even if the rest of them were. Well, I tried to gloss over that by copying off a few frames after each keyframe the first time through the file, and then using the preserved ones in lieu of corrupt ones later on. It worked like a charm, but only if there was only one seek needed. Apparently, the first seek might corrupt a few frames, but seeking again after that just breaks the whole video.

I briefly considered having the random-access wrapper open a new Xuggler container object and fast-forward from the beginning every time it needed to seek backwards, but I passed on that for the time being, in favor of MediaTools.

Pipelines

The MediaTools library also “does a great job of abstracting away the usual packet-pushing Xuggler API.” Rather than trying to hide the serial access, though, MediaTools embraces it. Essentially, you build a bundle of objects that represents the data flow (e.g., source video to conversion routines to destination file), and then in one magical line you execute it.

A MediaTool pipeline is built using the concept of events. You make a listener and register it with an object that broadcasts events, and the broadcasting object keeps track of that reference and calls the listener with events as they happen. This is a familiar concept in Java GUI programming and probably a lot of other things (things I haven’t experienced yet), but this time around I find it funny, ’cause seeing this use of listeners gave me the epiphany I needed to appreciate Haskell’s arrows. I’ve never used arrows, but I miss them already. I think MediaTools made an appropriate choice though; arrows may be a more natural abstraction, but I think they aren’t as easy to get the hang of as listeners, which defeats the purpose of an easier-alternative API.

(Incidentally, I think Marsyas uses a very similar pipeline metaphor. I haven’t looked deeply enough into Marsyas to draw a meaningful comparison, though.)

In my own MediaTools-powered code, I’m finding that I’m using absolutely no Groovy except in my sandbox scripts, and even my Groovy sandboxes are mostly copied-and-pasted from equivalent Java ones. Groovy still excels after the pipeline has done its job, when I want to do things like print out totals and histograms, but I think all the subclassing and put-this-one-in-that-one code is as brief in Java as it’ll ever be.

In Groovy’s favor, a network of listeners like this is naturally hierarchical. (It’s a bit of a pain to write a listener that waits for more than one event before doing something, since you have to manage state manually. As long as you’re managing to avoid that pain, your pipeline has a single input and several outputs that branch out from there. In fact, the “one magical line” that executes the pipeline depends on there being only one root input, and I think it would get pretty complicated with more than one.) When you’re taking a bunch of building blocks and putting them together hierarchically, you have a great candidate for a Groovy markup-style DSL.

The thing is, I’m not accustomed to making markup-style DSLs just yet, and even if I were, this is not the time. The listeners I’ve made and considered making have all sorts of different structures, so for the time being I don’t have many patterns worth abstracting away.

The Guts

So, what all did I accomplish in less than a day? Here are the details, in English about as clear as I can muster.

Scene Detection, a Diabolical Transition

As I mentioned, I implemented the scene detector. The algorithm I’m using is slightly different than the one I was using before. Before, I had an algorithm that took a tolerance and a clip as input and that output a list of booleans that indicated where the breaks were. However, giving the tolerance at the beginning like that was just a temporary kludge. I really want MVTron to detect the tolerance by default most of the time; having to explicitly tell it things about every video you give it ruins the fun that would come from just throwing it at a directory of music and videos and seeing what it came up with. In order for auto-detected tolerances to be practical, they need to be detected after the preprocessing is done, so that new and better tolerance detection methods can be swapped in without taking hours to re-scan all the footage.

As it turns out, translating my algorithm from tolerance-given-at-preprocessing-time to tolerance-given-never wasn’t very straightforward. Surprise surprise, right? It actually might not be the caveat you’d expect.

Under ideal circumstances, I would just take a difference between adjacent frames at preprocessing time, find outliers at composition time, and be done with it. However, I’m lucky enough that one of my test videos has an example of a pretty understandable phenomenon. There are frames A-B-C in this video where it’s obvious that frames A and C would constitute a scene break except that frame B is blurred almost exactly halfway between them. (I’m pretty sure it was telecined so that the pixel rows actually alternated between the two scenes, then scaled down so that those rows blended together.) This makes frame breaks A-B and B-C have intensity far below the other scene breaks, even getting dangerously close to the breaks between animation cels, but despite this, I’d rather have both A-B and B-C detected as scene breaks rather than neither.

The original algorithm worked by measuring all the A-C intensities as well (or else adding the A-B and B-C intensities, which in my tests has similar results), and then putting “unexpected” scene breaks detected that way (that is, A-C scene breaks without apparent A-B or B-C breaks) into the result as a special case.

The new algorithm doesn’t make a boolean stream until an intensity stream is finally analyzed to determine a threshold. Instead it works on combining intensity streams. In fact, it only uses the A-B intensity stream, so in a sense it combines that stream with itself. Here it is in Groovy:

// Some product fuzzy logic operations, which operate on values between 0 and 1,
// inclusive. The or defined here will be used below.
def not = { 1 - it };
def and = { a, b -> a * b };
def or = { a, b -> not( and( not( a ), not( b ) ) ) };

// By now, diff has been defined such that
// diff == [ 0 ] + (frame break intensities gained from preprocessing) + [ 0 ].
// If there are 0 frames, then diff == [ 0 ] instead.

// The focusedDiff accentuates each intensity of diff in one of two ways: If an
// intensity is a local maximum, it's augmented with its greatest neighbor. If
// it isn't a local maximum, it's augmented with itself. This has the effect of
// holding back local maxima with small neighbors, which are crisp already.
def focusedDiff =
    numberOfFrames == 0 ? [ 0 ] : [ 0 ] + (1..<numberOfFrames).collect { or(
        diff[ it ],
        Math.min( diff[ it ], Math.max( diff[ it - 1 ], diff[ it + 1 ] ) )
    ) } + [ 0 ];

(For what it’s worth, since min( J, max( I, K ) ) = max( min( I, J ), min( J, K ) ), this sequence operation could be reasoned about without referring to the elements, calling it something like “an accentuation zip of a sequence by the pairwise maximum of the pairwise minimum of a zero-bookended version of itself.”)

Scene Detection Grit

To evaluate the numeric difference between two frames, I calculate the root-mean-square of the differences in R, G, and B values. This is the same as the root-mean-square of the pixel color differences, for a pixel difference calculation that’s proportional to the colors’ Euclidean distance in the sRGB color space. My code makes it easy to swap in a better color metric (or a better frame metric altogether) if that turns out to help, but this is working just fine. I actually think it might be slightly less useful than the same metric without all the squaring and root-taking, which is closer to what I used in AviSynth (which was the average luma of the absolute difference of the frames).

To guess a tolerance for scene breaks, I just take the maximum frame break intensity and divide by two. This algorithm could definitely use some work, but I’ll have to test on some more videos or read up some more before I can figure out how to improve it.

Audio Analysis

I’m just barely understanding most of what I’m trying to do here. I read a lot about DFTs and DCTs, and I only learn enough to program something, and I guess if I can explain it to a computer then I must understand it perfectly, but it doesn’t feel that way. I think I’m shooting for a Mel-frequency cepstrum here at some point, and I have a pretty good idea of how to get one from the sample data, but I’ve had to work myself up to the concepts one at a time, and I stopped coding on this after I’d successfully invoked an FFT and made a histogram of the peak frequencies at each audio frame in a test video. (Yeah, I’ve only tested this with videos rather than music so far, mostly ’cause the videos were handy and I knew I could load them.)

I didn’t try to write the FFT code myself. Instead I’m using a library, JTransforms. I don’t really care where I get my FFT, as long as it’s fast, accurate, and feature-complete. If I don’t have to maintain it, awesome.

Conclusion

The point I’m trying to make is, it probably took me about as long to write this summary as it took to try some stuff, fail, and start MVTron all over with MediaTools, and yet the MediaTools code was faster and tackled things I hadn’t even gotten around to attempting before. I think that says something for Xuggler/MediaTools.

Why? Not-invented-here syndrome, in a sense. The IF system I would like to work with is not nearly the system that the TADS 3 language and libraries help out with. Just to name a couple of cases, I dream of experimenting with undo history and parser grammar beyond what the TADS 3 system and grammar syntax support. This wouldn’t make TADS 3 a bad language for the job, but I expect this would make its domain-specific features into dead weight when they didn’t suit my needs, and when that happens I’m afraid I’ll find myself programming in a domain-limited equivalent of Groovy after all.

Since Then

Between the time I made that post and now, a few months have gone by, and I’ve done a lot of different things:

• I took a deeper look at Inform 7.
• Reassured that neither TADS 3 nor Inform 7 sat perfectly well with me, I spent some time pondering what kind of IF system design I was really looking for.
• I re-implemented my website as a PLT Scheme servlet, and in the process I taught myself PLT Scheme (not to mention what a servlet is). It turns out I have no excuse for continuation dispatching anywhere on my site yet—it’s entirely static, in fact—but that was the draw for me (not to mention that I’ve always wanted to use some kind of lisp on a project).
• I taught myself enough to set up a VPS with an Ubuntu-Nginx-MzScheme stack so that my site would actually run. You can see it now at www.rocketnia.com.
• I refactored some JavaScript code for a Mega Man IV and V password generator I wrote back in 2007. (There are two games, but it’s essentially one generator because the passwords have so much in common.) I fixed some potential memory leaks, separated the code into its own file (rather than dumping it in the same page as the HTML), and cleaned up the API considerably, not only for its own sake but also so that any future password generators I might make could have APIs consistent with it. And then I went and posted it on my website.
• I spent some time working on a Game Boy disassembler in Groovy. It’s actually a pretty awesome disassembler right now, at least for my personal use, but it isn’t exactly an end-user product. It’s organized so as to be extremely configurable (to support SNES or GBA disassembly, for instance), so it’s more like a disassembler library/framework for programmers to use. I’m comfortable with that, too. After all, if you’re using a disassembler… you miiight be a programmer. ^_- Still, a product for programmers could still be packaged up nicely for programmers, and this project isn’t quite there yet.

In this time, I was immersed in a heaping helping of languages (Inform 7, PLT Scheme, Nginx configuration files, XHTML, CSS, JavaScript, Groovy, and Game Boy assembly, not to mention that I lurk on the Arc forum and try to help out with rules debates on the Magic: the Gathering forum), and I had a bunch of thoughts about language design.

I was hoping one of these ideas might fester into meaningful blog post material, but the “next blog post” project hanging in the back of my mind never seemed near to completion. And my theme for this blog is projects I think about but which are potentially too grandiose for me to complete anytime soon. Irony. If only the irony of not having anything to write about writer’s block were worth writing about.

Instead of waiting any longer, I’ll just tell you my half-thought-out ideas and get it over with! ^_^ I may revisit them and flesh them out in the future, if I’m lucky enough to figure out how I want to do that, but for now this is what I’ve got.

Couldn’t Be More Natural

When I was studying Inform 7, the syntax was very overwhelming.

From the Wikipedia article:

“Hello Deductible” by “I.F. Author”

The story headline is “An Interactive Example”.

The Living Room is a room. “A comfortably furnished living room.”
The Kitchen is north of the Living Room.
The Front Door is south of the Living Room.
The Front Door is a door. The Front Door is closed and locked.

The insurance salesman is a man in the Living Room. “An insurance salesman in a tacky polyester suit. He seems eager to speak to you.” Understand “man” as the insurance salesman.

A briefcase is carried by the insurance salesman. The description is “A slightly worn, black briefcase.” Understand “case” as the briefcase.

The insurance paperwork is in the briefcase. The description is “Page after page of small legalese.” Understand “papers” or “documents” or “forms” as the paperwork.

Instead of listening to the insurance salesman for the first time:

say “The salesman bores you with a discussion of life insurance policies. From his briefcase he pulls some paperwork which he hands to you.”; move the insurance paperwork to the player.

This language bears a striking similarity with English, as you might have noticed. When I see this, I can’t help but think that I don’t know what part does what, exactly, and that I’m fooled into reading the surface English meaning rather than the meaning inferred by the compiler. Fortunately, there are plenty of tutorials on the language, and I’m sure it’s quicker to be confident in this syntax than in some others.

While I was distracted by this new syntax, I remembered another syntax in the same sort of family, which I’m quite comfortable with: Magic: The Gathering card syntax.

When Elsewhere Flask enters the battlefield, draw a card.
Sacrifice Elsewhere Flask: Choose a basic land type. Each land you control becomes that type until end of turn.
(Elsewhere Flask)

Split second (As long as this spell is on the stack, players can’t cast spells or activate abilities that aren’t mana abilities.)
Target creature gets +5/+5 and gains shroud until end of turn. (It can’t be the target of spells or abilities.)
(Stonewood Invocation)

If a spell would deal damage to a creature or player, it deals that much damage minus 1 to that creature or player instead.
(Benevolent Unicorn)

It got me to thinking, just how much more can Inform 7 and other programming languages learn from games like Magic that have dynamically changing rules? Maybe some sort of tabletop-game-inspired module system could be cooked up, which would allow for compile units to contradict each other and modify each other, and even ride on the state of a dynamic system, with well-defined behavior… behavior even non-expert humans get the hang of executing.

Right now, it’s just a solution looking for a problem, but I imagine a tabletop-game-inspired programming language or framework would at least be especially well suited to the implementation of tabletop games, if nothing else. Inasmuch as systems like IF works and adventure games can be mocked up in tabletop game dressings, it might be a suitable fit for them, too.

Don’t Pick on Lisps

One thing I’ve noticed about homoiconic programming languages with s-expressions (Common Lisp, Scheme, Emacs Lisp, Clojure, Arc, HL, etc.) is that people lump them together a lot.

See? Even I’m doing it. (I spent the last section lumping legalese dialects together, too. What’s next? Curly-bracket languages?)

A seemingly popular topic among fans (or detractors) of lisps is to talk about why these languages aren’t more popular and what can be or should have been done about that. I’m finding this to be a less and less interesting topic.

Homoiconicity (having code be written in the same format as data literals are) is a great feature in a language if you want to manipulate code as data. Who wants to do that? Programmers. All programmers. Programming is the manual process of manipulating code as data.

That doesn’t mean every programmer benefits from a homoiconic language. It just means, for all the programmers, computer scientists, mathematicians, and philosophers in the world, there are a lot of ways to imagine manipulating code as data (in fact, an insurmountable amount, inasmuch as we would like a language to solve the halting problem), and so a homoiconic language’s work is never done.

Lisps are doing fine. There is no flaw that binds and dooms them all. But they’re not a great end-all and be-all either.

Yes, I’m totally a hypocrite for writing this. I’m talking about lisps without talking in-depth about features of lisps. That’s why I’ve been putting off posting this; I figured I’d come up with something better to say. And I may, but then that would do better as a post of its own rather than one polluted with this rhetoric. ^_^

CSS Is a Hammer, and Keyword Parameters Look like a Nail

I was refactoring my Mega Man IV and V password generator JavaScript code, just minding my business, when suddenly I discovered the joy of keyword parameters.

I wanted the API to be useful in case someone wanted to put multiple copies of one or both generators on the same page, but that wouldn’t be possible under the current setup, which required every form field to have a specific id attribute. Not only would duplicated id attributes on a single page be bad XHTML, but it would also make it impossible for the two generators to distinguish those fields from each other. My solution, for the moment, was to add a parameter to the generator initializer that would specify a prefix to put on each of the ids it expected. This way, two generators could be initialized with different prefixes, and the fields could be labeled in a way that wasn’t ambiguous.

However, this is not a solution that gives complete control to the person using the API. For instance, maybe someday someone will want to use very specific ids on their page for some crazy-but-true reason, and it would be nice if they could specify each and every one of those ids for the initializer… or maybe even a callback function that accepts ids and returns references to page elements on the fly.

Now I have three ideas for how to provide this customization point, one with a quick syntax (just a single prefix string) that probably suffices for all actual cases, one (an id-to-id map) that’s nice for some offbeat cases, and one absurdly powerful option (a callback) that represents what all the other options would ultimately be used to generate anyway. Which should I choose?

Well, I had just come out of programming my website in Scheme, where I had gotten used to using keyword parameters to make them optional.

Also, I couldn’t help but notice that CSS gets away with having a whole lot of border properties that all represent various parts of the same information:

border
border-top        border-right        border-bottom        border-left

border-style
border-top-style  border-right-style  border-bottom-style  border-left-style

border-color
border-top-color  border-right-color  border-bottom-color  border-left-color

border-width
border-top-width  border-right-width  border-bottom-width  border-left-width

I chose to take a lesson from CSS and have the API potentially support many different keyword parameters all for the same thing, giving an error if the caller overspecifies the parameters in a way that doesn’t make sense. Having made that plan, I implemented just the prefix option (all I need so far for personal use) and moved on with my life.

I explained all this because it got me thinking about the relationship between keyword parameters and metadata, as exemplified by CSS. When you’re using CSS and HTML, you “tag” your HTML tags with class keywords and define what those keywords mean in the CSS. Many of these keywords, like border, were taken care of by HTML attributes before CSS came along. Those attributes are no longer necessary because CSS essentially sets them for you. (Technically, it’s more the other way around, I think—as in, style-like HTML attributes are less fundamental to the document’s style representation than CSS is—but I’m going to pretend for now.)

In Groovy, a common idiom is to use “markup,” declarative code that’s shaped like XML. A tag is created by calling a method (the tag’s name) with some keyword arguments (the tag’s attributes) and a closure (the tag’s body), which can contain more method calls like this one. This is done to generate XML code and to construct Swing GUIs, for instance, and it can be used for other kinds of in-language DSLs as well. Regular Groovy code fits right in with this, so it’s easy to use variables and looping constructs for templates and repeated sections.

(I haven’t tested any of the following Groovy, and it isn’t very beautiful code in some places, but the general syntax is all I’m trying to show anyway.)

def show = myMarkupBuilder.slideShow( defaultDuration: 2000.ms ) {
    img( src: "normal-photo.jpg" )
    img( src: "crazy-photo.jpg", zoomFactor: 2, rotationDegrees: 40 )
    vectorArt {
        int r = 10;  // radius of a single circle, and margin between circles
        int n =  5;  // number of circles
        rectangle( left: 0, top: 0, width: 200, height: 4 * r, color: BLUE )
        for ( x in (0..<n).collect { (it * 3 + 1) * r } )
            ellipse( left: x, top: r, width: 2 * r, height: 2 * r, color: YELLOW )
    }
}

But can it do what CSS does?

Before I answer that, I should mention that Nginx configuration files use a syntax very similar to both this and CSS. It’s a header with a tag name and some parameters, followed by a body area within curly brackets, or in some cases followed by no body at all. (Didn’t I tell you there’d be curly-bracket languages? Only one of them is a programming language, but still.)

Back to the question: Can Groovy markup take advantage of CSS techniques? Actually, it can simulate parts of CSS without much effort. For instance, here’s a simple way to define parameters in one place and use them in another, achieving is the kind of abstraction CSS classes provide:

def briefClass = [ duration: 200.ms ];
def subliminalClass = [ duration: 24.ms ];
def crazyClass = subliminalClass + [ center: [ 200, 200 ], src: "crazy-photo.jpg" ];

// A class maker like this can be used when the parameters you want to specify when
// defining the class aren't necessarily the parameters the markup method accepts.
def ellipseClassMaker = { Map options ->
    if ( options == null || options.empty ) return [:];
    return options.collect { k, v -> (
        if ( k == "radius" && v in Number )
            return [ width: v * 2, height: v * 2 ];
        return [ (k): v ];
    ) }.sum();
};

def sampleCircleClass = ellipseClassMaker( radius: 200, color: PURPLE );

def show = myMarkupBuilder.slideShow( defaultDuration: 2000.ms ) {
    img( src: "normal-photo.jpg" )
    img( briefClass + [ src: "brief-photo.jpg" ] )
    for ( frame in 0..40 )
        img( crazyClass + [ zoomFactor: 1 + frame / 40.0, rotationDegrees: frame ] )
    vectorArt { ellipse( sampleCircleClass + [ left: 20, top: 20 ] ) }
}

Unfortunately, this still doesn’t achieve a large part of what makes CSS great. CSS can style an HTML document without using CSS classes at all, by using selectors that apply to nodes based on those nodes’ names and attributes, and by selecting nodes based on their positions relative to nodes from another selection. This sort of recursive data structure querying is a natural fit for Groovy, but if you want to do a query and apply a style, you depend the markup builder to provide enough of the XML-shaped data directly in its result for you to find and modify. And even then, you can’t get all the loop and variable information out of the plain Groovy code you used. (Actually, Groovy does let you manipulate Groovy code as data, but it isn’t easy yet.)

So part of me is wondering, just like with the tabletop-game-inspired programming system, what a stylesheet-inspired programming system would be like. What’s a superficial enough parameter in the code that it’s worthwhile to pretend it isn’t there and let a stylesheet handle it? Hmm, it seems like that would cover just the same (exc)use people find for using global variables, heh. It could also specify what namespace a part of the program uses, but then module import lines do that already. I imagine it could be used to separate the concept of a program from its optimization, in a way that may be similar to Common Lisp “declarations” (not that I’m familiar with those), but I think optimization would need more extensive program transformations than just parameter insertion, so it would just end up being a macro system at some point.

In the end, I don’t really have a clue what CSS-style programming would be good for. As with all these ideas, though, I guess I’ll have it at the back of my mind for when it’s ready.

width

One of the first things I noticed about the IF languages was that superficially, TADS 3 the language is incredibly close to Groovy. It uses C-style imperative syntax, it has closures, it uses dynamic typing, it minimizes the difference between properties and methods, and it has what look like interpolated strings (which print automatically, meaning it takes a bit of output-rerouting to use them for anything else). Now, I’m not saying these are all great features, but I knew right away that if I was going to be comfortable enough in any IF language to implement the kinds of crazy things I find myself wanting to implement all the time, it was going to be TADS. I could port my general-purpose Groovy code to TADS as I needed to.

With that thought on my mind, I realized: Could I port the TADS IF libraries to Groovy? Maybe Groovy could provide the wonderful land of Java libraries and cross-platform-ness that it has, and maybe TADS could provide the wonderful parser it has, and maybe Inform could provide inspiration for some wonderful relational database approaches or something, and they could all come together in a sweet train wreck of paradigms. Sounds good to me… except that Groovy isn’t (in my typical experience) capable of all that TADS is as a language:

• Looping through every existing object is a built-in language feature of TADS, and it makes sense because a game in TADS is primarily composed of a whole lot of singleton objects that get reattached to each other in various ways over the course of a game session. (I suppose the same could be said of a lot of game engines.) It’s like having each superclass keep a list of weak references to its singleton subclass instances. Do draw another comparison, it’s like every class is an enum.
• TADS lets you view the stack trace. You can see which functions are being called, which arguments they’re being passed, where they are in the source code, and maybe a couple of other things (but conspicuously, you can’t see local variables). I think the primary use is for printing stack traces, but I get the impression that it’s also capable of the kind of “rule for printing the name of the lemon sherbet while listing contents” antics that Inform 7 uses (although I do suspect that usage would horribly confuse TADS programmers).
• The syntax for declaring a bunch of simple singletons in a hierarchical is-in relationship is pretty much ideal, in my opinion. It’s about as straightforward-looking as YAML, and yet it’s the same syntax that’s used for every object in the program.
• There’s a special syntax for specifying cross-referenced grammar rules for the purposes of parsing IF commands. I don’t think it’s enormously expressive—it’s BNF without frills like the Kleene star, and subclassing GrammarProd to help with the odd cases looks to be tempting but impossible—but apparently it gets the job done. And with the help of the reflective getGrammarInfo(), you can hijack the whole BNF-style syntax system and interpret it manually, confounding those poor traditional TADS programmers any way you see fit.
• TADS has undo! Sheesh! What programming languages can boast that? Hmmm… okay, anything with program-wide continuations or syntax-friendly monads can, but only sometimes.

With all this in TADS’s favor, is there any reason for me not to switch? Well, for my general programming purposes, there are a few that would come up every so often: lack of thread support, lack of operator overloading, lack of AST transformations, etc. But for the purposes of my IF writing, there were two excuses that nagged at me:

• I feel like I need namespaces. Eventually, I want to make or witness a gargantuan interactive story with, say, hundreds of hand-written multilinear conversation scripts. I don’t have any actual experience making that kind of story, but I’m pretty sure it’s essential to reuse the same names in similar situations. That way you can develop banter on the side in one big brainstorming pool and gradually slip them into bigger conversations like pieces of a jigsaw puzzle without having to worry about introducing name conflicts among subtly different versions of the same basic concept.
• I feel like I need better undo support. The comment on undo() in the TADS system file tadsgen.h says that it’ll “undo all changes to persistent objects, up to the most recent savepoint,” but then it goes on to say that some early undo data might become unavailable because “The system automatically limits the UNDO log’s total memory consumption, according to local system parameters.” If my intuition is right all the UNDO log needs to know is the game transcript plus whatever hidden information was used to determine nondeterministic branching, then the UNDO log shouldn’t be nearly big enough to warrant being recycled for the sake of memory. In fact, I consider it to be the last resort; in case of a memory crisis, if at least the transcript can be dumped, then all the live data of the player’s session, complete with undo capability, can be recovered from that. I fear that what must be happening is that the TADS UNDO log keeps several snapshots of all the game’s memory, or at best keeps tabs on events like object creation and property modification.

Alas, these are only excuses. I’m pretty sure the namespaces I’m looking for can be implemented reasonably well using nested objects, and whatever undo functionality I could implement in Groovy, if it’s any improvement at all, could just be implemented in TADS in almost exactly the same way.

So why do I still feel like I’d be better off in Groovy? Existing Java graph theory libraries to mooch off of? Potential code reuse in story-intensive MU* servers? I don’t know. It’s just an irrational feeling, I think.

Pretty soon, I noticed the same kind of thing happening in all kinds of shows, including Chuck and (I think) CSI. There seems to be a tendency toward a kind of rising hum or twang. Yet, that’s not the only option; Boston Legal’s score does the same thing, but it’s composed of jazzy vocals and stuff.

So, you’ve got rising action, a twist, an obligatory sound effect, and a quick scene change. I guess it’s a formula.

So why do I bring it up? I’ll get to that in a second. (Er, make that several seconds. -Future Me)

There’s a project, Mikkamon, I’ve had in mind for several years now. As a picky reader of interactive stories (picky to the point of not doing it as often as I’d like to), I want to find a work that meets the following criteria:

• It should be a simulation. An interactive story doesn’t technically have to give me adequate evidence with which to make an informed decision… but that kind of interactive story accomplishes little more than purely spectator story does.  I might turn the page, but I can’t blame myself for what happens on the other side. In order for my involvement to make a real difference, there needs to be the sense of a real underlying formula that I’m manipulating.
• It should enable me to be myself. First of all, it shouldn’t be a contest. If there’s only one “right” way to get from here to the final act, I’m having a moral shoved down my throat. Second of all, it should really be me in the main character’s seat, not some everyman protagonist. I’m not really capable of being myself unless I’m capable of frankly asserting to the other characters that they’re all fictional.
• It should have meaningful characters. Video games typically dodge the issue of characters. If the point of the experience is to run and jump, then by golly, the player should be worried about running and jumping, not about how Goomba A is going to respond to seeing Goomba B dive off a cliff. As such, games can get by with having extremely shallow characters who each do one thing and do it consistently. If the game I’m playing isn’t so much a game as it is a story, then what I’m worrying about is character interaction in the first place, and so I need a different sort of character.
• It should support a well-crafted plot structure. If the reader’s character is the reader and the reader is allowed to do pretty much anything, the plot is probably going to suffer a little no matter what the author does. Despite that complication, I think it’s essential that this interactive plot compensate for that, somehow channeling the reader’s own actions into consequences that will persuade him or her to proceed to act in the fashion of an archetypical character. Just because the reader is allowed to break the fourth wall doesn’t mean the fourth wall will sit back and take it. One of the ultimate goals of this is that if a reader wants to show another reader what he or she has done, the other reader should find it interesting.
• It should focus on communicating specifically to me. What’s the point of having a story be interactive if it’s still trying to say the same things to everybody? Since I’m essentially telling it all about myself, the story should figure out who I am and guide me on a tailor-made spirit quest through some of my toughest dilemmas. This requirement goes hand in hand with requiring the plot to be well-crafted: As long as the reader and the protagonist are the same person, the protagonist’s struggle shouldn’t just be an allegory for one of the reader’s struggles. It should be the reader’s struggle. I don’t know if I should require this too strongly. If the story offends absolutely everybody who picks it up and the way they “fix it” is by putting it back down (i.e. refusing to assume the mantle of hero), I guess it won’t be very meaningful for anybody.
• It should allow me to micromanage my character. Significant plot interaction in video games, if it happens at all, happens only two or three times in the entire game, in my experience. Interaction in visual novels I’ve played (read: tried to play) is appalling; I’m given a choice once every five minutes, and the rest of the time the narrator drones on and on about what I’m supposedly doing. Gamebooks and the odd non-visual-novel game disc I’ve gone through usually provide interaction points once per dramatic encounter, which makes them sorta cool, but then they go and spoil it by having several of the “wrong turns” lead to instant death. The way I see it, interactive fiction (the text-based kind), with its “ASK SALLY ABOUT DEATH RAY” tech in combination with finely-branched conversation trees, has what I’m looking for here.
• Finally, it should be humongous. In particular, I want to see something with enough content and depth to fill a 26-episode 22-minute-per-episode TV cartoon show. I estimate that most video game RPGs I’ve played seem to have approximately enough for 52 such episodes, but they have very little plot branching to deal with; everybody sees the same 52 episodes every time. If I want branches to occur several times per conversation, and if I want readers to be essentially capable of doing whatever they want to do, with a medley of intimately personalized plots scrambling to keep up with them, then there’s going to be a whole lot of branching going on. Given the rest of the requirements, this one might be hastily scrapped for being ridiculous, and yet I think it’s actually intrinsically successful. I think it probably takes about 26 episodes’ worth of constant input (and 26 episodes’ worth of subplots) for an interactive story to comprehend a reader and produce something awesome for him or her… and besides, if I make a meaningful impact on the story in the first 10 episodes, I’m spending the next 16 in a world I really like and care about.

Ooh, ooh. I just distracted myself. Maybe the 26 episodes should be divided into three general-purpose acts. In the first act, the reader does pretty much whatever he or she wants to do and has the chance to make an impact on the world. In the second act, based on what impact was made, the story has determined what the reader’s preferences are and injects something that will threaten the reader’s now-home. In the third act, based on the reader’s responses to the second act’s terrorism, either the reader or some other character has become a formidable proponent of some cause or other, so a final countdown commences and, well, stuff happens. It’s not easy to be specific about the end, lol.

Anyway, what does all this have to do with dramatic sound effects? At this point (if you’ve been paying attention ^_- ) it’s probably at least a little obvious. The main hurdle with writing an interactive story like the one I’m looking for is that branches too fast and for too long. If it’s 26 episodes long and it has a yes-or-no branch merely once per episode, that’s 2^25 possible game states during the final episode.

A simple gamebook can’t possibly handle the full complexity of that. (Imagine, “If you choose not to, turn to page 353,261,763!”) Recycling of material and some more intensive corner-cutting mechanisms are in order. Two such mechanisms I’ve thought about are as follows:

• The reader’s character doesn’t have to be in every scene. (Every conversation without that character is a conversation with no branching.) Plenty of stories cut from one set of characters to another, and I don’t see why this one needs to be any different. In fact, it adds some potential dramatic questions: Does the reader take advantage of his or her knowledge of other characters’ secret conversations or not? How does the reader explain away that ESP if he or she does use it? How do the other characters react if they know the reader might be listening in at any moment?
• Procedural generation needs to be just about everywhere. The layout of an episode needs to be procedurally generated in order to account for the author-unpredictable combinations of actions that might take place during a single episode, and character actions need to be procedurally generated in order to account for whatever that character might believe and feel at the time. Even dialogues/scenes, which the characters theoretically determine for themselves, probably need to have procedural nudging (“coaching”?) to make sure that they flow nicely and fit snugly into larger plots.

The dramatic sound effect and, more specifically, the “pithy comment, beat, cut” pattern it’s used in are probably pretty useful to keep in mind when trying to implement both of these mechanisms at once.

Well, Marsyas is an open-source C++ project that apparently is exactly as ambitious as MVTron would like to be in exactly the same ways. Although its main focus is music and other audio, apparently there’s MarsyasX, a branch or something, which is a reimagining of Marsyas to be more video-inclusive. Considering Marsyas’s seeming focus on feature extraction, similarity detection, and… well, all kinds of stuff, it seems like MVTron would be just another application in the sea over there. That takes a lot of (self-inflicted) pressure off of me as a lone programmer. :-p

It almost looks at this point like MVTron will end up being a project entirely submerged in Marsyas, maybe even to the point that it’s written in C++… but I guess I shouldn’t be so hasty. I’ve only just heard of Marsyas, and besides, there’s an entry on their ideas page calling for “Porting the Marsyas dataflow architecture to Java.” ^_-