Announcing Malloy 4.0

VSCode

The next major update to the Malloy syntax is here.

October 3, 2023 by the Malloy Team

While there was never an official "3.0" version we are calling this release 4.0 because this is the fourth major renovation of the language. It could also be thought of as "1.0" because this is when we stop using the word "experimental" to describe the language. At this point we have experimented enough. We believe Malloy is now the best language for working with data stored in an SQL server.

If you haven't tried Malloy yet, you can try it right now with just your browser by

• going to the Malloy Examples Repository on Github

• log into Github

• press '.' (the period key) to launch VSCode

• Install the Malloy Extension

The 4.0 Release Timeline

While Malloy 4.0 was in development, it was considered acceptable to make breaking changes to the language as we discovered better ways to express data computations. Beginning with 4.0, it is our intention to keep code compatibility for existing code, even as we continue to add capabilities to the language.

The rollout of 4.0 will happen in stages so users will have time to upgrade their code before it stops working.

4.0 Differences

Incompatible Changes

With the Malloy 4.0 language changes, we're making Malloy more consistent readable and learnable. Some of these changes will require users to edit existing code. The IDE will issue warnings to flag code which needs to change, once the rollout begins. For examples showing both the old and new syntaxes, refer to the 4.0 Messages FAQ document.

• Source extensions now use extend {}

• Query refinement now uses + {}

• Reusable query operations in a source are now defined with view: instead of query:

• Source extensions in a query must be contained in an extend: { }

  • The declare: statement has been removed, use dimension: or measure:

  • join_: statements in a query must be in the extend: {} block

• The new run statement, run: replaces query: with no name

• The sql: statement is replaced by SQL sources, e.g. duckdb.sql("SELECT ...")

• Table sources have new syntax: connection_name.table('table_path') instead of table('connection_name:table_path')

• Projections are performed with select: instead of project:

• The from() function to create a source from a query is no longer needed

• When nesting Malloy code inside a SQL string, the syntax is %{ malloy query } and not %{ malloy query }%

• Explicit aggregate locality (using source.) is now required in some cases for sum() and avg()

• The filter shortcut {? } has been removed

• The expression to run an existing query is now query_name instead of -> query_name

• The -> after the is in a view: or nest: statement is not longer needed

• New syntax for distinct counts, count(expression) instead of count(distinct expression)

• The syntax for counts is now just count(), and count(*) is deprecated

• Malloy has a function library for doing common data manipulation operations

  • Access to the database's internal functions directly must be through the raw sql function syntax

New Features

These features are all active now. Most are not "new" in that they have already appeared in the Release version of the Malloy VS Code extension, even though they were part of the 4.0 project. Code which uses these will continue to work. They are listed here to provide a complete record of "What Is Malloy 4.0."

• There are three kinds of string literals, ', ", and """, which can be used interchangeably anywhere a literal is legal

• Casting to a specific database type is now legal using a quoted type name

• New safe cast operator :::

• Addition of annotations and tags

• New null-coalescing operator ??

• Selective imports of objects from other files

• Analytic/window functions using calculate: