Change Log

VSCode

Breaking changes indicated with *

We will use this space to highlight major and/or breaking changes to Malloy.

Malloy 4.0 Preview (v0.0.40 - v0.0.86)

In this set of changes, several new syntactical constructs have been introduced, in some cases directly replacing existing syntaxes. The old syntaxes will be deprecated in version 4.0. For the time being, both sets of syntaxes are legal. To preview warnings (and eventually errors) that will be added in 4.0, you can include the m4warnings compiler flag in your source code: ##! m4warnings.

Views

To resolve some confusion about the difference between "source-level queries" and "model-level queries," source-level queries have been renamed to "views." To define views in a source, use view: instead of query:. The latter will be deprecated in 4.0. At the top level of a model, queries can still be defined and named with query:.

For more information about views, see the Views section.

Connection Table Method

New syntax for defining sources based on tables, duckdb.table('data/users.parquet') has been introduced. The old syntax, table('duckdb:data/users.parquet') still works for the time being, but will be deprecated in 4.0.

See the Connections section for more details.

SQL Source Method

New syntax for defining sources based on SQL queries, duckdb.sql("""select * ... """), has been introduced. The old syntax, sql: name is { select: """select * ..."""; connection: "duckdb" } still works for the time being, but will be deprecated in 4.0.

This makes from_sql no longer necessary, and it will be deprecated with the sql: statement.

See the SQL Sources section for more details.

Source Extensions and Query Refinements

Previously, there was one gesture used for source extensions and query refinements, source_or_query { extensions_or_refinements }. Now there is new sepearate syntax for each: some_source extend { source_extensions } and some_query + { source_refinements }. The old syntax still works for the time being, but will be deprecated in 4.0.

This makes from no longer required, and it will be deprecated with the old extension/refinement syntax.

See the Query Refinement and Source Extension sections for more details.

Deprecation of Leading Arrows for Query References

Previously in order to reference an existing query (e.g. to use as the source for another query), it was necessary to include a leading arrow: run: -> some_query -> { select: * }. Now the leading arrow is no longer required, and will be deprecated in 4.0.

Addition of Leading Arrow for Queries Defined in Sources

An optional leading arrow can now be placed in query definitions inside of sources:

source: my_source is duckdb.table('some_table') extend {
  view: my_query is {
    ...
  }
}

Deprecation of The Filter Shortcut

Previously you could use a shortcut syntax for filtering sources, queries, or measures, {? condition }. This will be deprecated in version 4.0.

Instead of the shortcut, use source_name extend { where: condition }, query_name + { where: condition }, or measure_name { where: condition }.

New Syntax for Source Extensions Defined in Queries

There is new syntax for specifying source extensions inside queries:

query: my_query is some_source -> {
  extend: {
    dimension: value is 1
  }
  group_by: value
}

This will replace declare: and join_*: in queries, and they will be deprecated in 4.0.

See the Source extensions section for more details.

Run Statement

There is new syntax for specifying an unnamed query that should be runnable in the host application, run: flights -> by_carrier. This will replace the old syntax, query: flights -> by_carrier, which still works for the time being, but will be deprecated in 4.0. See the Run Statement section for details.

v0.0.39

*More strict expression types

Previously, group_by:, nest:, aggregate:, dimension:, measure:, etc. would all allow fields of the wrong expression type (dimension, measure, query). Now that is an error.

*Function argument typechecking

Function arguments are now typechecked for a set of "built-in" functions. All unknown functions will error. See the functions documentation and a detailed description of what changed.

Forced function call syntax

For functions which are not "built-in", or to call the native SQL version of a function, there is special syntax:

// Exclamation point indicates to not typecheck arguments and directly call the named SQL function
// Return type is by default the same as the first argument
dimension: cuberoot_value is cbrt!(value)
// A return type may be specified after the exclamation point
dimension: timestamp_value is timestamp_seconds!timestamp(value)

Calculations (analytic functions / window functions)

There is a new keyword calculate: which can appear in a query, for use with window functions.

run: flights -> { 
  group_by: carrier
  calculate: prev_carrier is lag(carrier)
}

QUERY RESULTS

HTML JSON SQL

[
  {
    "carrier": "AA",
    "prev_carrier": null
  },
  {
    "carrier": "AS",
    "prev_carrier": "AA"
  },
  {
    "carrier": "B6",
    "prev_carrier": "AS"
  },
  {
    "carrier": "CO",
    "prev_carrier": "B6"
  },
  {
    "carrier": "DL",
    "prev_carrier": "CO"
  }
]

SELECT 
   base."carrier" as "carrier",
   LAG((base."carrier")) OVER(  ORDER BY  base."carrier" asc NULLS LAST ) as "prev_carrier"
FROM '../data/flights.parquet' as base
GROUP BY 1
ORDER BY 1 asc NULLS LAST

*New functions are in the global namespace

New functions are in the global namespace, which means that top level objects (SQL blocks, queries, sources) in a Malloy file cannot have the same name as a built-in function.

v0.0.10

The apply operator is now ? and not :

In the transition from filters being with an array like syntax ...

sourceName :[ fieldName: value1|value2 ]

The use of : as the apply operator became a readability problem ...

sourceName { where: fieldName: value1|value2 }

As of this release, use of the : as an apply operator will generate a warning, and in a near future release it will be a compiler error. The correct syntax for apply is now the ? operator. As in

sourceName { where: fieldName ? value1|value2 }

0.0.9

Deprecation of brackets for lists of items

Prior to version 0.0.9, lists of things were contained inside [ ]. Going forward, the brackets have been removed. Our hope is that this will be one less piece of punctuation to track, and will make it easier to change from a single item in a list to multiple without adding in brackets.

For example, this syntax:

query: duckdb.table('../data/airports.parquet') -> {
  top: 10
  group_by: [
    faa_region
    state
  ]
  aggregate: [
    airport_count is count()
    count_public is count() { where: fac_use = 'PU' },
    average_elevation is round(elevation.avg(),0)
  ]
  where: [
    faa_region: 'ANM' | 'ASW' | 'AWP' | 'AAL' | 'ASO' ,
    major = 'Y' ,
    fac_type = 'AIRPORT'
  ]
}

Is now written:

query: duckdb.table('data/airports.parquet') -> {
  top: 10
  group_by:
    faa_region
    state
  aggregate:
    airport_count is count()
    count_public is count() { where: fac_use = 'PU' },
    average_elevation is round(elevation.avg(),0)
  where:
    faa_region: 'ANM' | 'ASW' | 'AWP' | 'AAL' | 'ASO' ,
    major = 'Y' ,
    fac_type = 'AIRPORT'
}