Parameters Experiment

VSCode

The parameters experiment enables the ability to add parameters to source definitions.

Enable the parameters experiment with:

##! experimental.parameters

This feature is still in development. Feedback is appreciated via Slack or the dedicated GitHub discussion.

TL;DR:

Sources can be parameterized like source: my_source(param::string is "default") is ...
Sources with parameters can be invoked like run: my_source(param is "value") -> { ... }
Parameters can be used in expressions inside the source, in invocations of sources being extended, and in joins
The goal is that eventually parameters will have true lexical scope inside the source, usable anywhere inside the source

##! experimental.parameters

source: flights(min_distance::number) is duckdb.table('../data/flights.parquet') extend {
  measure: flight_count is count()
  where: distance >= min_distance
  view: by_distance_bucket is { 
    group_by: distance_bucket is round(distance, -3) 
    aggregate: c is count() 
  }
}

run: flights(min_distance is 1000) -> by_distance_bucket

QUERY RESULTS

distance_bucket	c
1,000	50,831
2,000	36,305
3,000	2,617
4,000	249

[
  {
    "distance_bucket": 1000,
    "c": 50831
  },
  {
    "distance_bucket": 2000,
    "c": 36305
  },
  {
    "distance_bucket": 3000,
    "c": 2617
  },
  {
    "distance_bucket": 4000,
    "c": 249
  }
]

SELECT 
   ROUND(base."distance", -3) as "distance_bucket",
   COUNT(1) as "c"
FROM '../data/flights.parquet' as base
WHERE base."distance">=1000
GROUP BY 1
ORDER BY 2 desc NULLS LAST

run: flights(min_distance is 2000) -> by_distance_bucket

QUERY RESULTS

distance_bucket	c
2,000	10,786
3,000	2,617
4,000	249

[
  {
    "distance_bucket": 2000,
    "c": 10786
  },
  {
    "distance_bucket": 3000,
    "c": 2617
  },
  {
    "distance_bucket": 4000,
    "c": 249
  }
]

SELECT 
   ROUND(base."distance", -3) as "distance_bucket",
   COUNT(1) as "c"
FROM '../data/flights.parquet' as base
WHERE base."distance">=2000
GROUP BY 1
ORDER BY 2 desc NULLS LAST

New Syntax

Sources can be declared with parameters:

source: my_source(param::string) is ...

Declared parameters may have a type and/or a default value:

source: my_source(param is 1) is ...
source: my_source(param::number is 1 + 1) is ...
Error: source: my_source(param) is ... because it has neither a type nor a default value
Types may be any of the normal field types: string, number, date, timestamp, or boolean
- Error: source: my_source(param is null) because null is not a "regular" Malloy type
- A default value of null may be specified either by specifying a type and default value null or specifying a casted null
  - source: my_source(param::string is null)
  - source: my_source(param is null::string)

Source usages may specify parameters:

run: my_source(param is 1) -> ...
Error: run: my_source(1) because parameter names are required
If a parameter has a default value, it need not be specified:

source: my_source(param is 1) is ...
run: my_source -> { ... } // Ok
run: my_source() -> { ... } // Ok
run: my_source(param is 2) -> { ... } // Ok

If a parameter does not have a default value, it must be specified
Passing 0 params is equivalent to not passing params, so a source without parameters may be invoked run: no_params_source() -> { ... }

New Semantics

Parameters are only inherited from one source to an extended source explicitly:

source: my_source(param is 1) is ...
source: ext_source_1 is my_source(param is 1) // No parameters
source: ext_source_2(param::number) is my_source(param) // One parameter, but without default value

Constant expressions are allowed anywhere where sources can be invoked

run: my_source(param is 1 + 1)
source: new_source is my_source(param is concat("foo", "bar"))
join_one: my_source is my_source(param is 1) on ...
query: my_query is my_source(param is 1) -> ...

Parameters can only be used in a few limited places:

In source invocations in source declarations:
- source: ext_source(ext_param::string) is my_source(param is ext_param) extend ...
- Convenience syntax: my_source(param) is sugar for my_source(param is param):
  - source: ext_source(param::string) is my_source(param) extend ...
In source invocations in join declarations
- source: my_source(param::string) is ... { join_one: other_source is other_source(param) ... }
- source: my_source(param::string) is ... { join_one: other_source with param = other_source.value }
- source: my_source(param::string) is ... { join_one: other_source on param }
In dimension/measure expressions defined in the source:
- source: my_source(param::string) is ... { dimension: param_value is param }
- source: my_source(param::number) is ... { measure: count_times_param is count() * param }
- This includes in SQL functions: dimension: x is sql_number("${ param }") (note this requires the sql_functions experiment)
In the first stage of views defined in the source:
- source: my_source(param::string) is ... { view: my_view is { group_by: param } }
- This includes nests:
  - source: my_source(param::string) is ... { view: my_view is { nest: x is { group_by: param } } }

Parameters cannot be used in the following places (and likely will not in the future):

Anywhere outside of the initial source declaration
- Error: run: my_source(param is 1) -> { group_by: param }

Parameters of a source are not included in index: * or select: *

Limitations

Parameters cannot be used in the following places (but likely will in the future):

In subsequent stages of views defined in the source:
- Error: source: my_src(param::string) is ... { view: my_view is { ... } -> { group_by: param } }
In the sources of queries used to define sources or joins:
- Error: source: my_source(param is 1) is base_source(param) -> { ... }
- Error: source: my_source(param is 1) ... { join_one: joined is other(param) -> { ... } on ...
In SQL sources:
- Error: source: my_source(param is 1) is conn.sql("""%{ other(param) -> { ... } }""")
- Error: source: my_source(param is 1) is conn.sql("SELECT * ... WHERE %{ param } = 1")

Parameters cannot currently be annotated (but they likely will be in the future)

source: my_source(
  # param_tag=1 // <— syntax error
  param::string
) is ...

For the time being, shadowing of parameters by fields or vice versa is illegal

source: my_source(param::string) is ... extend {
  dimension: param is ... // Error
}

source: base_source is ... extend {
  dimension: param
}

source: my_source(
  param::string  // Error
) is base_source

Likely in the future this will be legal, but there will be some other syntax to help navigate conflicts