Re: Data Quality Monitoring using Data Metric Functions Snowflake

An detailed video overview of the Data Metric Functions in Snowflake:

https://www.youtube.com/watch?v=M4MadvJH_Dc

Saqib

On Mon, Jun 10, 2024 at 8:54 PM Ali, Saqib <docbook.xml@gmail.com> wrote:

Another excellent blogpost on Snowflake Data Metric Functions (DMF) by Peter Beck:
https://medium.com/snowflake/monitoring-data-quality-using-snowflake-data-metric-functions-6b04229c6597

Couple of interesting insights from the article:

1. Snowflake runs DMFs under a warehouse it creates on your behalf. You are billed for this consumption under a special "Data Quality Monitoring" section of your bill. This enables you to see immediately what the costs associated with this kind of activity are. You can also query DATA_QUALITY_MONITORING_USAGE_HISTORY to see your consumption.
2. Scheduling of DMFs is simple, and you don't need to build your own target table to store the results.

Saqib

On Thu, May 30, 2024 at 9:55 PM Ali, Saqib <docbook.xml@gmail.com> wrote:

Another excellent blogpost on Snowflake Data Metric Functions (DMF) by Arihant Shashank:
https://medium.com/@arihant.shashank01/native-custom-data-quality-monitoring-in-snowflake-fcf002ec9afb

On Tue, May 7, 2024 at 8:42 AM Ali, Saqib <docbook.xml@gmail.com> wrote:

Another excellent blogpost on Snowflake Data Metric Functions (DMF) by Maja Ferle:

https://medium.com/snowflake/monitoring-data-quality-natively-in-snowflake-6cdbcef70de5

saqib

On Sat, Apr 27, 2024 at 12:30 PM Ali, Saqib <docbook.xml@gmail.com> wrote:

here is another blogpost on Data Metrics Functions in Snowflake by Mandar Athavale:

https://medium.com/@athavale.mandar/snowflake-data-metric-functions-and-alerts-21d610bbcb67

Saqib

On Wed, Apr 17, 2024 at 7:50 AM Ali, Saqib <docbook.xml@gmail.com> wrote:

Also, Jessica Zhang has blogged the Built-in Data Quality Metrics in Snowflake:

https://medium.com/@jesszhangcyz/monitor-data-pipelines-using-snowflakes-data-metric-functions-0df71c46f04a

On Thu, Apr 11, 2024 at 10:20 AM Ali, Saqib <docbook.xml@gmail.com> wrote:

Augusto Kiniama Rosa also has a detailed blogpost on the Built-in Data Quality Metrics in Snowflake:
https://medium.com/snowflake/snowflake-built-in-quality-metrics-3417d7d9f040

On Mon, Apr 8, 2024 at 8:55 PM Ali, Saqib <docbook.xml@gmail.com> wrote:

A nice overview of Data Quality Monitoring using Data Metric Functions Snowflake by Sandamali De Zoysa:

https://www.youtube.com/watch?v=cwHLfWg6Da4

Saqib

Re: Snowflake now supports RANGE Clause in Window Functions

A detailed blogpost by Yogitha Chilukuri on the RANGE Clause in Snowflake Windows Function:

https://medium.com/snowflake/now-in-public-preview-snowflakes-range-based-window-frame-will-accelerate-your-time-series-d7b0699f8344

On Mon, Jul 8, 2024 at 7:06 AM Ali, Saqib <docbook.xml@gmail.com> wrote:

Long awaited RANGE Clause in Window Functions is now available in Snowflake. Here are some practical examples of using RANGE Clause in the Window Functions:

https://qosf.com/RANGE-clause-in-window-functions-practical-examples.html

Snowflake now supports RANGE Clause in Window Functions

Long awaited RANGE Clause in Window Functions is now available in Snowflake. Here are some practical examples of using RANGE Clause in the Window Functions:

https://qosf.com/RANGE-clause-in-window-functions-practical-examples.html

Snowflake now supports Variable Binding in Queries using TO_QUERY()

Snowflake now supports Variable Binding in Queries using TO_QUERY():

https://docs.snowflake.com/en/user-guide/querying-construct-at-runtime

https://docs.snowflake.com/en/sql-reference/functions/to_query

Looking forward to trying it out soon.

JSON is now a first class citizen in Azure SQL Server

Azure SQL now supports JSON DataType. It is a binary data type that is way more performant than using a non-binary datatype to store JSON.

 

https://www.youtube.com/watch?v=WigP5PLQXhY

More info:

https://learn.microsoft.com/en-us/sql/t-sql/data-types/json-data-type?view=azuresqldb-current

https://learn.microsoft.com/en-us/sql/t-sql/functions/json-objectagg-transact-sql?view=azuresqldb-current

https://learn.microsoft.com/en-us/sql/t-sql/functions/json-arrayagg-transact-sql?view=azuresqldb-current

Re: dbt snapshots as a first class citizen

This is scheduled for Jun 13, 2024 08:00 AM Pacific Time (US and Canada). Please register here if you are interested:
https://dbtlabs.zoom.us/meeting/register/tZIpdu6tpjkuGNws_4KvdwezAva62xp0COlt#/registration

Saqib

On Sun, Jun 9, 2024 at 9:02 PM Ali, Saqib <docbook.xml@gmail.com> wrote:

If you have worked with DBT Snapshot before, you know that it has some challenges. DBT is working on addressing these challenges. DBT is hosting a session to discuss the challenges and potential design approaches:
https://dbtlabs.zoom.us/meeting/register/tZIpdu6tpjkuGNws_4KvdwezAva62xp0COlt#/registration

Please sign up if you are interested. Here is a detailed discussion on the issues and challenges with DBT Snapshots:

https://github.com/dbt-labs/dbt-core/discussions/7018

Saqib

Re: Data Quality Monitoring using Data Metric Functions Snowflake

Another excellent blogpost on Snowflake Data Metric Functions (DMF) by Peter Beck:
https://medium.com/snowflake/monitoring-data-quality-using-snowflake-data-metric-functions-6b04229c6597

Couple of interesting insights from the article:

1. Snowflake runs DMFs under a warehouse it creates on your behalf. You are billed for this consumption under a special "Data Quality Monitoring" section of your bill. This enables you to see immediately what the costs associated with this kind of activity are. You can also query DATA_QUALITY_MONITORING_USAGE_HISTORY to see your consumption.
2. Scheduling of DMFs is simple, and you don't need to build your own target table to store the results.

Saqib

On Thu, May 30, 2024 at 9:55 PM Ali, Saqib <docbook.xml@gmail.com> wrote:

Another excellent blogpost on Snowflake Data Metric Functions (DMF) by Arihant Shashank:
https://medium.com/@arihant.shashank01/native-custom-data-quality-monitoring-in-snowflake-fcf002ec9afb

On Tue, May 7, 2024 at 8:42 AM Ali, Saqib <docbook.xml@gmail.com> wrote:

Another excellent blogpost on Snowflake Data Metric Functions (DMF) by Maja Ferle:

https://medium.com/snowflake/monitoring-data-quality-natively-in-snowflake-6cdbcef70de5

saqib

On Sat, Apr 27, 2024 at 12:30 PM Ali, Saqib <docbook.xml@gmail.com> wrote:

here is another blogpost on Data Metrics Functions in Snowflake by Mandar Athavale:

https://medium.com/@athavale.mandar/snowflake-data-metric-functions-and-alerts-21d610bbcb67

Saqib

On Wed, Apr 17, 2024 at 7:50 AM Ali, Saqib <docbook.xml@gmail.com> wrote:

Also, Jessica Zhang has blogged the Built-in Data Quality Metrics in Snowflake:

https://medium.com/@jesszhangcyz/monitor-data-pipelines-using-snowflakes-data-metric-functions-0df71c46f04a

On Thu, Apr 11, 2024 at 10:20 AM Ali, Saqib <docbook.xml@gmail.com> wrote:

Augusto Kiniama Rosa also has a detailed blogpost on the Built-in Data Quality Metrics in Snowflake:
https://medium.com/snowflake/snowflake-built-in-quality-metrics-3417d7d9f040

On Mon, Apr 8, 2024 at 8:55 PM Ali, Saqib <docbook.xml@gmail.com> wrote:

A nice overview of Data Quality Monitoring using Data Metric Functions Snowflake by Sandamali De Zoysa:

https://www.youtube.com/watch?v=cwHLfWg6Da4

Saqib