In this post we are going to cover tasks in Snowflake.
A task is a piece of SQL that Snowflake runs for you on a schedule. It is the only object in this course that acts with no session behind it, which means the interesting part is not the SQL it runs. It is the switches between existing and running, and there are three ways a task can look perfectly fine and never do anything.
We will start with one task that writes one row, and build it up step by step. Every query below was run on a real Snowflake account, and every result you see is the output it gave back.
Episode 137 of the Snowflake SQL: Zero to Hero series on TechBrothersIT.
- Let's create some sample data
- Now let's create a task
- What happens if we run it by hand?
- Now let's resume it properly
- And the shortest schedule it will take
- A schedule can be a cron string instead
- What happens if we leave the warehouse out?
- A WHEN condition, and what SKIPPED means
- And now the refusals
- And the cleanup, which matters more here than usual
Let's create some sample data
A task needs somewhere to write, so that we can tell whether it ran. One table, two columns, and a default timestamp so every row records when it arrived.
-- Sample data for this post.
USE DATABASE DEMO_DB;
USE SCHEMA PUBLIC;
-- The table the task is going to write into.
CREATE OR REPLACE TABLE HEARTBEAT_T137 (
NOTE VARCHAR(40),
WRITTEN TIMESTAMP_NTZ DEFAULT CURRENT_TIMESTAMP()
);SELECT COUNT(*) AS ROWS_IN_IT FROM HEARTBEAT_T137;| ROWS_IN_IT |
|---|
| 0 |
Empty. And here is where the session is pointing, because a task is going to need a warehouse by name.
SELECT CURRENT_DATABASE() AS THE_DATABASE,
CURRENT_SCHEMA() AS THE_SCHEMA,
CURRENT_WAREHOUSE() AS THE_WAREHOUSE,
CURRENT_VERSION() AS THE_VERSION;| THE_DATABASE | THE_SCHEMA | THE_WAREHOUSE | THE_VERSION |
|---|---|---|---|
| DEMO_DB | PUBLIC | COMPUTE_WH | 10.35.101 |
Now let's create a task
A warehouse, a schedule of one minute, and one INSERT as the body. The body goes after the word AS and it is a single statement.
CREATE OR REPLACE TASK BEAT_T137
WAREHOUSE = COMPUTE_WH
SCHEDULE = '1 MINUTE'
AS
INSERT INTO HEARTBEAT_T137 (NOTE) VALUES ('the task ran');| status |
|---|
| Task BEAT_T137 successfully created. |
Created. Now the obvious next move is to ask the catalog what it made, with SHOW TASKS, and DESCRIBE TASK for one of them.
SHOW TASKS LIKE '%\_T137';
DESCRIBE TASK BEAT_T137;Those two grids are not on this page, and that is deliberate. A task listing has twenty seven columns and one of them is created_by_user, which holds the account's own user name. Publishing a grid that identifies somebody is not worth a screenshot, and writing its values into a sentence instead would be worse, because every result on this blog is supposed to be a captured grid you can check. So the two statements are here to run yourself, and everything this post claims is read off a grid that has no such column.
Two things are worth saying about those statements anyway, and neither of them is a value. The first is that DESCRIBE TASK gives you the same twenty seven columns SHOW TASKS does, in the same order. It is not a column list for the task's output, the way DESCRIBE TABLE is. DESCRIBE TASK is SHOW TASKS for one task.
The second is that the columns worth looking for in it are warehouse, schedule, predecessors, state, definition and condition. The one that decides everything in this post is state.
The documentation is explicit that a new task is created in a suspended state. Nobody asks for that, it is simply how a task arrives. So does our task exist, and has it done anything?
SELECT COUNT(*) AS ROWS_IN_IT FROM HEARTBEAT_T137;| ROWS_IN_IT |
|---|
| 0 |
Zero. The task exists and it has done nothing at all, and it will carry on doing nothing for as long as you leave it alone. That is the first of the three ways a task can look fine and never run.
What happens if we run it by hand?
EXECUTE TASK fires a task now, outside its schedule. Let's try it on a task that is still suspended.
EXECUTE TASK BEAT_T137;| status |
|---|
| Task BEAT_T137 is scheduled to run immediately. |
Accepted. Task BEAT_T137 is scheduled to run immediately. So let's ask the table what arrived.
SELECT NOTE, COUNT(*) AS HOW_MANY
FROM HEARTBEAT_T137
GROUP BY NOTE;| NOTE | HOW_MANY |
|---|
Nothing. Not one row.
That is worth stopping on, because the statement we just ran said the task was running. So let's ask the history instead of the table.
SELECT NAME,
STATE,
SCHEDULED_FROM,
ERROR_CODE,
ERROR_MESSAGE
FROM TABLE(INFORMATION_SCHEMA.TASK_HISTORY(
TASK_NAME => 'BEAT_T137'))
ORDER BY SCHEDULED_TIME;| NAME | STATE | SCHEDULED_FROM | ERROR_CODE | ERROR_MESSAGE |
|---|---|---|---|---|
| BEAT_T137 | SUCCEEDED | EXECUTE TASK | NULL | NULL |
| BEAT_T137 | SUCCEEDED | SCHEDULE | NULL | NULL |
| BEAT_T137 | SUCCEEDED | EXECUTE TASK | NULL | NULL |
| BEAT_T137 | SUCCEEDED | SCHEDULE | NULL | NULL |
| BEAT_T137 | EXECUTING | EXECUTE TASK | NULL | NULL |
Read this grid carefully, because it holds more than the run we just fired. TASK_HISTORY returns recent runs for a task name, and this demo file had been run before, so the rows above the last one are from an earlier execution. Two of them say SCHEDULED_FROM of SCHEDULE, and our task has never been resumed.
The row that belongs to us is the last one, ordered by scheduled time, and it still says EXECUTING.
So the task did run. Our SELECT simply ran first. EXECUTE TASK returns as soon as the run is queued, not when it finishes. Anything you check on the next line is a race, and this demo loses it on purpose so you can see it happen.
That is the second way a task can look fine and never run. It ran. You checked too early.
One more question while we are here: did firing it by hand wake it up? The answer is in a SHOW TASKS listing, so it is not a grid this post can publish — but you can see it in the next section without the catalog at all.
Now let's resume it properly
Count the rows first, so there is something to compare against.
SELECT COUNT(*) AS ROWS_BEFORE FROM HEARTBEAT_T137;| ROWS_BEFORE |
|---|
| 0 |
Zero. Which is the answer to the question we just left hanging: the hand run from the last section has not written anything yet, and no schedule has fired either, because the task is still suspended. Firing a task by hand does not resume it.
So let's resume it.
ALTER TASK BEAT_T137 RESUME;| status |
|---|
| Statement executed successfully. |
Now it is on its own schedule. And now a wait.
SELECT SYSTEM$WAIT(70) AS WAITED;| WAITED |
|---|
| waited 70 seconds |
SYSTEM$WAIT is a wait, not a measurement. It is here so the task has time to fire, and it says nothing whatever about how long anything took. Nothing in this course is timed.
And let's suspend it again straight away, so nothing else fires while we look.
ALTER TASK BEAT_T137 SUSPEND;| status |
|---|
| Statement executed successfully. |
SELECT COUNT(*) AS ROWS_AFTER FROM HEARTBEAT_T137;| ROWS_AFTER |
|---|
| 2 |
Two rows. The hand run from the previous section finally landed, and the schedule fired once. Our table was created fresh at the top of this post, so those two rows are this run's and nothing else's.
SELECT NAME,
STATE,
SCHEDULED_FROM,
ERROR_CODE
FROM TABLE(INFORMATION_SCHEMA.TASK_HISTORY(
TASK_NAME => 'BEAT_T137'))
ORDER BY SCHEDULED_TIME;| NAME | STATE | SCHEDULED_FROM | ERROR_CODE |
|---|---|---|---|
| BEAT_T137 | SUCCEEDED | EXECUTE TASK | NULL |
| BEAT_T137 | SUCCEEDED | SCHEDULE | NULL |
| BEAT_T137 | SUCCEEDED | EXECUTE TASK | NULL |
| BEAT_T137 | SUCCEEDED | SCHEDULE | NULL |
| BEAT_T137 | SUCCEEDED | EXECUTE TASK | NULL |
| BEAT_T137 | SUCCEEDED | SCHEDULE | NULL |
The history shows both kinds of run, all succeeded. It also shows more rows than our table has, for the same reason as before: it is recent history for the name, not a record of this run. The table counts what this run wrote. The history counts what the name has done lately.
And the shortest schedule it will take
Here is something this demo had wrong. The shortest schedule a task takes is widely said to be one minute, and the wait above was written on that belief. So let's ask for one second.
CREATE OR REPLACE TASK TOOFAST_T137
WAREHOUSE = COMPUTE_WH
SCHEDULE = '1 SECOND'
AS
INSERT INTO HEARTBEAT_T137 (NOTE) VALUES ('too fast');091851 (42601): Cannot set schedule less than 10 seconds.
091851, Cannot set schedule less than 10 seconds. Which names the real floor, so let's take it at its word.
CREATE OR REPLACE TASK TENSEC_T137
WAREHOUSE = COMPUTE_WH
SCHEDULE = '10 SECOND'
AS
INSERT INTO HEARTBEAT_T137 (NOTE) VALUES ('ten seconds');| status |
|---|
| Task TENSEC_T137 successfully created. |
Accepted. And one second under it, to be sure the floor is where the message says it is.
CREATE OR REPLACE TASK NINESEC_T137
WAREHOUSE = COMPUTE_WH
SCHEDULE = '9 SECOND'
AS
INSERT INTO HEARTBEAT_T137 (NOTE) VALUES ('nine seconds');091851 (42601): Cannot set schedule less than 10 seconds.
Refused, with the same 091851. So the floor is ten seconds, the belief about one minute was wrong, and the wait in this demo was longer than it ever needed to be.
A schedule can be a cron string instead
An interval is one way to write a schedule. A cron expression is the other. This one is six in the morning, every day, in UTC.
CREATE OR REPLACE TASK CRONBEAT_T137
WAREHOUSE = COMPUTE_WH
SCHEDULE = 'USING CRON 0 6 * * * UTC'
AS
INSERT INTO HEARTBEAT_T137 (NOTE) VALUES ('the cron task ran');| status |
|---|
| Task CRONBEAT_T137 successfully created. |
Created, and suspended like any other task. So its history should be empty.
SELECT NAME, STATE, SCHEDULED_TIME, COMPLETED_TIME, SCHEDULED_FROM
FROM TABLE(INFORMATION_SCHEMA.TASK_HISTORY(
TASK_NAME => 'CRONBEAT_T137'))
ORDER BY SCHEDULED_TIME;| NAME | STATE | SCHEDULED_TIME | COMPLETED_TIME | SCHEDULED_FROM |
|---|
Zero rows. A suspended task that has never been fired has no history at all, which is a useful thing to recognise: an empty history grid is not a sign that something broke.
And while we are here, it is worth knowing what TASK_HISTORY actually gives you, because the five columns we have been selecting are not all of them.
SELECT *
FROM TABLE(INFORMATION_SCHEMA.TASK_HISTORY(
TASK_NAME => 'BEAT_T137'))
LIMIT 2;That grid is not on this page either: its rows carry a QUERY_ID and a SCHEDULED_BY_USER, and neither of those belongs on a blog. What is safe to say is the shape. It returns twenty eight columns, and the ones worth knowing about are NEXT_SCHEDULED_TIME, RUN_ID, ATTEMPT_NUMBER, GRAPH_VERSION and CONDITION_TEXT. There is more in there than the five columns everybody selects.
What happens if we leave the warehouse out?
A task normally names the warehouse it runs on. Let's write one with no WAREHOUSE clause at all and see whether it is even accepted.
CREATE OR REPLACE TASK SERVERLESS_T137
SCHEDULE = '1 MINUTE'
AS
INSERT INTO HEARTBEAT_T137 (NOTE) VALUES ('the serverless task ran');| status |
|---|
| Task SERVERLESS_T137 successfully created. |
Accepted, which on a trial account is not what I expected. A task with no warehouse clause runs on serverless compute, and the way you ask for serverless compute is by not asking for a warehouse.
Let's fire it by hand and read the table the instant the statement returns.
EXECUTE TASK SERVERLESS_T137;| status |
|---|
| Task SERVERLESS_T137 is scheduled to run immediately. |
SELECT NOTE, COUNT(*) AS HOW_MANY
FROM HEARTBEAT_T137
GROUP BY NOTE
ORDER BY NOTE;| NOTE | HOW_MANY |
|---|---|
| the task ran | 2 |
Only the old rows, under the old note. The serverless task's row is not there.
So let's wait, and ask again.
SELECT SYSTEM$WAIT(20) AS WAITED;| WAITED |
|---|
| waited 20 seconds |
SELECT NOTE, COUNT(*) AS HOW_MANY
FROM HEARTBEAT_T137
GROUP BY NOTE
ORDER BY NOTE;| NOTE | HOW_MANY |
|---|---|
| the serverless task ran | 1 |
| the task ran | 2 |
And now there are two notes, and the serverless one has written its row. The same asynchrony as before, on a different kind of compute.
That row is also the proof that it ran at all. We never named a warehouse anywhere, and work still happened.
A WHEN condition, and what SKIPPED means
A task can carry a condition that decides whether its body runs. The usual one is "only run if this stream has something in it", so let's build that.
CREATE OR REPLACE TABLE FEED_T137 (
SALE_ID INT,
AMOUNT NUMBER(10,2)
);| status |
|---|
| Table FEED_T137 successfully created. |
CREATE OR REPLACE STREAM FEEDSTRM_T137 ON TABLE FEED_T137;| status |
|---|
| Stream FEEDSTRM_T137 successfully created. |
SELECT SYSTEM$STREAM_HAS_DATA('FEEDSTRM_T137') AS THE_STREAM_HAS_DATA;| THE_STREAM_HAS_DATA |
|---|
| FALSE |
FALSE. Remember that. Now a table for the guarded task to write into, and the task itself. The WHEN clause sits between the schedule and the body.
CREATE OR REPLACE TABLE LANDED_T137 (
SALE_ID INT,
AMOUNT NUMBER(10,2)
);| status |
|---|
| Table LANDED_T137 successfully created. |
CREATE OR REPLACE TASK GUARDED_T137
WAREHOUSE = COMPUTE_WH
SCHEDULE = '1 MINUTE'
WHEN
SYSTEM$STREAM_HAS_DATA('FEEDSTRM_T137')
AS
INSERT INTO LANDED_T137 (SALE_ID, AMOUNT)
SELECT SALE_ID, AMOUNT FROM FEEDSTRM_T137 WHERE METADATA$ACTION = 'INSERT';| status |
|---|
| Task GUARDED_T137 successfully created. |
The catalog keeps that expression in a column of its own, called condition, separate from the definition column that holds the body. You can see both in SHOW TASKS, which is one of the listings this post does not publish.
Now let's run it by hand with the stream still empty.
EXECUTE TASK GUARDED_T137;| status |
|---|
| Task GUARDED_T137 is scheduled to run immediately. |
SELECT COUNT(*) AS ROWS_THAT_LANDED FROM LANDED_T137;| ROWS_THAT_LANDED |
|---|
| 0 |
Nothing landed. But we know by now that could just be the race again, so the table cannot answer this. Ask the history.
SELECT NAME, STATE, ERROR_CODE, ERROR_MESSAGE
FROM TABLE(INFORMATION_SCHEMA.TASK_HISTORY(
TASK_NAME => 'GUARDED_T137'))
ORDER BY SCHEDULED_TIME;| NAME | STATE | ERROR_CODE | ERROR_MESSAGE |
|---|---|---|---|
| GUARDED_T137 | SKIPPED | 0040003 | Conditional expression for task evaluated to false. |
| GUARDED_T137 | SUCCEEDED | NULL | NULL |
| GUARDED_T137 | SKIPPED | 0040003 | Conditional expression for task evaluated to false. |
| GUARDED_T137 | SUCCEEDED | NULL | NULL |
| GUARDED_T137 | SCHEDULED | NULL | NULL |
SKIPPED, with error code 0040003 and the message Conditional expression for task evaluated to false. Not failed. Not succeeded. Skipped, which is a state of its own.
The other rows in that grid are earlier runs of the same name again. The one that answers our question is the most recent SKIPPED row.
So EXECUTE TASK does evaluate WHEN. It is a real guard, not a hint for the scheduler, and running a task by hand does not get you past it.
Now let's give the stream something
INSERT INTO FEED_T137 (SALE_ID, AMOUNT) VALUES (1, 10.00), (2, 20.00);| number of rows inserted |
|---|
| 2 |
SELECT SYSTEM$STREAM_HAS_DATA('FEEDSTRM_T137') AS THE_STREAM_HAS_DATA;| THE_STREAM_HAS_DATA |
|---|
| TRUE |
TRUE. Let's run the task again, and read the message it gives back carefully.
EXECUTE TASK GUARDED_T137;| status |
|---|
| Task GUARDED_T137 is scheduled to run immediately after the current run of the task tree is completed. |
A different sentence this time. scheduled to run immediately after the current run of the task tree is completed. The previous run is still finishing, and the message says so.
SELECT SALE_ID, AMOUNT FROM LANDED_T137 ORDER BY SALE_ID;| SALE_ID | AMOUNT |
|---|
SELECT SYSTEM$STREAM_HAS_DATA('FEEDSTRM_T137') AS THE_STREAM_HAS_DATA;| THE_STREAM_HAS_DATA |
|---|
| TRUE |
SELECT COUNT(*) AS ROWS_THE_STREAM_OFFERS FROM FEEDSTRM_T137;| ROWS_THE_STREAM_OFFERS |
|---|
| 2 |
Nothing landed yet, the function still says TRUE, and the stream is still offering its two rows. Which is the statement returning first, exactly as in the earlier section, and not the task failing to do its job. This demo checked too early again, and it cannot tell you any more than that.
And now the refusals
Let's change the cron task's schedule to an interval.
ALTER TASK CRONBEAT_T137 SET SCHEDULE = '5 MINUTE';| status |
|---|
| Statement executed successfully. |
A SET SCHEDULE replaces the old schedule outright. The cron string is not merged with anything, it is simply gone. Let's put a comment on it as well.
ALTER TASK CRONBEAT_T137 SET COMMENT = 'changed by episode 137';| status |
|---|
| Statement executed successfully. |
And now take the schedule away entirely.
ALTER TASK CRONBEAT_T137 UNSET SCHEDULE;| status |
|---|
| Statement executed successfully. |
Accepted. The task is still there, with its comment, and it has no schedule. So what happens if we try to resume it?
ALTER TASK CRONBEAT_T137 RESUME;091453 (22000): Task DEMO_DB.PUBLIC.CRONBEAT_T137 should have a SCHEDULE, AFTER, FINALIZE or WHEN to be resumed.
091453, should have a SCHEDULE, AFTER, FINALIZE or WHEN to be resumed. Which is a genuinely good error message, because it names all four ways a task can be told when to run.
A warehouse that does not exist.
CREATE OR REPLACE TASK NOWH_T137
WAREHOUSE = NO_SUCH_WAREHOUSE_T137
SCHEDULE = '1 MINUTE'
AS
INSERT INTO HEARTBEAT_T137 (NOTE) VALUES ('no warehouse');091083 (42601): Nonexistent warehouse NO_SUCH_WAREHOUSE_T137 was specified.
091083, at create time rather than at run time. Now a task with no schedule and no parent, created that way from the start.
CREATE OR REPLACE TASK ORPHAN_T137
WAREHOUSE = COMPUTE_WH
AS
INSERT INTO HEARTBEAT_T137 (NOTE) VALUES ('orphan');| status |
|---|
| Task ORPHAN_T137 successfully created. |
Accepted. So a task with no trigger of any kind is a legal object. You just cannot resume it, which we have already seen. That is the third way a task can look fine and never run.
And three questions about a task that is not there.
EXECUTE TASK NOSUCHTASK_T137;002003 (02000): SQL compilation error: Task 'DEMO_DB.PUBLIC.NOSUCHTASK_T137' does not exist or not authorized. Your primary role ACCOUNTADMIN or one of your secondary roles must have at least one privilege granted on TASK DEMO_DB.PUBLIC.NOSUCHTASK_T137.
DESCRIBE TASK NOSUCHTASK_T137;002003 (02000): SQL compilation error: Task 'DEMO_DB.PUBLIC.NOSUCHTASK_T137' does not exist or not authorized. Your primary role ACCOUNTADMIN or one of your secondary roles must have at least one privilege granted on TASK DEMO_DB.PUBLIC.NOSUCHTASK_T137.
002003 both times. But the history function:
SELECT COUNT(*) AS ROWS_IN_THE_HISTORY
FROM TABLE(INFORMATION_SCHEMA.TASK_HISTORY(
TASK_NAME => 'NOSUCHTASK_T137'));| ROWS_IN_THE_HISTORY |
|---|
| 0 |
Zero rows, and no error at all. A history function will not tell you the task is missing. If you are monitoring a task by its history and somebody drops it, your monitoring query keeps working and keeps returning nothing.
And the cleanup, which matters more here than usual
A task left running keeps spending money and keeps writing rows after you have stopped paying attention. So every one of them is suspended before it is dropped.
ALTER TASK IF EXISTS CRONBEAT_T137 SUSPEND;| status |
|---|
| Statement executed successfully. |
ALTER TASK IF EXISTS BEAT_T137 SUSPEND;| status |
|---|
| Statement executed successfully. |
ALTER TASK IF EXISTS GUARDED_T137 SUSPEND;| status |
|---|
| Statement executed successfully. |
ALTER TASK IF EXISTS SERVERLESS_T137 SUSPEND;| status |
|---|
| Statement executed successfully. |
Then the drops, and then the check that matters.
SHOW TASKS LIKE '%\\_T137';| created_on | name | id | database_name | schema_name | owner | comment | warehouse | schedule | predecessors | state | definition | condition | allow_overlapping_execution | error_integration | last_committed_on | last_suspended_on | owner_role_type | config | task_relations | last_suspended_reason | success_integration | scheduling_mode | target_completion_interval | execute_as_user | overlap_policy | created_by_user |
|---|
An empty listing. This one is safe to show precisely because it has no rows in it, so there is nothing in it to identify anybody. Nothing of this episode's is left running on the account.
SELECT CURRENT_DATABASE() AS THE_DATABASE,
CURRENT_SCHEMA() AS THE_SCHEMA;| THE_DATABASE | THE_SCHEMA |
|---|---|
| DEMO_DB | PUBLIC |
SELECT COUNT(*) AS PROBE_TABLES_LEFT
FROM INFORMATION_SCHEMA.TABLES
WHERE TABLE_SCHEMA = 'PUBLIC'
AND ENDSWITH(TABLE_NAME, '_T137');| PROBE_TABLES_LEFT |
|---|
| 0 |
Key points
- A task is created suspended. It exists, and it has done nothing.
DESCRIBE TASKreturns the same twenty seven columnsSHOW TASKSdoes. It is not a column list.EXECUTE TASKfires a suspended task without resuming it.EXECUTE TASKreturns when the run is queued, not when it finishes, so the next statement you write is a race.- The target table is the honest witness.
0rows before the resume and2after it: one from the hand run and one from the schedule. TASK_HISTORYreturns recent runs for a task name, so its grids can hold runs from before today's.- The shortest schedule is ten seconds, not one minute.
'1 SECOND'and'9 SECOND'both give091851. - A cron string is stored in the catalog exactly as you typed it.
- Leaving the
WAREHOUSEclause out gives you a serverless task, and it runs. - A
WHENcondition is a real guard. A false one lands in the history asSKIPPEDwith0040003. UNSET SCHEDULEthenRESUMEgives091453, which names all four triggers a task can have.TASK_HISTORYfor a task that does not exist returns zero rows and no error.
Episode 138 — task DAGs. One scheduled task with dependents: why only the root may have a schedule, what order the tree runs in, and what Snowflake lets you resume.
Every query and every result grid on this page was captured from a live Snowflake account (version 10.35.101) running the demo script for this episode. The SHOW TASKS and DESCRIBE TASK grids are deliberately left out, because a task listing includes a created_by_user column holding the account's own user name, and so is the SELECT * from TASK_HISTORY, whose rows carry a query id and the user that scheduled the run. Their statements are shown so you can run them yourself. Nothing from those grids has been retyped into the text. Every count in the text is read off one of the grids on this page. The two SYSTEM$WAIT calls are waits, not measurements: nothing is timed and no performance claims are made anywhere in this series. Behaviour follows the Snowflake CREATE TASK documentation.



No comments:
Post a Comment
Note: Only a member of this blog may post a comment.