Snowflake SQL Tutorial 127 - PARTIALLY_LOADED Is A Successful Load With Rows Missing

Snowflake SQL · Episode 127

In this post we are going to cover ON_ERROR in Snowflake, which is the option that decides what COPY INTO does when the data in the file is wrong.

The default is that nothing loads. One bad row takes the whole statement down, and that includes the good files sitting in the same statement. Every other setting is a decision about how much bad data you will accept quietly.

We will start with the default and work through the settings one at a time, and along the way we will get the rejected rows back out verbatim. Every query below was run on a real Snowflake account, and every result you see is the output it gave back.

Episode 127 of the Snowflake SQL: Zero to Hero series on TechBrothersIT.

Let's set up two files, one clean and one broken

We need a stage, a format, a table, and two files. One of the files is clean. The other has two problems in it, on purpose, and they are two different kinds of problem.

Sample setup
-- Sample setup for this post.
USE DATABASE DEMO_DB;
USE SCHEMA PUBLIC;

CREATE OR REPLACE STAGE STAGE_T127;

CREATE OR REPLACE FILE FORMAT CSV_T127
    TYPE = CSV
    SKIP_HEADER = 1
    FIELD_OPTIONALLY_ENCLOSED_BY = '"';

CREATE OR REPLACE TABLE TARGET_T127 (
    SALE_ID INT,
    CITY    VARCHAR(20),
    AMOUNT  NUMBER(10,2)
);

-- One clean file and one with two problems in it, both with a header row.
-- Point these at a folder of your own.
PUT 'file://C:/your/folder/t127_good.csv' @STAGE_T127/in/ AUTO_COMPRESS = FALSE;
PUT 'file://C:/your/folder/t127_bad.csv'  @STAGE_T127/in/ AUTO_COMPRESS = FALSE;

-- A second format whose field delimiter never occurs in the file,
-- so we can read the raw lines instead of a parsed grid.
CREATE OR REPLACE FILE FORMAT RAW_T127
    TYPE = CSV
    FIELD_DELIMITER = '~';

Let's check both files are on the stage.

Query
LIST @STAGE_T127/in/;
Result2 rows · 4 columns
namesizemd5last_modified
stage_t127/in/t127_bad.csv112df5e99ab8c7c6d82635674365ed90fd0Thu, 1 Oct 2026 14:22:00 GMT
stage_t127/in/t127_good.csv80b2482de277162b89324a3a08e965652fThu, 1 Oct 2026 14:21:59 GMT

Both there. The broken one is the bigger of the two, because it has an extra field on one of its lines.

Let's look at the broken file line by line

To see what is wrong with a file you have to see the raw lines, not a parsed grid. A parsed grid has already made the decision we are arguing about.

So we read it through RAW_T127, the format whose field delimiter is a tilde. A tilde never occurs in this file, so every line comes back whole in $1.

Query
SELECT METADATA$FILE_ROW_NUMBER AS LINE_NUMBER, $1 AS THE_WHOLE_LINE
FROM @STAGE_T127/in/ (FILE_FORMAT => 'RAW_T127', PATTERN => '.*t127_bad.*')
ORDER BY LINE_NUMBER;
Result5 rows · 2 columns
LINE_NUMBERTHE_WHOLE_LINE
1sale_id,city,amount
24,London,310.00
35,Austin,not_a_number
46,Dublin,55.00,extra_column
57,Oslo,88.25

Now we can see it. The header line, then London, then 5,Austin,not_a_number, where the amount is a word. Then 6,Dublin,55.00,extra_column, which has a fourth field. Then Oslo.

Two different kinds of wrong. One is a value that will not convert, and one is a line with too many fields. Remember that, because Snowflake tells them apart for us later.

Let's run the simplest COPY INTO across both files

No ON_ERROR anywhere. Just a load, over the whole prefix, so both files are in the one statement.

Query
COPY INTO TARGET_T127
FROM @STAGE_T127/in/
FILE_FORMAT = (FORMAT_NAME = 'CSV_T127');
Query failed — Snowflake returned an error
100016 (22000): Field delimiter ',' found while expecting record delimiter '\n'
  File 'in/t127_bad.csv', line 4, character 15
  Row 3, column "TARGET_T127"["AMOUNT":3]
  If you would like to continue loading when an error is encountered, use other values such as 'SKIP_FILE' or 'CONTINUE' for the ON_ERROR option. For more information on loading options, please run 'info loading_data' in a SQL client.

Field delimiter ',' found while expecting record delimiter, at line four of the bad file. That is the line with the extra field on it.

Now the part that matters. Let's count the table.

Query
SELECT COUNT(*) AS ROWS_IN_THE_TABLE FROM TARGET_T127;
Result1 row · 1 column
ROWS_IN_THE_TABLE
0

Zero. Not zero from the bad file. Zero altogether. The clean file was in that statement too and none of it went in.

So the default is ABORT_STATEMENT, and it means the statement, not the file.

Is ABORT_STATEMENT really the default?

Rather than tell you it is, let's spell it out and compare what comes back.

Query
COPY INTO TARGET_T127
FROM @STAGE_T127/in/
FILE_FORMAT = (FORMAT_NAME = 'CSV_T127')
ON_ERROR = ABORT_STATEMENT;
Query failed — Snowflake returned an error
100016 (22000): Field delimiter ',' found while expecting record delimiter '\n'
  File 'in/t127_bad.csv', line 4, character 15
  Row 3, column "TARGET_T127"["AMOUNT":3]
  If you would like to continue loading when an error is encountered, use other values such as 'SKIP_FILE' or 'CONTINUE' for the ON_ERROR option. For more information on loading options, please run 'info loading_data' in a SQL client.

The identical message, line and character included. That is the proof.

Now let's try ON_ERROR = CONTINUE

CONTINUE says: skip the rows you cannot read and load the rest.

Query
COPY INTO TARGET_T127
FROM @STAGE_T127/in/
FILE_FORMAT = (FORMAT_NAME = 'CSV_T127')
ON_ERROR = CONTINUE
FORCE = TRUE;
Result2 rows · 10 columns · scroll →
filestatusrows_parsedrows_loadederror_limiterrors_seenfirst_errorfirst_error_linefirst_error_characterfirst_error_column_name
stage_t127/in/t127_good.csvLOADED3330NULLNULLNULLNULL
stage_t127/in/t127_bad.csvPARTIALLY_LOADED4242Numeric value 'not_a_number' is not recognized310"TARGET_T127"["AMOUNT":3]

And it succeeded. Two rows in the report, and one of them has a status we have not seen before.

The clean file is LOADED. Three parsed, three loaded, no errors. The bad one is PARTIALLY_LOADED. Four parsed, two loaded, two errors seen, and the first of those errors is spelled out in the grid.

That grid is wide, so let's pull the columns that matter out of it with a RESULT_SCAN.

Query
SELECT "file"          AS THE_FILE,
       "status"        AS THE_STATUS,
       "rows_parsed"   AS ROWS_PARSED,
       "rows_loaded"   AS ROWS_LOADED,
       "errors_seen"   AS ERRORS_SEEN,
       "first_error"   AS FIRST_ERROR
FROM TABLE(RESULT_SCAN(LAST_QUERY_ID()))
ORDER BY THE_FILE;
Result2 rows · 6 columns · scroll →
THE_FILETHE_STATUSROWS_PARSEDROWS_LOADEDERRORS_SEENFIRST_ERROR
stage_t127/in/t127_bad.csvPARTIALLY_LOADED422Numeric value 'not_a_number' is not recognized
stage_t127/in/t127_good.csvLOADED330NULL

File, status, rows parsed, rows loaded, errors seen and the first error. Those six are the ones to read on any load.

error_limit is not always a number you chose

Have a look at error_limit in the wide grid above. We did not set a limit anywhere in that statement, and the two files came back with different values for it. Each one is that file's own rows_parsed. error_limit only reports the number you asked for when you actually asked for one, which we do further down.

So what actually survived?

This is the question the report does not answer. Let's look at the table.

Query
SELECT SALE_ID AS SALE_ID, CITY AS CITY, AMOUNT AS AMOUNT
FROM TARGET_T127
ORDER BY SALE_ID;
Result5 rows · 3 columns
SALE_IDCITYAMOUNT
1Toronto100.00
2Boston250.50
3Zurich75.25
4London310.00
7Oslo88.25

Five rows. The three from the clean file, and then London and Oslo out of the bad one. Austin and Dublin are not there.

Query
SELECT COUNT(*)    AS ROWS_IN_THE_TABLE,
       SUM(AMOUNT) AS TOTAL_AMOUNT
FROM TARGET_T127;
Result1 row · 2 columns
ROWS_IN_THE_TABLETOTAL_AMOUNT
5824.00

Five rows and 824.00.

This is the shape of the problem

Nothing warned us. The statement reported success, the row count looks perfectly plausible, and two rows of real data are gone. If this load ran on a schedule and you only looked at whether it succeeded, you would never know.

Which is why the next grid is the one that matters.

Can we get the rejected rows back?

Yes. The rows that did not make it are not lost. VALIDATE hands them back, and it takes a job id, where '_last' means the most recent load into that table.

Query
SELECT * FROM TABLE(VALIDATE(TARGET_T127, JOB_ID => '_last'));
Result2 rows · 12 columns · scroll →
ERRORFILELINECHARACTERBYTE_OFFSETCATEGORYCODESQL_STATECOLUMN_NAMEROW_NUMBERROW_START_LINEREJECTED_RECORD
Field delimiter ',' found while expecting record delimiter '\n'in/t127_bad.csv41575parsing10001622000"TARGET_T127"["AMOUNT":3]346,Dublin,55.00,extra_column
Numeric value 'not_a_number' is not recognizedin/t127_bad.csv31047conversion10003822018"TARGET_T127"["AMOUNT":3]235,Austin,not_a_number

Twelve columns. Two rows, one for each bad line.

REJECTED_RECORD is the one you want. It holds the raw line from the file, exactly as it was. 6,Dublin,55.00,extra_column and 5,Austin,not_a_number, whole. So you can fix them, load them somewhere else, or just look at them.

And CATEGORY tells the two kinds of wrong apart. parsing for the line with the extra field, conversion for the word where a number should be. With a different CODE for each: 100016 and 100038.

Now let's skip the whole file instead

CONTINUE made the decision row by row. SKIP_FILE makes it file by file. Let's start from an empty table.

Query
TRUNCATE TABLE TARGET_T127;
Result1 row · 1 column
status
Statement executed successfully.
Query
COPY INTO TARGET_T127
FROM @STAGE_T127/in/
FILE_FORMAT = (FORMAT_NAME = 'CSV_T127')
ON_ERROR = SKIP_FILE
FORCE = TRUE;
Result2 rows · 10 columns · scroll →
filestatusrows_parsedrows_loadederror_limiterrors_seenfirst_errorfirst_error_linefirst_error_characterfirst_error_column_name
stage_t127/in/t127_good.csvLOADED3310NULLNULLNULLNULL
stage_t127/in/t127_bad.csvLOAD_FAILED4012Numeric value 'not_a_number' is not recognized310"TARGET_T127"["AMOUNT":3]

A third status. The clean file is LOADED. The bad one is LOAD_FAILED, with four parsed, nothing loaded and two errors.

Query
SELECT SALE_ID AS SALE_ID, CITY AS CITY, AMOUNT AS AMOUNT
FROM TARGET_T127
ORDER BY SALE_ID;
Result3 rows · 3 columns
SALE_IDCITYAMOUNT
1Toronto100.00
2Boston250.50
3Zurich75.25

Three rows, all from the clean file. So SKIP_FILE does what it says, and the good file still gets through. That is the difference from the default.

Let's put a number on how many errors we will take

SKIP_FILE_n says: fail the file once it has this many errors in it. The bad file has two. Let's allow three.

Query
TRUNCATE TABLE TARGET_T127;
Result1 row · 1 column
status
Statement executed successfully.
Query
COPY INTO TARGET_T127
FROM @STAGE_T127/in/
FILE_FORMAT = (FORMAT_NAME = 'CSV_T127')
ON_ERROR = SKIP_FILE_3
FORCE = TRUE;
Result2 rows · 10 columns · scroll →
filestatusrows_parsedrows_loadederror_limiterrors_seenfirst_errorfirst_error_linefirst_error_characterfirst_error_column_name
stage_t127/in/t127_good.csvLOADED3330NULLNULLNULLNULL
stage_t127/in/t127_bad.csvPARTIALLY_LOADED4232Numeric value 'not_a_number' is not recognized310"TARGET_T127"["AMOUNT":3]

PARTIALLY_LOADED, and look at error_limit now. It is the number we asked for.

Query
SELECT COUNT(*) AS ROWS_IN_THE_TABLE FROM TARGET_T127;
Result1 row · 1 column
ROWS_IN_THE_TABLE
5

Five rows, the same as CONTINUE gave us.

Now a limit the file cannot meet

Query
TRUNCATE TABLE TARGET_T127;
Result1 row · 1 column
status
Statement executed successfully.
Query
COPY INTO TARGET_T127
FROM @STAGE_T127/in/
FILE_FORMAT = (FORMAT_NAME = 'CSV_T127')
ON_ERROR = SKIP_FILE_1
FORCE = TRUE;
Result2 rows · 10 columns · scroll →
filestatusrows_parsedrows_loadederror_limiterrors_seenfirst_errorfirst_error_linefirst_error_characterfirst_error_column_name
stage_t127/in/t127_good.csvLOADED3310NULLNULLNULLNULL
stage_t127/in/t127_bad.csvLOAD_FAILED4012Numeric value 'not_a_number' is not recognized310"TARGET_T127"["AMOUNT":3]

LOAD_FAILED for the bad file, with the limit reported as one and errors seen as two.

Query
SELECT COUNT(*) AS ROWS_IN_THE_TABLE FROM TARGET_T127;
Result1 row · 1 column
ROWS_IN_THE_TABLE
3

Three rows. Only the clean file again.

And the percentage version

SKIP_FILE_n% works out the limit from the rows in the file. Our bad file parses four rows, so half of it is two, and the file has two errors. Right on the line.

Query
COPY INTO TARGET_T127
FROM @STAGE_T127/in/
FILE_FORMAT = (FORMAT_NAME = 'CSV_T127')
ON_ERROR = 'SKIP_FILE_50%'
FORCE = TRUE;
Result2 rows · 10 columns · scroll →
filestatusrows_parsedrows_loadederror_limiterrors_seenfirst_errorfirst_error_linefirst_error_characterfirst_error_column_name
stage_t127/in/t127_good.csvLOADED3310NULLNULLNULLNULL
stage_t127/in/t127_bad.csvLOAD_FAILED4022Numeric value 'not_a_number' is not recognized310"TARGET_T127"["AMOUNT":3]

error_limit two, errors_seen two, and LOAD_FAILED.

Reaches, not passes

So the file fails when errors_seen reaches error_limit, not when it goes past it. If you want to tolerate two bad rows, you have to allow three.

Can we ask what would go wrong without loading anything?

VALIDATION_MODE is the option for that. RETURN_ERRORS says: tell me what is wrong and load nothing.

Query
TRUNCATE TABLE TARGET_T127;
Result1 row · 1 column
status
Statement executed successfully.
Query
COPY INTO TARGET_T127
FROM @STAGE_T127/in/
FILE_FORMAT = (FORMAT_NAME = 'CSV_T127')
VALIDATION_MODE = RETURN_ERRORS;
Result2 rows · 12 columns · scroll →
ERRORFILELINECHARACTERBYTE_OFFSETCATEGORYCODESQL_STATECOLUMN_NAMEROW_NUMBERROW_START_LINEREJECTED_RECORD
Field delimiter ',' found while expecting record delimiter '\n'in/t127_bad.csv41575parsing10001622000"TARGET_T127"["AMOUNT":3]346,Dublin,55.00,extra_column
Numeric value 'not_a_number' is not recognizedin/t127_bad.csv31047conversion10003822018"TARGET_T127"["AMOUNT":3]235,Austin,not_a_number

The same twelve column grid VALIDATE gave us, with the same two rows, the same categories and the same rejected records.

But this time nothing was loaded, and that is the claim worth checking rather than believing.

Query
SELECT COUNT(*) AS ROWS_IN_THE_TABLE FROM TARGET_T127;
Result1 row · 1 column
ROWS_IN_THE_TABLE
0

Zero rows. So it really is a dry run.

RETURN_ALL_ERRORS is the same idea without a cap on how many it reports.

Query
COPY INTO TARGET_T127
FROM @STAGE_T127/in/
FILE_FORMAT = (FORMAT_NAME = 'CSV_T127')
VALIDATION_MODE = RETURN_ALL_ERRORS;
Result2 rows · 12 columns · scroll →
ERRORFILELINECHARACTERBYTE_OFFSETCATEGORYCODESQL_STATECOLUMN_NAMEROW_NUMBERROW_START_LINEREJECTED_RECORD
Field delimiter ',' found while expecting record delimiter '\n'in/t127_bad.csv41575parsing10001622000"TARGET_T127"["AMOUNT":3]346,Dublin,55.00,extra_column
Numeric value 'not_a_number' is not recognizedin/t127_bad.csv31047conversion10003822018"TARGET_T127"["AMOUNT":3]235,Austin,not_a_number

The same two rows, because two is all there were in this file.

What happens when we ask for a few rows instead?

There is a third form, RETURN_n_ROWS, and I assumed it would show me the rows it was about to load. Let's ask for five of them.

Query
COPY INTO TARGET_T127
FROM @STAGE_T127/in/
FILE_FORMAT = (FORMAT_NAME = 'CSV_T127')
VALIDATION_MODE = RETURN_5_ROWS;
Query failed — Snowflake returned an error
100016 (22000): Field delimiter ',' found while expecting record delimiter '\n'
  File 'in/t127_bad.csv', line 4, character 15
  Row 3, column "TARGET_T127"["AMOUNT":3]

It did not show me anything. 100016, the same parsing error the default load gave us, and it aborted.

So RETURN_n_ROWS is not a safe dry run. It reads the file the way a real load does and it hits the bad line like any other load would. I got that one wrong, and it is on the page as it came back.

And VALIDATION_MODE on a transforming COPY is not supported at all.

Query
COPY INTO TARGET_T127 (SALE_ID, CITY, AMOUNT)
FROM (
    SELECT $1::INT, UPPER($2::VARCHAR), $3::NUMBER(10,2)
    FROM @STAGE_T127/in/ (FILE_FORMAT => 'CSV_T127')
)
VALIDATION_MODE = RETURN_ERRORS;
Query failed — Snowflake returned an error
002302 (0A000): SQL Compilation error: Expression 'SALE_ID' not supported within a VALIDATE expression.

Expression 'SALE_ID' not supported within a VALIDATE expression. So the dry run and the transforming load are two things you cannot have at once.

One last thing: both option values are checked by name

A misspelled option value could easily be ignored in silence, which would be the worst possible behaviour here. Let's check both of them.

Query
COPY INTO TARGET_T127
FROM @STAGE_T127/in/
FILE_FORMAT = (FORMAT_NAME = 'CSV_T127')
ON_ERROR = KEEP_CALM_AND_CARRY_ON
FORCE = TRUE;
Query failed — Snowflake returned an error
001008 (22023): SQL compilation error:
invalid value [KEEP_CALM_AND_CARRY_ON] for parameter 'ON_ERROR'
Query
COPY INTO TARGET_T127
FROM @STAGE_T127/in/
FILE_FORMAT = (FORMAT_NAME = 'CSV_T127')
VALIDATION_MODE = RETURN_EVERYTHING;
Query failed — Snowflake returned an error
001008 (22023): SQL compilation error:
invalid value [RETURN_EVERYTHING] for parameter 'VALIDATION_MODE'

Both refused by name, both 001008. Good.

And VALIDATE on a table that never had a load into it.

Query
SELECT * FROM TABLE(VALIDATE(CUSTOMERS, JOB_ID => '_last'));
Query failed — Snowflake returned an error
002018 (22023): SQL compilation error:
Invalid argument [The specified Job UUID does not load into the specified table] for table function. Table function argument is required to be a constant.

The specified Job UUID does not load into the specified table. So '_last' is the last load into that table, and there has not been one.

And let's put everything back

Everything here was made by the script at the top, so everything in it can go.

Cleanup
-- Put the schema back the way we found it.
TRUNCATE TABLE TARGET_T127;
REMOVE @STAGE_T127/;

DROP STAGE       IF EXISTS STAGE_T127;
DROP FILE FORMAT IF EXISTS RAW_T127;
DROP FILE FORMAT IF EXISTS CSV_T127;
DROP TABLE       IF EXISTS TARGET_T127;
Query
LIST @STAGE_T127/;
Result0 rows · 4 columns
namesizemd5last_modified
Query
SELECT CURRENT_DATABASE()                 AS THE_DATABASE,
       CURRENT_SCHEMA()                   AS THE_SCHEMA,
       (SELECT COUNT(*) FROM INFORMATION_SCHEMA.TABLES
        WHERE RIGHT(TABLE_NAME, 5) = '_T127') AS PROBE_TABLES_LEFT,
       (SELECT COUNT(*) FROM INFORMATION_SCHEMA.STAGES
        WHERE STAGE_NAME LIKE '%_T127%')      AS PROBE_STAGES_LEFT,
       (SELECT COUNT(*) FROM INFORMATION_SCHEMA.FILE_FORMATS
        WHERE RIGHT(FILE_FORMAT_NAME, 5) = '_T127') AS PROBE_FORMATS_LEFT,
       (SELECT COUNT(*) FROM CUSTOMERS)   AS DEMO_DB_IS_READABLE;
Result1 row · 6 columns · scroll →
THE_DATABASETHE_SCHEMAPROBE_TABLES_LEFTPROBE_STAGES_LEFTPROBE_FORMATS_LEFTDEMO_DB_IS_READABLE
DEMO_DBPUBLIC00010

No probe tables, no probe stages, no probe formats, and the database still reads.

Key points

  • The default is ON_ERROR = ABORT_STATEMENT, and it means the statement. One bad row in one file left the clean file in the same statement with zero rows loaded.
  • Read a file's raw lines by giving the format a FIELD_DELIMITER that never occurs in it.
  • A value that will not convert is a conversion error. A line with too many fields is a parsing error.
  • ON_ERROR = CONTINUE gives a third status, PARTIALLY_LOADED, and the statement reports success.
  • On this load that meant five rows and 824.00 in the table, with two rows of real data silently missing.
  • VALIDATE(table, JOB_ID => '_last') returns twelve columns, including REJECTED_RECORD, which is the raw line exactly as it was in the file.
  • CATEGORY and CODE tell the two kinds of bad row apart: 100016 and 100038.
  • SKIP_FILE fails the file and lets the clean one through.
  • SKIP_FILE_n and SKIP_FILE_n% set error_limit, and the file fails when errors_seen reaches it, not when it passes it.
  • VALIDATION_MODE = RETURN_ERRORS really does load nothing. We checked the table afterwards.
  • RETURN_n_ROWS is not a safe dry run. It hit the bad line and aborted with 100016.
  • VALIDATION_MODE is not supported on a transforming COPY, and both option values are checked by name.
Next in this series

Episode 128 — COPY INTO a location. Unloading, which is the same statement pointed the other way. We will look at the prefix it writes into, the file names it chooses for you, and the compression nobody asked for.

Every query and every result grid on this page was captured from a live Snowflake account (version 10.35.101) running the demo script for this episode. The sales data is made up for teaching, and the two broken lines in t127_bad.csv are broken deliberately. The two PUT statements take a path on your own machine, so the folder is written as C:/your/folder and their captured grids are not shown. Cells reading NULL in the COPY report are real nulls. Nothing on this page is timed and no performance claims are made anywhere in this series. The option behaviour follows the Snowflake COPY INTO <table> documentation.

No comments:

Post a Comment

Note: Only a member of this blog may post a comment.