{"id":1179,"date":"2012-08-13T12:10:00","date_gmt":"2012-08-13T12:10:00","guid":{"rendered":"http:\/\/bicortex.com\/?p=1179"},"modified":"2012-08-15T08:25:28","modified_gmt":"2012-08-15T08:25:28","slug":"implementing-ssis-package-checkpoint-the-power-of-restartibility","status":"publish","type":"post","link":"https:\/\/bicortex.com\/bicortex\/implementing-ssis-package-checkpoint-the-power-of-restartibility\/","title":{"rendered":"How to implement SSIS Package Checkpoints &#8211; The Power of Restartability"},"content":{"rendered":"<p style=\"text-align: justify;\">Data transfers can take a substantial amount of time and when the package fails during execution it would be counter-productive at best and dangerous at worst to restart the whole package and start from the beginning. This can be prevented by means of using checkpoints which store the execution log inside a file to enable rerunning the routine without repeating the already executed steps. To see how that works let\u2019s create a simple data source and data destination table by executing the following script.<\/p>\n<pre class=\"brush: sql; title: ; notranslate\" title=\"\">\r\n--CREATE SOURCE TABLE\r\nIF EXISTS (SELECT * FROM sys.objects WHERE object_id = OBJECT_ID(N'Tbl_Source'))\r\nDROP TABLE Tbl_Source\r\nCREATE TABLE Tbl_Source\r\n(\r\nID int NULL,\r\nFirst_Name varchar (50) NULL,\r\nLast_Name varchar (50) NULL,\r\nAge int NULL,\r\nAge_Category varchar (50) NULL\r\n)\r\nON &#x5B;PRIMARY]\r\n\r\n--CREATE DESTINATION TABLE\r\nIF EXISTS (SELECT * FROM sys.objects WHERE object_id = OBJECT_ID(N'Tbl_Destination'))\r\nDROP TABLE Tbl_Destination\r\nCREATE TABLE Tbl_Destination\r\n(\r\nID INT NULL,\r\nFirst_Name varchar (50) NULL,\r\nLast_Name varchar (50) NULL,\r\nAge int NULL,\r\nAge_Category varchar (50) NULL\r\n)\r\nON &#x5B;PRIMARY]\r\n\r\n--POPULATE SOURCE TABLE\r\nINSERT INTO Tbl_Source\r\n(ID, First_Name, Last_Name, Age, Age_Category)\r\nSELECT 1, 'John', 'Smith', 44, NULL\r\nUNION ALL\r\nSELECT 2, 'Chris', 'Martin',23, NULL\r\nUNION ALL\r\nSELECT 3, 'Jerry', 'March',59, NULL\r\nUNION ALL\r\nSELECT 4, 'Kate', 'Johnson',20, NULL\r\nUNION ALL\r\nSELECT 5, 'Mary', 'McKinnon',37, NULL\r\n<\/pre>\n<\/p>\n<p style=\"text-align: justify;\">This should create a rudimentary source table which will contain 6 records and a destination table which we will be loading the source data into. Next, let\u2019s create a simple SSIS package. When you finished lying out and joining all the components together you should be able to see a structure which resembles the image below.<\/p>\n<p style=\"text-align: justify;\"><a href=\"http:\/\/bicortex.com\/bicortex\/wp-content\/post_content\/\/2012\/08\/SSIS_2012_Pckg_CheckPoints_ControlFlow.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-1182\" title=\"SSIS_2012_Pckg_CheckPoints_ControlFlow\" src=\"http:\/\/bicortex.com\/bicortex\/wp-content\/post_content\/\/2012\/08\/SSIS_2012_Pckg_CheckPoints_ControlFlow.png\" alt=\"\" width=\"580\" height=\"109\" srcset=\"https:\/\/bicortex.com\/bicortex\/wp-content\/post_content\/2012\/08\/SSIS_2012_Pckg_CheckPoints_ControlFlow.png 580w, https:\/\/bicortex.com\/bicortex\/wp-content\/post_content\/2012\/08\/SSIS_2012_Pckg_CheckPoints_ControlFlow-300x56.png 300w\" sizes=\"auto, (max-width: 580px) 100vw, 580px\" \/><\/a><\/p>\n<p style=\"text-align: justify;\">To step through the individual components, first, place a \u2018Date Flow Task\u2019 on the Control Flow and going into \u2018Data Flow\u2019 tab place \u2018OLE DB Source\u2019 and \u2018OLE DB Destination\u2019, link them together and establish the mapping between the columns from Tbl_Source to Tbl_Destination. Go back to the \u2018Control Flow\u2019 and place another \u2018Execute SQL Task\u2019, call it \u2018SQL Task \u2013 Raise Error\u2019. Then create a connection to the database and write the following statement in the \u2018SQLStatement\u2019 property.<\/p>\n<pre class=\"brush: sql; title: ; notranslate\" title=\"\">\r\n--DUMMY SELECT TO THROW AN ERROR\r\nSELECT 1\/0\r\n<\/pre>\n<p style=\"text-align: justify;\">The purpose of this is for the package to fail and return an error on runtime. Once more, add \u2018Execute SQL Task\u2019, call it \u2018SQL Task \u2013 Update Age Category\u2019 and input the following query which will be responsible for updates to the \u2018Age Category\u2019 column in the destination table.<\/p>\n<pre class=\"brush: sql; title: ; notranslate\" title=\"\">\r\n--UPDATE tbl_Destination WITH Age_Category\r\nUPDATE Tbl_Destination\r\nSET Age_Category = CASE WHEN Age &lt; 18 THEN 'Juvenile'                         \r\n                        WHEN Age &gt;=18 AND Age &lt; 50 THEN 'Middle Age'                         \r\n                        WHEN Age &gt; 50 THEN 'Senior' END\r\n<\/pre>\n<p style=\"text-align: justify;\">At this point all we have got is just a simple package which, when run, will fail on step two (division by zero). As the purpose of this exercise is to implement restartability, we need to enable package checkpoint functionality so the next time the package is run, it will start from the step which has failed and hasn\u2019t been executed, rather than from the very start. This will prevent the data from the destination table to be transferred twice for the exactly the same records and prevent duplication and issues with further processing and output. To enable checkpoints in our package, thus creating a point of failure where the package will restart from, click on an empty area in the \u2018Control Flow\u2019 and go to \u2018Properties\u2019 window. Next, in the \u2018Checkpoints\u2019 category set \u2018SaveCheckpoints\u2019 as \u2018True\u2019, \u2018CheckpointUsage\u2019 as \u2018IfExists\u2019 and set \u2018CheckpointFileName\u2019 to a file with any name, ending in .XML. For the purpose of this post, it is probably best to save it in your local c:\\ drive for convenience but in a production environment you will need to have a designated folder for this purpose. Finally, change the \u2018FailPackageOnFailure\u2019 property to \u2018True\u2019 on all \u2018Control Flow\u2019 tasks as per below.<\/p>\n<p><a href=\"http:\/\/bicortex.com\/bicortex\/wp-content\/post_content\/\/2012\/08\/SSIS_2012_Pckg_CheckPoints_Properties.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-1187\" title=\"SSIS_2012_Pckg_CheckPoints_Properties\" src=\"http:\/\/bicortex.com\/bicortex\/wp-content\/post_content\/\/2012\/08\/SSIS_2012_Pckg_CheckPoints_Properties.png\" alt=\"\" width=\"580\" height=\"152\" srcset=\"https:\/\/bicortex.com\/bicortex\/wp-content\/post_content\/2012\/08\/SSIS_2012_Pckg_CheckPoints_Properties.png 580w, https:\/\/bicortex.com\/bicortex\/wp-content\/post_content\/2012\/08\/SSIS_2012_Pckg_CheckPoints_Properties-300x78.png 300w\" sizes=\"auto, (max-width: 580px) 100vw, 580px\" \/><\/a><\/p>\n<p style=\"text-align: justify;\">It is time to run the package for the first time now and during its first run you should get an error half way through the execution.<\/p>\n<p><a href=\"http:\/\/bicortex.com\/bicortex\/wp-content\/post_content\/\/2012\/08\/SSIS_2012_Pckg_CheckPoints_ErrorRun.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-1189\" title=\"SSIS_2012_Pckg_CheckPoints_ErrorRun\" src=\"http:\/\/bicortex.com\/bicortex\/wp-content\/post_content\/\/2012\/08\/SSIS_2012_Pckg_CheckPoints_ErrorRun.png\" alt=\"\" width=\"580\" height=\"105\" srcset=\"https:\/\/bicortex.com\/bicortex\/wp-content\/post_content\/2012\/08\/SSIS_2012_Pckg_CheckPoints_ErrorRun.png 580w, https:\/\/bicortex.com\/bicortex\/wp-content\/post_content\/2012\/08\/SSIS_2012_Pckg_CheckPoints_ErrorRun-300x54.png 300w\" sizes=\"auto, (max-width: 580px) 100vw, 580px\" \/><\/a><\/p>\n<p style=\"text-align: justify;\">This is expected as our SQL statement \u2018SELECT 1\/0\u2019 should throw an exception. Also, our destination table should be incomplete as the UPDATE Statement has not been yet executed and all the \u2018Age_Category\u2019 fields should be unpopulated (NULL values). Let\u2019s change this SQL code in &#8216;SQL Task &#8211; Raise Error&#8217; component into the following statement in order to rectify this error.<\/p>\n<pre class=\"brush: sql; title: ; notranslate\" title=\"\">\r\n--CORRECT SELECT STATEMENT TO AVOID ERROR\r\nSELECT 1\/1\r\n<\/pre>\n<p style=\"text-align: justify;\">Now, let\u2019s run the package again. This time around you should notice that instead of running from the beginning, the package should start from step 2 i.e. \u2018SQL Task \u2013 Raise Error\u2019 and finish successfully.<\/p>\n<p style=\"text-align: justify;\"><a href=\"http:\/\/bicortex.com\/bicortex\/wp-content\/post_content\/\/2012\/08\/SSIS_2012_Pckg_CheckPoints_OKRun.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-1190\" title=\"SSIS_2012_Pckg_CheckPoints_OKRun\" src=\"http:\/\/bicortex.com\/bicortex\/wp-content\/post_content\/\/2012\/08\/SSIS_2012_Pckg_CheckPoints_OKRun.png\" alt=\"\" width=\"580\" height=\"105\" srcset=\"https:\/\/bicortex.com\/bicortex\/wp-content\/post_content\/2012\/08\/SSIS_2012_Pckg_CheckPoints_OKRun.png 580w, https:\/\/bicortex.com\/bicortex\/wp-content\/post_content\/2012\/08\/SSIS_2012_Pckg_CheckPoints_OKRun-300x54.png 300w\" sizes=\"auto, (max-width: 580px) 100vw, 580px\" \/><\/a><\/p>\n<p style=\"text-align: justify;\">Also, the previously unpopulated \u2018Age_Category\u2019 column in &#8216;Tbl_Destination&#8217; table should now be populated with the corresponding values from the UPDATE statement.<\/p>\n<p style=\"text-align: justify;\">The trick to this clever behavior lies in the XML file the SSIS process created in our selected location. This file stores the entry for each task executed by the package and in case of unsuccessful execution; it sets the re-start entry to the task which encountered an error. If the package runs without any issues, the file will be deleted automatically. Below is the content of the file opened in WordPad.<\/p>\n<p><a href=\"http:\/\/bicortex.com\/bicortex\/wp-content\/post_content\/\/2012\/08\/SSIS_2012_Pckg_CheckPoints_XML_View.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-1200\" title=\"SSIS_2012_Pckg_CheckPoints_XML_View\" src=\"http:\/\/bicortex.com\/bicortex\/wp-content\/post_content\/\/2012\/08\/SSIS_2012_Pckg_CheckPoints_XML_View.png\" alt=\"\" width=\"580\" height=\"110\" srcset=\"https:\/\/bicortex.com\/bicortex\/wp-content\/post_content\/2012\/08\/SSIS_2012_Pckg_CheckPoints_XML_View.png 580w, https:\/\/bicortex.com\/bicortex\/wp-content\/post_content\/2012\/08\/SSIS_2012_Pckg_CheckPoints_XML_View-300x56.png 300w\" sizes=\"auto, (max-width: 580px) 100vw, 580px\" \/><\/a><\/p>\n<p style=\"text-align: justify;\">Package checkpoints is a really handy way of cutting back on package execution times when an error is encountered, particularly in case of long running or complex builds.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Data transfers can take a substantial amount of time and when the package fails during execution it would be counter-productive at best and dangerous at worst to restart the whole package and start from the beginning. This can be prevented by means of using checkpoints which store the execution log inside a file to enable [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[32,5,50],"tags":[49,13],"class_list":["post-1179","post","type-post","status-publish","format-standard","hentry","category-how-tos","category-sql","category-ssis","tag-sql","tag-ssis"],"aioseo_notices":[],"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/bicortex.com\/bicortex\/wp-json\/wp\/v2\/posts\/1179","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/bicortex.com\/bicortex\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/bicortex.com\/bicortex\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/bicortex.com\/bicortex\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/bicortex.com\/bicortex\/wp-json\/wp\/v2\/comments?post=1179"}],"version-history":[{"count":22,"href":"https:\/\/bicortex.com\/bicortex\/wp-json\/wp\/v2\/posts\/1179\/revisions"}],"predecessor-version":[{"id":1207,"href":"https:\/\/bicortex.com\/bicortex\/wp-json\/wp\/v2\/posts\/1179\/revisions\/1207"}],"wp:attachment":[{"href":"https:\/\/bicortex.com\/bicortex\/wp-json\/wp\/v2\/media?parent=1179"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/bicortex.com\/bicortex\/wp-json\/wp\/v2\/categories?post=1179"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/bicortex.com\/bicortex\/wp-json\/wp\/v2\/tags?post=1179"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}