Dataload utility to load custom table

Posted by Unknown on

Often you may come across requirements to load custom data from an external data feed into a custom WCS table, OOB Dataload utility provides a Mediator called "TableObjectMediator" which can be used for such purpose, This technique does not require any coding and can be quickly setup by defining configuration files to map data feed with custom table.

This example also demonstrates loading of system generated and custom unique key from an external CSV file, any new records will be inserted and existing records (based on primary and unique) keys will be updated.

Step 1


Create a custom table, in this example my primary key is system generated and we will use IDResolver for this purpose, all other data will be fed from the input CSV file.

----- Define custom_table_derby.sql with following content -----

connect '..\db\mall;create=true'; 

CREATE TABLE XMYTable (
MY_SYSTEM_ID BIGINT NOT NULL,
MY_ID BIGINT NOT NULL,
firstName VARCHAR(254),
lastName VARCHAR(254)
);

ALTER TABLE XMYTable
       ADD PRIMARY KEY (MY_SYSTEM_ID);
       
CREATE UNIQUE INDEX I0000_XMYTable ON XMYTable
(
       MY_ID     ASC
);

INSERT INTO KEYS (KEYS_ID, TABLENAME, COLUMNNAME, COUNTER, PREFETCHSIZE, LOWERBOUND, UPPERBOUND, OPTCOUNTER)
VALUES ((SELECT MAX(KEYS_ID) + 1 FROM KEYS), 'xmytable', 'MY_SYSTEM_ID', 0000, 1, 1000, 999999, 1);

commit;

------------------------------------------

Step 2:

Run the SQL for Derby as follows.

C:\IBM\WCDE_ENT70\bin>ij.bat "..\samples\dataload\custom_table_derby.sql"

Step 3:


Define wc-dataload.xml file with following content

<?xml version="1.0" encoding="UTF-8"?>
<_config:DataLoadConfiguration
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.ibm.com/xmlns/prod/commerce/foundation/config ../../../../xml/config/xsd/wc-dataload.xsd"
    xmlns:_config="http://www.ibm.com/xmlns/prod/commerce/foundation/config">
  <_config:DataLoadEnvironment configFile="./wc-dataload-env.xml" />
  
  <_config:LoadOrder commitCount="100" batchSize="1" dataLoadMode="Replace">
         <!--_config:property name="firstTwoLinesAreHeader" value="true" /-->
         <_config:LoadItem name="XMYTable" businessObjectConfigFile="wc-loader-custom.xml" >
              <_config:DataSourceLocation location="MyData.csv" />
         </_config:LoadItem>
  </_config:LoadOrder> 

</_config:DataLoadConfiguration>

NOTE: For initial load we should replace dataLoadMode="Insert"

Step 4:


Define wc-loader-custom.xml with following content

<?xml version="1.0" encoding="UTF-8"?>
<_config:DataloadBusinessObjectConfiguration 
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.ibm.com/xmlns/prod/commerce/foundation/config ../../../../xml/config/xsd/wc-dataload-businessobject.xsd"
    xmlns:_config="http://www.ibm.com/xmlns/prod/commerce/foundation/config">
<_config:DataLoader className="com.ibm.commerce.foundation.dataload.BusinessObjectLoader" >
   <_config:DataReader className="com.ibm.commerce.foundation.dataload.datareader.CSVReader" firstLineIsHeader="true" useHeaderAsColumnName="true" >
                     <_config:Data>
                           <_config:column number="1" name="MY_ID" />
                           <_config:column number="2" name="firstname" />
                           <_config:column number="3" name="lastname" />
</_config:Data>
   </_config:DataReader>
   <_config:BusinessObjectBuilder className="com.ibm.commerce.foundation.dataload.businessobjectbuilder.TableObjectBuilder">
     <_config:Table name="XMYTable">
      <!-- System generated primary key-->
<_config:Column name="MY_SYSTEM_ID" value="MY_SYSTEM_ID" valueFrom="IDResolve">
          <_config:IDResolve tableName="XMYTable" generateNewKey="true" />
          </_config:Column>
<!-- Unique key fed from the CSV file-->
<_config:Column name="MY_ID" value="MY_ID">
</_config:Column>
<_config:Column name="firstName" value="firstname">
</_config:Column>
<_config:Column name="lastName" value="lastname">
</_config:Column>
     </_config:Table>
<_config:BusinessObjectMediator
className="com.ibm.commerce.foundation.dataload.businessobjectmediator.TableObjectMediator">
</_config:BusinessObjectMediator>
</_config:BusinessObjectBuilder>
</_config:DataLoader>
</_config:DataloadBusinessObjectConfiguration>

Step 5:

Define an input CSV file with following content, first rows of the CSV indicates field names

MY_ID,firstname,lastname
1234,mydata_old,mydata_old
1234,mydata_new, mydata_new

Step 6:


c:\IBM\WCDE_ENT70\bin> dataload.bat ..\samples\dataload\wc-dataload.xml -D.level=FINER

You should see a similar output, as highlighted in the logs it has successfully processed two record from input feed. the first one was inserted and second record was updated as the record already exists in the table.

================================================================================
==
WebSphere Commerce Data Load
================================================================================
==

Load started at: Fri Oct 05 20:50:46 CDT 2012
Initialization completed in 4.204 seconds.

Processing XMYTable...

--------------------------------------------------------------------------------
--
Load summary for load item: XMYTable.
--------------------------------------------------------------------------------
--
Business Object Configuration: wc-loader-custom.xml
Data loader mode: Replace.
Batch size: 1.
Commit count: 100.
Error Tolerance Level: 1.
Error Count: 0.
Amount of data processed: 3.
Amount of business objects processed: 2.
Amount of business objects committed: 2.
Data loader initialization time: 0 seconds.
Data loader execution began: Fri Oct 05 20:50:50 CDT 2012
Data loader execution ended: Fri Oct 05 20:50:51 CDT 2012
Data loader completed in 0.85 seconds.
Total flush time: 0 seconds.
Total commit time: 0.001 seconds.
CSV file location: ..\samples\dataload\MyData.csv.
Affected tables (1):
Table name: XMYTABLE, Affected number of rows: 2.
--------------------------------------------------------------------------------
--


Program exiting with exit code: 0.
Load completed successfully with no errors.

Load ended at: Fri Oct 05 20:50:51 CDT 2012
Load completed in 5.07 seconds.


Review the log generated at following location to debug in case of issues
C:\IBM\WCDE_ENT70\logs\wc-dataload.log



11 comments:

  1. Excellent post! I appreciate your efforts. Keep it up. If you want to promote your business globally then I can help you. Read the given links here.

    Kenya Trade Data

    Kenya Export Data

    Kenya Import Export Data Provider

    Search Kenya Shipment Data

    ReplyDelete
  2. I am excited to watch your upcoming posts too. it encourages me to write and i have written on some hs codes. you can check my writings.

    HS Code 2201|
    HS Code 2202|
    HS Code 2203|
    HS Code 2204|
    HS Code 2205|
    HS Code 2206|
    HS Code 2207|
    HS Code 2208|
    HS Code 2209|
    HS Code 2301

    ReplyDelete
  3. Usually I never comment on blogs but your article is so convincing that I never stop myself to say something about it. You’re doing a great job Man,Keep it up.

    HS Code 2912|
    HS Code 2914|
    HS Code 2915|
    HS Code 2916|
    HS Code 2917|
    HS Code 2918|
    HS Code 2920|
    HS Code 2921|
    HS Code 2922|
    HS Code 2923

    ReplyDelete
  4. Found your post interesting to read. I cant wait to see your post soon. Good Luck for the upcoming update.This article is really very interesting and effective.

    HS Code 2924|
    HS Code 2925|
    HS Code 2926|
    HS Code 2927|
    HS Code 2929|
    HS Code 2930|
    HS Code 2931|
    HS Code 2932|
    HS Code 2933|
    HS Code 2934

    ReplyDelete
  5. Global data gathers all particular data about the products, and those all elegances are more valuable for exchanging. By the export import data trading business increments quickly. Thus, trading information includes all the details about the products that might help in various ways. Get the effective exchange and make your exchanging business undeniable level. Furthermore, get the shipment in universally.

    ReplyDelete
  6. Trade Data India helps to know about the export rules and notifications, excise duty savings, trade analysis, reducing risks, etc. The easy access to export data of India helps exporters to plan out their strategies and come up with better plans for trade business.

    ReplyDelete
  7. Intriguing analysis with solid research cited. A couple assumptions may be debatable but the core argument rings true in my view. A fresh take was refreshing to see. Could have explored one or two tangents further imo but overall well presented ideas. tarpaulins

    ReplyDelete
  8. This article tackles a complex issue and breaks it down in a way that's easily understandable. Thanks for simplifying it.
    heavy duty tarpaulin

    ReplyDelete