5. ODI-SCM Technical Manual¶
5.1. The Configuration (INI) File¶
The ODI-SCM command line commands are controlled by environment variables. These environment variables are loaded from, and saved to, the configuration file.
The configuration file in use by the ODI-SCM command line commands is always specified by the environment variable ODI_SCM_INI. This variable though is not frequently accessed by the commands. Instead, the environment should be loaded (the environment variables set) from the configuration file using the command “OdiScmEnvSet”. This command must always be invoked in the current shell (CMD.EXE) environemt, using CALL, for the “OdiScmEnvSet” command to have any useful effect.
The configuration file is updated, currently, only by the ODI-SCM commands that perform source code downloads, from the SCM system, (the OdiScmGet process) and import the source code into the ODI repository (the generated output of the OdiScmGet process).
So, the configuration file is really the persisted environment for the ODI-SCM command system. This, together with the ODI-SCM metadata that is maintained in ODI respository forms the complete system configuration.
| Section Name | Key Name | Key Description | Example Values |
|---|---|---|---|
| OracleDI | Admin Pass | Password of the database user with DBA privileges. Used when creating database users when creating ODI repositories. | xe |
| Admin User | User name of the database user with DBA privileges. Used when creating database users when creating ODI repositories. | system |
|
| Common | Path of ODI 11g common libraries directory. | C:\oracle\product\11.1.1\Oracle_ODI_1\oracledi.common |
|
| Drop With Purge | Determines whether repository tables
are purged from the Oracle database
rather than sent to the recycle bin
(dependent upon the DB settings)
Allowable values are Yes and
No. |
Yes |
|
| Encoded Pass | Encoded password of the ODI user. | fJyaPZ,YfyDCeWogjrmEZOr |
|
| Home | Path of ODI home directory. This is the directory containing the
|
C:\OraHome_1\oracledi 10g
C:\oracle\product\11.1.1\Oracle_ODI_1\oracledi\agent 11g |
|
| Java Home | Path of the JDK to be used with ODI.
This is the directory containing the
bin directory containing the
java.exe binary. |
C:\Program Files\Java\jdk1.6.0_45 |
|
| Oracle Home | Path of the Oracle Home for the ODI installation. This is normally the parent directory of the ODI Home directory. This path is used to find additional resources such as Java class archives (JAR) files and the standard ODI demo files for the some of the ODI-SCM demos. This entry is required when running the fast- forward of ODI-SCM demos, that use the standard ODI demo Hypersonic SQL databases, and when creating master and work repositories using the ODI-SCM tools. | C:\oracle\product\11.1.1\Oracle_ODI_1 |
|
| Pass | Unencoded password of the ODI user. | SUNOPSIS |
|
| Repository ID | ID number of the master and work repository. Used by operations that create an ODI repository, such as the AutoRebuild process. | 100 |
|
| SDK | Path of the ODI 11g SDK root directory. | C:\oracle\product\11.1.1\Oracle_ODI_1\oracledi.sdk |
|
| Secu Driver | Class name of the JDBC driver used to connect to the ODI repository. | oracle.jdbc.driver.OracleDriver |
|
| Secu Encoded Pass | Encoded password of the ODI master respository database user/owner. | gofpxBz5aa37kmG6I3eLyhVkiscy |
|
| Secu Pass | Unencoded password of the ODI master repository database user/owner. | odirepofordemo2 |
|
| Secu URL | JDBC URL of the ODI master repository. | jdbc:oracle:thin:@localhost:1521:xe |
|
| Secu User | Name of the ODI master repository database user/owner. | odirepofordemo2 |
|
| Secu Work Rep | Name of the ODI work repository attached to the master repository. | WORKREP |
|
| User | User name of the ODI user. | SUPERVISOR |
|
| Version | Version number of ODI. Currently only the major version number is significant to ODI-SCM. solution. |
|
|
| SCM System | Branch URL | The least significant part of the SCM URL. Typically, for TFS this is the Project and branch/folder path and for SVN this is the path within the root of the repository. |
|
| Global User Name | A user name to use to logon to the SCM system instead of the default user. For SVN the default user is the cached user, previously used to access the SVN repository. For TFS the default user is the currently logged in Windows user. | somedomain\anotheruser |
|
| Global User Password | The password of the user specified in the Global User Name key, if any. | thesecretstring |
|
| SQL Server AS Working Copy Root | The root directory, relative to the working copy root directory, of the SSAS code. Must be within the working copy directory tree. I.e. avoid using ”..” in this path. | src/ssas |
|
| SQL Server IS Working Copy Root | The root directory, relative to the working copy root directory, of the SSIS code. Must be within the working copy directory tree. I.e. avoid using ”..” in this path. | src/ssis |
|
| System URL | The most significant part of the SCM URL. Typically, for TFS this is the server and Team Project Collection, and for SVN this is the repository root URL. |
|
|
| Type Name | The name of the type of SCM system. temporary/working files. Must be set to SVN or TFS. | SVN |
|
| Working Copy Root | The root directory of the SVN working copy / TFS workspace. | C:/OdiScmWalkThrough/Repo2WorkingCopy |
|
| OracleDI Working Copy Root | The root directory, relative to the working copy root directory, of the ODI code. Must be within the working copy directory tree. I.e. avoid using ”..” in this path. | src/oracledi |
|
| Working Root | The working directory where the ODI-SCM export mechanism can create temporary/working files. | C:/OdiScmWalkThrough/Temp2 |
|
| Workspace Name | The Team Foundation (Server, TFS) workspace name of the working copy. temporary/working files. Currently used only by the OdiScmAutoRebuild process to destroy and recreate the TFS workspace for the working copy. | myworkspace1 |
|
| Tools | FitNesse Class Name | The Java class name of the entry point to FitNesse. Used to execute ODI unit tests built with DbFit and other FIT/SLIM fixtures via the FitNesse engine. |
|
| FitNesse Home | The home/installation diretory of FitNesse. | C:\dbfit-complete-2.1.0 |
|
| FitNesse Java Home | Path of the JVM (JRE or JDK) to be
used with FitNesse. This is the
directory containing the bin
directory containing the
java.exe binary. |
C:\Program Files\Java\jdk1.6.0_45 |
|
| Jisql Additional Classpath | Additional Java class directories and/or archives required for ODI-SCM operations against the ODI repository. No longer used, in general. |
C:\MyApp\bin;D:\AppLib\tools.jar;D:\AppLib\classes.zip |
|
| Jisql Home | Path of the Jisql home directory.
This is the directory containing the
runit.bat script and the lib
directory. |
C:\Jisql\jisql-2.0.11 |
|
| Jisql Java Home | Path of the JVM (JRE or JDK) home
directory to use with Jisql.
This is the directory containing the
bin directory containing the
java.exe binary. |
C:\Program Files\Java\jdk1.6.0_45 |
|
| MSBuild Home | Path of the Microsoft MSBuild build
tool home directory. This is the
directory containing the bin
directory containing the
msbuild.exe binary. |
C:\Program Files (x86)\MSBuild\12.0 |
|
| Oracle Home | Path of the Oracle client home
directory. This is the directory
containing the bin directory
containing the imp.exe and
exp.exe binaries. |
C:\oraclexe\app\oracle\product\11.2.0\server |
|
| SQL Server Home | Path of the SQL Server client home directory. This is the directory containing the subdirectories that contain the deployment tools for SSIS and SSAS projects. | C:\Program Files (x86)\Microsoft SQL Server\120 |
|
| Visual Studio Home | Path of the Visual Studio command
line interface, devenv.com, used
to build SSIS and SSAS projects. |
C:\Program Files (x86)\Microsoft Visual Studio 12.0 |
|
| Team Explorer Everywhere Java Home | Path of the JVM (JRE or JDK) home directory to use with Microsoft TEE, if appropriate. | C:\Program Files\Java\jdk1.6.0_45 |
|
| UnxUtils Home | Path of the UnxUtils distribution
home directory. This is the
directory containing the bin and
usr directories. |
C:\UnxUtils |
|
| Generate | Build Test Scope | Determines whether an incremental
or full set of unit test executions
are generated, to be executed after
the completion of the build process.
Valid values are incremental and
full. |
incremental |
| Export Ref Phys Arch Only | Controls whether ODI-SCM export
operations (export and flush) will
export non reference Topology
objects. For more on this subject
see the Reference Topology
section in the ODI-SCM Technical
Manual. Valid values are Yes and
No. |
No |
|
| Export Cleans ImportRep Objects | Controls whether the ODI-SCM export will remove SnpMImportRep and SnpImportRep objects from ODI object source files. Removing these allows ODI-SCM to populate a repository from source object files where the repository is not the original repository having the repository’s ID. The operation is normally blocked by the ODI import API but ODI-SCM makes this operation safe. Not applicable to ODI 10g. | Yes |
|
| OracleDI Context | The code of the ODI context used to execute ODI-SCM operations, such as flushing out code or configuring the components. | GLOBAL |
|
| Output Tag | The character string used as part of the names of the directories and files generated by the OdiScmGet process. If empty, then a tag composed of the current date and is used. | DemoEnvironment2 |
|
| Import Object Batch Size Max | The maximum number of ODI object
source files imported in a single
import operation.
Massive import performance gains
can be achieved by specifying a
value for this option. Valid values
are positive integers. A value of
1 means no optimisation. |
200 |
|
| Import Resets Flush Control | Controls whether the ODI-SCM import
process updates the ODI-SCM flush
control metadata. Valid values are
Yes and No. |
Yes |
|
| Scenario Export Markers | A sstring of one or more object markers used to indicate which source objects may have their Scenarios exported to the file system, and hence available to check into the source code control system rather than being removed from the export file created for the source object. | SMILEYS.SMILEY_2,MYMARKERS.MARKER17 |
|
| Scenario Source Markers | A string of one or more object
markers used to indicate which
source objects should have a
Scenario generated, by ODI-SCM,
after being imported into the ODI
repository. The format of a marker
in the list of markers is:
<Marker Group Code>.<Marker Code>
Markers in the list are separated by
comma (,) characters. |
SMILEYS.SMILEY_2,PRIORITIES.PRIORITY_1 |
|
| DML Script File Name Pattern N | Where N >= 0, a set of file name regular expression patterns, that specify the names of database DML scripts to be executed following all database DDL script execution. | dml.*\.sql |
|
| SQL Statement Delimiter | A string of one or more characters used as statement delimiters when running user SQL scripts. Used to to avoid conflicts with strings in user SQL scripts. | // |
|
| Test | ODI Standards Script | An optional path and name of a SQL script used to check the ODI code, in the repository, for naming, design, etc, standards violations. If specified this script will be run as part of the ODI-SCM generated ODI imports. The author of the script can choose to simply highlight the issues, or cause a failure in the imports, by coding the script appropriately. Applies only to incremental builds only. I.e. not to the initial build of an empty repositroy. | C:\Scripts\DemoODINamingStandardTest.sql |
| FitNesse Output Format | The output format of FitNesse tests.
Valid values are html and
text. html format is useful
for debugging test failures but
text format is required for the
correct detection of test failures
whilst running post build tests. |
text |
|
| FitNesse Port | The TCP port that FitNesse will use for its test runner processes, such as fit.FitServer or the SLiM test runner. | 8086 |
|
| FitNesse Root Page Root | The path of the root FitNesse page. | C:\MyWorkingCopy\FitNesseRoot |
|
| FitNesse Root Page Name | The name of the root FitNesse page. | FitNesseRoot |
|
| FitNesse Unit Test Root Page Name | The parent page path of the ODI object unit test FitNesse pages. | MyProject.EtL.UnitTests |
|
| OracleDI Context | The code of the ODI context used to execute post build tests. | GLOBAL |
|
| Import Controls | OracleDI Imported Revision | Tracks the highest revision number, from the SCM system, that has been imported into the ODI repository. This entry is updated by ODI-SCM generated ODI import scripts. | 123 |
| Working Copy Revision | Tracks the highest revision number, from the SCM system, that has been applied to the working copy. This entry is updated by the OdiScmGet process. | 123 |
|
| Notify | User Name | Tracks the highest revision number, from the SCM system, that has been imported into the ODI repository. This entry is updated by ODI-SCM generated ODI import scripts. | Mark Matten |
| Email Address | The email address used to notify the user of the completion (success or failure) of build processes. | mattenm@odietamo.org.uk |
|
| SMTP Server | The host name or IP addresss of an SMTP server used to send email notifications. | mail.yourdomain.co.uk |
|
| On Build Status | Whether to send a notification on
build success, build failure,
both or neither.
Valid values are success,
failure, both and
neither. |
both |
|
| Misc | Resources Root | Path of the directory used for miscellaneous resource files. | C:\OdiScmResources |
| Temp Root | Path of the directory used for general temporary, working, files. Used to override the default use of %TEMP%-else-%TMP% to place files. | C:\Temp |
|
| Data Servers | <Data Server Name> | A repeating key of varying name. JDBC connection properties for a data server type to which database objects will be deployed. Properties and their values are separated by “+”. The properties must appear be in the following format: -
Note that the “Drop With Purge” key is only relevant for Oracle data servers. |
hrsy1=DBMS Type+mysql+JDBC URL+jdbc:mysql:myDbServer12+User
Name+usr3+Password+psswd3 |
| Logical Physical Schema Mappings | <Logical Schema Name> | A repeating key of varying name. Physical schemas into which database objects will be deployed, from DDL and SPL scripts, and into which meta /reference data will created, from DML scripts, from the SCM system. Properties and their values are separated by “+”. The properties must appear be in the following format: -
|
billing=Data Server+s1+Database+billdb4+Schema+billing_main
+Token Values+ReplaceThis=WithThis/AndThis=ByThis |
| SSIS | Server Name | The name of the Integration Services server from which packages are executed and to which projects are deployed. | mysqlserver01 |
| Catalogue Path | The Integration Services catalogue (SSISDB) path from which packages are executed and to which projects are deployed. | MyFolder |
A example configuration file with all sections and keys listed:
[OracleDI]
Admin Pass=xe
Admin User=system
;Version=10.1.3.5.6_02
;Home=C:\oracledi
;Java Home=C:\Java\jdk1.5.0_22
;For ODI10g, Common and SDK are not used.
;Common=
;SDK=
;For ODI11g, enable Common and SDK.
Drop With Purge=Yes
Common=C:\oracle\product\11.1.1\Oracle_ODI_1\oracledi.common
SDK=C:\oracle\product\11.1.1\Oracle_ODI_1\oracledi.sdk
Encoded Pass=fJyaPZ,YfyDCeWogjrmEZOr
Home=C:\oracle\product\11.1.1\Oracle_ODI_1\oracledi\agent
Java Home=C:\Program Files\Java\jdk1.6.0_45
Oracle Home=C:\oracle\product\11.1.1\Oracle_ODI_1
Pass=SUNOPSIS
Repository ID=200
Secu Driver=oracle.jdbc.driver.OracleDriver
Secu Encoded Pass=gofpxBz5aa37kmG6I3eLyhVkiscy
Secu Pass=odirepofordemo2
Secu URL=jdbc:oracle:thin:@localhost:1521:xe
Secu User=odirepofordemo2
Secu Work Rep=WORKREP
User=SUPERVISOR
Version=11.1.1.6.4
[SCM System]
Branch URL=.
Global User Name=
Global User Password=
OracleDI Working Copy Root=src/oracledi
System URL=file:///C:/OdiScmWalkThrough/SvnRepoRoot
SQL Server AS Working Copy Root=src/ssas
SQL Server IS Working Copy Root=src/ssis
Type Name=SVN
Working Copy Root=C:/OdiScmWalkThrough/Repo2WorkingCopy
Working Root=C:/OdiScmWalkThrough/Temp2
;Workspace Name is for TFS, only, and only used by repository rebuilds.
Workspace Name=
[Tools]
FitNesse Class Name=fitnesseMain.FitNesseMain
FitNesse Java Home=C:\Program Files\Java\jdk1.6.0_45
FitNesse Home=C:\dbfit-complete-2.0.0-rc5
Jisql Additional Classpath=
Jisql Home=C:\Jisql\jisql-2.0.11
Jisql Java Home=C:\Program Files\Java\jdk1.6.0_45
MSBuild Home=C:\Program Files (x86)\MSBuild\12.0
Oracle Home=C:\oraclexe\app\oracle\product\11.2.0\server
SQL Server Home=C:\Program Files (x86)\Microsoft SQL Server\120
Team Explorer Everywhere Java Home=C:\TEE
UnxUtils Home=C:\UnxUtils
Visual Studio Home=C:\Program Files (x86)\Microsoft Visual Studio 12.0
[Generate]
Build Test Scope=full
DML Script File Name Pattern 0=^dml-schema-.*\.sql
Export Ref Phys Arch Only=No
Export Cleans ImportRep Objects=Yes
Import Object Batch Size Max=100
Import Resets Flush Control=Yes
OracleDI Context=GLOBAL
Output Tag=DemoEnvironment2
Scenario Source Markers=ODISCM.HAS_SCENARIO
Scenario Export Markers=ODISCM.ALLOW_SCENARIO_EXPORT
SQL Statement Delimiter=//
[Test]
ODI Standards Script=
; "html" or "text".
FitNesse Output Format=text
FitNesse Port=8086
FitNesse Root Page Root=C:/OdiScmWalkThrough/Repo1WorkingCopy/SvnRepoRoot
FitNesse Root Page Name=FitNesseRoot
FitNesse Unit Test Root Page Name=OdiScmDemo
OracleDI Context=GLOBAL
[Import Controls]
OracleDI Imported Revision=12345
Working Copy Revision=12345
[Notify]
User Name=
Email Address=
SMTP Server=
[Misc]
Resources Root=C:\OdiScmResources
Temp Root=X:\Temp\OdiScm
[Data Servers]
hrsy1=DBMS Type+mysql+JDBC URL+jdbc:mysql:myDbServer12+User Name+usr3+Password+psswd3
[Logical Physical Schema Mappings]
billing=Data Server+hrsy1+Database+billdb4+Schema++Token Values+ReplaceThis=WithThis/AndThis=ByThis
[SSIS]
Server Name=mysqlserver01
Catalogue Path=MyFolder
5.2. The Get Process¶
The OdiScmGet command is the command that updates the working copy, from the SCM system, and generates the scripts to update the ODI repository with the new/changed files from the Get operation.
5.2.1. Dealing with Conflicts¶
Details of how to handle conflicts between your code and incoming code from the Get process - coming very soon!
Details of how to handle check in conflicts - coming very soon!
5.3. The Flush Process¶
The ODI repository flush is the process that exports additions and changes, made to the ODI repository (either via the ODI UI, or the ODI 11g SDK) to the working copy so that the new/changed code can be added and checked in to the SCM system.
The flush process is invoked either from the command prompt, using the OdiScmFlushRepository command, or from the ODI Designer UI, by executing the Scenario:
ODI-SCM (project) -> COMMON (folder) -> Packages -> OSUTL_FLUSH_REPOSITORY -> Scenarios -> OSUTL_FLUSH_REPOSITORY Version 001
Note: you might see the version number 1 instead of 001 depending upon the version of ODI you’re using.
5.4. Reference Topology¶
Details coming soon!
5.5. Build Import Performance¶
ODI’s startcmd.bat / OdiImportObject interface is extremely slow when used to import a large number of granular (small) ODI object source files. This is partially because of the overhead of starting up a JVM for each import operation. But, the main performance impediment is OdiImportObject itself. The size of the data file being imported does not appear to be a major contributor to the time it takes the import operation to complete. We can only think there must be some kind of global repository integrity validation step that occurs for each import.
The main weapon we have to improve performance is batching of ODI object source files into a larger import source file. Batching is controlled via the configuration file entry Import Object Batch Size Max in the section [Generate].
We have experienced massive performance boosts by specifying a value for this parameter. On a Windows XP desktop PC, with 3GB of RAM, Intel i5 CPU, we’ve seen our imports, of around 11,000 source files, go from around 20 hours to under one hour!
5.6. Configuring Your SVN Client¶
If you’re using Subversion, not TFS, with ODI-SCM, then you will need to prevent SVN from automatically merging changes in ODI object source files. We do not want to let SVN merge changes, coming from the SVN repository into the working copy, with changes made to the ODI object source file, via the ODI UI (and exported via ODI-SCM).
This is because SVN will perform a textual merge of the two sets of changes and produce a new merged (text) file. Although the ODI object source files, produced by ODI-SCM, are text (XML) files the textual merge performed by SVN is not guaranteed to produce a usable/coherent ODI object source file.
So when we prevent SVN from doing this SVN will highlight any conflicts between theirs (the incoming changes from the SVN repository) and ours (the code we’ve exported from our ODI repository) at the source file level.
How we deal with any conflicts that we come across is discussed in another section.
To tell SVN not to automatically merge ODI object source files, we tell SVN to treat these file types as binary file types. SVN will not attempt to merge changes for binary files (because the results are unlikely to be useful). We tell SVN to treat the ODI source object files as binary by assigning each file the SVN property svn:mime-type and a property value of application/octet-stream. This property is assigned to the file when the file is first created in SVN repository.
5.6.1. The SVN Configuraton File¶
The SVN configuration file, named config, is created by SVN the first time that the SVN command line client (svn.exe) is run. On Windows systems it exists in a directory called Subversion that is located in AppData directory of the user’s profile directory. The user’s profile directory has different locations depending upon the version of Windows being used.
E.g. on a Windows 7 machine, the config file might be:
C:\Users\Mark Matten\AppData\Roaming\Subversion\config
E.g. on a Windows XP machine, the config file might be:
C:\Documents and Settings\mattenm\AppData\Roaming\Subversion\config
For more information on this subject see the SVN book, online at http://svnbook.red-bean.com/en/1.7/svn.advanced.confarea.html.
To enable to automatic property assignment, ensure that in the [miscellany] section of the configuration file ensure that the entry enable-auto-props is set to yes. I.e.:
enable-auto-props = yes
In the [auto-props] section of the configuration file add an entry, for each of the ODI object types that are exportable by ODI-SCM. You can copy and paste the following into your configuration file:
*.SnpTechno = svn:mime-type=application/octet-stream
*.SnpConnect = svn:mime-type=application/octet-stream
*.SnpPschema = svn:mime-type=application/octet-stream
*.SnpLschema = svn:mime-type=application/octet-stream
*.SnpContext = svn:mime-type=application/octet-stream
*.SnpProject = svn:mime-type=application/octet-stream
*.SnpFolder = svn:mime-type=application/octet-stream
*.SnpTrt = svn:mime-type=application/octet-stream
*.SnpPackage = svn:mime-type=application/octet-stream
*.SnpPop = svn:mime-type=application/octet-stream
*.SnpVar = svn:mime-type=application/octet-stream
*.SnpUfunc = svn:mime-type=application/octet-stream
*.SnpSequence = svn:mime-type=application/octet-stream
*.SnpGrpState = svn:mime-type=application/octet-stream
*.SnpModFolder = svn:mime-type=application/octet-stream
*.SnpModel = svn:mime-type=application/octet-stream
*.SnpSubModel = svn:mime-type=application/octet-stream
*.SnpTable = svn:mime-type=application/octet-stream