Cool Solutions

IT TechTalk at BrainShare 2010

mattclayton — Mon, 23 Nov 2009 13:29:13 -0700

One of the most popular and attended events at BrainShare is back! IT Tech Talk (formerly known as "Meet the Experts") will take place Wednesday night from 6:30 - 9:30 p.m. As always, you'll be able to mingle with our product engineers giving you an opportunity to discuss current and future product features and technologies. Heavy hors d'oeuvres and an open bar will be available.

Cool Solutions

How to Configure Access Gateway Embedded Service Provider to Reduce Access Gateway Load and Improve Performance

ncashell — Mon, 23 Nov 2009 11:01:01 -0700

Introduction:

The goal of the following document is to explain how to improve the Linux Access Gateway server performance and stability by including all attributes referenced by protected resource policies in a SAML assertion sent at authentication time.

In large production environments, it is commonplace to overload the Access Gateway to the point where utilization and server performance are negatively impacted. This document describes how attribute maps and SAML assertions can be used to significantly reduce traffic between Novell Access Manager Identity Servers and Access Gateways.

By understanding and taking advantage of some enhancements to Access Manager beginning with the release of 3.1 Support Pack 1 Interim Release 2, a lot of unnecessary work can be offloaded from the Access Gateway, improving performance, stability and the user experience.

Background information:

The Access Gateway (AG) is responsible for

Protecting web applications/services based on their distinct URLs
Providing required attributes to allow single sign on to back end applications/services.

Protection of such services often requires authentication. Because authentication is done at the Identity Server (IDS), the Access Gateway must be able to communicate with this IDS server to receive the authentication details. Such authentication details are sent via a SAML assertion.

The protection of services also requires authorization or single sign on decisions. Attributes required in the decision making process must be retrieved from the IDS, over what is called the SOAP back channel.

One can already design a solution leveraging roles (http://www.novell.com/documentation/novellaccessmanager31/policies/data/b995x1b.html) but there are many setups that require additional LDAP attributes.

Communication flow:

Although the AG host may have multiple proxy services defined, only ONE of those services hosts the Embedded Service Provider (ESP, and also known as the federation service) used to talk to the IDS via the SOAP back channel. Typically, the first reverse proxy/proxy service is used to host the federation service (via a reserved path of /nesp) although this is configurable. Any time the IDS needs to be invoked for authentication or policy evaluation, the user session details are always sent to the federation service (ESP) on the AG first before being redirected to the IDS. This AG federation service knows how to generate the required federation messages to send to the IDS.

When the AG proxy needs to execute a locally enabled policy (Identity Injection, Form Fill, or Authorization) the following steps are executed

The AG proxy sends a SOAP request to its local ESP to evaluate the policy.
If the ESP has all required identity information already cached, then the policy is evaluated locally and the response is returned to the proxy.
If the ESP does not have the information cached, it will query the authoritative ESP (the AG ESP that was originally involved in authenticating the user and so establishing the user session). If this is NOT a clustered environment, then this step is omitted. This query of the authoritative ESP involves:
1. Identifying which ESP in the cluster holds the user session details
2. Proxy'ing the SOAP request to the authoritative ESP to try and evaluate the information required to satisfy the policy
If the authoritative ESP has the required identity information, then it is returned to allow the policy to be evaluated. If not, then the initial ESP will query an IDS.
If the IDS has the user information already cached, then it is returned. If not, and the IDS is not authoritative for this user session then the same process used by the ESP is used to locate the IDS holding the user session. The IDS then , and then sends the SOAP request to that authoritative IDS.
As with the ESP communication above, this latter step is omitted if not in a clustered environment.
If the authoritative IDS does not have the identity information already cached, then it will make an LDAP call to retrieve the required information from the user store and return it.
As can be seen, the overhead (especially in clustered environments) of constantly communicating over the SOAP back channel during authentication and policy evaluation can have a major impact on performance.

Typical issues seen in clustered environments include ESP CPU utilization going very high, Tomcat running out of threads, LDAP server overloading, data read timeout during the proxy'ing of requests to the authoritative ESP/IDS servers.

Performance improvement options:

In order to mitigate such overhead, administrators should consider the benefits of identifying the attributes required by all ESP policies and including them in SAML assertions sent between the IDS and ESP at authentication time. The assertion includes authentication statements (about the subject that it is authenticating) as well as attribute statements (attributes about the user). The attributes from this attribute statement will be cached at the ESP and used locally when evaluating policies. The end result is an elimination of the requests to the IDS to retrieve user attributes required to satisfy the policy.

The following guidelines, when configured correctly, will result in a huge reduction in traffic over the SOAP back channel. This resulting traffic reduction will generate a corresponding performance increase on the AG servers.

Identify all attributes required on all policies enabled for each protected resource.
Identify all policies that are enabled on each defined protected resource. Go through each of these policies and note the attributes that are required by this policy. In the following example, a single policy is enabled on one protected resource. Although this may not be realistic, this exercise will show how to verify that the policy requires the following attributes: all user roles, LDAP cn, LDAP roomNumber, LDAP mail and LDAP title attributes.

Click to view.
Define an attribute set that will contain all attributes required by the AG enabled policies.
After identifying all the required attributes from the previous step, the administrator must go to the 'Shared Settings' tab on the IDP configuration and define a new attribute set. After creating a new attribute set and giving it a logical name, add each attribute required by clicking the new option. In this example there will be 5 entries: all roles, LDAP cn, LDAP roomNumber, LDAP mail and LDAP title attributes.

Note that the local attribute must include the attribute that the IDS will evaluate. There is an option to define the remote attribute name, but this is ignored for communications between IDS and ESP.

Click to view.
Select the AG or AG cluster configuration where the newly defined Attribute Set will be used
Go to the IDS configuration, and select the 'Liberty' tab. Under 'Trusted Providers', there will be a link to the AG cluster configuration name. If there is no AG cluster configuration, the IP address of the AG server will appear under this 'Trusted Providers' link.

Click to view.
Add the newly defined Attribute Set to the Liberty relationship between IDS and selected ESP.
After selecting the Trusted AG Service provider, select the attribute set defined in step 2 from the drop down menu. Once the attribute set is selected, the list of attributes from that Attribute set will appear on the right hand side of the screen. These are the attributes available for selection. Select each attribute and make sure that it moves across to the 'Send with Authentication' menu. Doing this will force the attributes to be resolved at authentication time, so that they are sent with the subject details in the SAML assertion.

Click to view.
Define the attribute refresh rate on the policy
When using LDAP attributes in an Identity Injection or Form Fill policy, the option to define a refresh rate exists. This refresh rate determines how often the AG proxy must go back to the ESP to determine whether the data is valid or stale. For performance purposes it is recommended that the 'Session' setting be defined, so that we only retrieve the attributes once during the session lifetime. Although no requests will go back over the back channel to the IDS server, it will reduce communication between the AG proxy and the ESP.

Click to view.
Injecting IDS user name and password to back end Web server
If the policy requires that the credential profile username and password be sent across to the back end Web server, the attribute map created above must include the credential profile details. Unlike regular LDAP attributes in the above example, these credential profile attributes MUST be mapped to a 'Remote Attribute' name. Note that this remote attribute name is case sensitive. The three credential profile attributes that need to be mapped are as follows:

Click to view.

When defining the attributes to send to the back end Liberty ESP, we will only need to send the UserName and userPassword. The userDN may be left in the available list as it is already sent over in a SAML assertion by default at authentication time.

Click to view.

Validating configuration

Before rolling out the changes in production, there are a number of simple tests that an administrator can perform to confirm that no unnecessary SOAP back channel requests are being made from the ESP to the IDS server when policies are being evaluated.

Turn on verbose logging at IDS server temporarily: Select the Identity Provider configuration tab in the Administration Console and click 'Logging'. Under 'Component File Logger' set the following components to verbose : Application, Liberty, Web Service Provider and Consumer.

Click to view.

Access a policy enabled protected resources on the AG and check the ESP and IDS log files. Look at the output of the catalina.out file on both the IDS/ESP servers

The catalina.out file includes all debug information, assuming the above IDP log settings are enabled. This file is located in /var/opt/novell/tomcat5/logs/ on both IDP/ESP servers. Navigating these log files in debug mode can be confusing to say the least, so key entries have been identified to look for, and which server and log file they are found in.

Verify that the SAML assertion sent at authentication time includes the AttributeStatement containing all defined attributes. This can be done by searching the catalina.out file on either the IDP or ESP for the "AttributeStatement" string. When a user authenticates to the above example setup, the output shown on both IDP/ESP servers is the following:

<saml:AttributeStatement>
                  <saml:Subject>
                     <saml:NameIdentifier Format="urn:liberty:iff:nameid:one-time" NameQualifier="https://lag129.lab.novell.com:443/nesp/idff/
metadata">
                      9xVBI/rKBwt4wLXim82945nMv+yYyrjmOFkNFg==
                     </saml:NameIdentifier>
                     <saml:SubjectConfirmation>
                        <saml:ConfirmationMethod>
                         urn:oasis:names:tc:SAML:1.0:cm:artifact
                        </saml:ConfirmationMethod>
                     </saml:SubjectConfirmation>
                  </saml:Subject>
                  <saml:Attribute AttributeName="ldapcn" AttributeNamespace="urn:oasis:names:tc:SAML:1.0:assertion">
                     <saml:AttributeValue>
                      XX
                     </saml:AttributeValue>
                   </saml:Attribute>
                  <saml:Attribute AttributeName="ldapmail" AttributeNamespace="urn:oasis:names:tc:SAML:1.0:assertion">
                     <saml:AttributeValue>
                      XX
                     </saml:AttributeValue>
                  </saml:Attribute>
                  <saml:Attribute AttributeName="userRoles" AttributeNamespace="urn:oasis:names:tc:SAML:1.0:assertion">
                     <saml:AttributeValue>
                      XX
                     </saml:AttributeValue>
                     <saml:AttributeValue>
                      XX
                     </saml:AttributeValue>
                     <saml:AttributeValue>
                      XX
                     </saml:AttributeValue>
                  </saml:Attribute>
                  <saml:Attribute AttributeName="title" AttributeNamespace="urn:oasis:names:tc:SAML:1.0:assertion">
                     <saml:AttributeValue>
                      XX
                     </saml:AttributeValue>
                   </saml:Attribute>
                  <saml:Attribute AttributeName="roomnum" AttributeNamespace="urn:oasis:names:tc:SAML:1.0:assertion">
                     <saml:AttributeValue>

When examining the log entry, note the following:

Some attributes are multivalued (such as the userRoles attribute ), and will therefore have multiple "AttributeValue" entries.

For security purposes, the "AttributeValue" includes an XX string and not the appropriate value

Verify that the attribute statement is processed correctly and that the attribute values are added to the local cache. This is visible from the catalina.out file on the ESP, where a separate entry should exist for each attribute added. The one below shows the mail attribute being added to an internal ESP structure.

<amLogEntry> 2009-09-30T11:10:25Z DEBUG NIDS WSC:
Method: WSCCacheAlreadyReadCache.add
Thread: http-127.0.0.1-8080-Processor22
Added WSCCacheAlreadyReadCacheSet:
NEPXurn~3Anovell~3Aldap~3A2006-02~2Fldap~3AUserAttribute~40~40~40~40WSCQLDAPToken~40~40~40~40~2FUserAttribute~5B~40ldap~3AtargetAttribute~3D~2
2mail~22~5D </amLogEntry>

Identify a AG policy referencing the required attributes and note the policy ID. This step involves looking through the /var/log/ics_dyn.log file on the Linux Access Gateway (LAG) when the LOG_LEVEL setting in /etc/laglogs.conf entry is set to 7 (default is 5). Search for the URL where the Identity Injection policy is enabled and locate the 'Sending eval' string. This includes an eval request and policyID that we can search for in the ESP to confirm that the data has been retrieved from local cache. The snippet below shows an eval request number of 2641 and a policyID of "57M81710-NL1N-L610-O816-54MM7N558146". We can specifically see the LAG sending a SOAP request to the ESP, and whether or not a response was obtained.

Sep 30 12:12:28 lag129 : AM#504506000: AMDEVICEID#ag-7AA324FFCBA4D4E: AMAUTHID#591E1B49DACAF72693531BCA
5C3FF802: AMEVENTID#410: IdInjection enabled for  the protected resource
:
Sep 30 12:12:28 lag129 : AM#504506000: AMDEVICEID#ag-7AA324FFCBA4D4E: AMAUTHID#591E1B49DACAF72693531BCA
5C3FF802: AMEVENTID#410: II:a5304b64 Sending EVAL Request 2641 policyId 57M81710-NL1N-L610-O816-54MM7N5
58146
Sep 30 12:12:28 lag129 : AM#504512000: AMDEVICEID#ag-7AA324FFCBA4D4E: AMAUTHID#0: AMEVENTID#7079: proce
ssSoapRequests - size 2 processed 1, deleted 0 (0, conFail 0 conTimeout 0) 0 (0)
Sep 30 12:12:28 lag129 : AM#504515000: AMDEVICEID#ag-7AA324FFCBA4D4E: AMAUTHID#0: AMEVENTID#0: Connection Established with peer 127.0.0.1:8080 (src 127.0.0.1:0)
Sep 30 12:12:28 lag129 : AM#504512000: AMDEVICEID#ag-7AA324FFCBA4D4E: AMAUTHID#0: AMEVENTID#2641: sentsoapRequest 2641  app a91de4c8  II
Sep 30 12:12:28 lag129 : AM#504512000: AMDEVICEID#ag-7AA324FFCBA4D4E: AMAUTHID#0: AMEVENTID#2641: backchannel receivedResp (app   a91de4c8  II )   (2641)[seg:0xa4b87430:0xa59355a0:1248]
Sep 30 12:12:28 lag129 : AM#504506000: AMDEVICEID#ag-7AA324FFCBA4D4E: AMAUTHID#591E1B49DACAF72693531BCA
5C3FF802: AMEVENTID#410: Received response for IdInjection EVAL request

Confirm that the ESP received the request, and returned the response from cache. Failure to see these steps probably indicates that the ESP has had to contact the IDS to retrieve the requested information. This is done by searching the catalina.out on the ESP for the policyID or EVAL number. When the policy is found, verify that the attributes we are requesting are filled from cache. The snippet below shows the LDAP cn attribute being retrieved from cache.

<amLogEntry> 2009-09-30T11:12:28Z VERBOSE NIDS Application: AM#501101020: AMDEVICEID#esp-7AA324FFCBA4D4ED: NXPESID#2641:  <?xml version="1.0"
encoding="UTF-8"?><Evaluate PolicyId="57M81710-NL1N-L610-O816-54MM7N558146" Verbose="on">
                                <ContextDataElement Enum="2551" Value="591E1B49DACAF72693531BCA5C3FF802"/>
                        </Evaluate> </amLogEntry>

<amLogEntry> 2009-09-30T11:12:28Z INFO NIDS Application: AM#501101050: AMDEVICEID#esp-7AA324FFCBA4D4ED: PolicyID#57M81710-NL1N-L610-O816-54MM7
N558146: NXPESID#2641:  Evaluating policy </amLogEntry>

:
:

<amLogEntry> 2009-09-30T11:12:28Z DEBUG NIDS WSC:
Method: WSC.fillFromCache
Thread: http-127.0.0.1-8080-Processor21
Processing set: NEPXurn~3Anovell~3Aldap~3A2006-02~2Fldap~3AUserAttribute~40~40~40~40WSCQLDAPToken~40~40~40~40~2FUserAttribute~5B~40ldap~3AtargetAttribute~3D~22cn~22~5D </amLogEntry>

<amLogEntry> 2009-09-30T11:12:28Z DEBUG NIDS WSC:
Method: WSC.fillFromCache
Thread: http-127.0.0.1-8080-Processor21
Request filled from WSC Already Read Cache! </amLogEntry>

<amLogEntry> 2009-09-30T11:12:28Z INFO NIDS WSP: AM#500103001: AMDEVICEID#esp-7AA324FFCBA4D4ED: AMAUTHID#591E1B49DACAF72693531BCA5C3FF802:  Fi
lled the user attribute request from data already in the web service consumer cache. </amLogEntry>

<amLogEntry> 2009-09-30T11:12:28Z INFO NIDS Application: AM#501101056: AMDEVICEID#esp-7AA324FFCBA4D4ED: AMAUTHID#591E1B49DACAF72693531BCA5C3FF
802: PolicyID#57M81710-NL1N-L610-O816-54MM7N558146: NXPESID#2641:  Data retrieval ok:  from cached String[] value </amLogEntry>

After validating this information, ensure that all log levels are set back to the default.

Cool Solutions

Install and Upgrade with EVMS Partitioned Server

sakila — Fri, 20 Nov 2009 13:55:02 -0700

Creating EVMS partitions on a clean disk machine:

Click Partitioning on the Installation Settings screen and select Create EVMS Based Proposal and click Next.

Click to view.
It creates a boot partition and lvm container with EVMS volumes on the second partition.

Click to view.

Creating EVMS partitions on top of existing partitions:

If the server has the partitions created already with a previous installation, delete the old partitions and create the EVMS partitions new.

Follow the steps below to clean up the disk and create the partitions.

Choose Create Custom Partition Setup

Click to view.
Then select disk

Click to view.
Press the button Use entire hard disk and choose the Create EVMS Based Proposal option.

Click to view.
It goes back to the Installation Settings screen with the EVMS partitions created.
It then deletes the existing partitions and creates a new boot partition and lvm container with EVMS volumes.

Click to view.

Upgrading a single hard disk server with the EVMS partition:

Install OES 2 SP1 with EVMS Partitioning to create NSS pools and Volumes.
Down the server and upgrade to OES 2 SP2. During System for Update screen, while mounting the partitions, it shows a window saying the /dev/evms/sda1 could not be mounted.
Select the button "Specify Mount options".

Click to view.
By default the path under Device will be /dev/evms/sda1.

Click to view.
Edit the Device to remove evms i.e /dev/evms/sda1 to /dev/sda1, then click OK.

Click to view.
Start the Update on the EVMS Partitioned server.

Click to view.

Cool Solutions

Error Codes of the SAP HR driver for Identity Manager - Part 1

geoffc — Fri, 20 Nov 2009 12:13:09 -0700

SAP HR Driver, error messages:

Table of Contents:

Introduction
Bad Password for account in SAP
Fatal iDOC processing error
Error on bad iDOC format

Introduction:

Novell Identity Manager has a list of various pre built drivers for a variety of systems. For example, Active Directory, Lotus Notes, SAP UM, SAP HR, PeopleSoft, BMC Remedy, and so on.

There are a couple of generic drivers as well, that handle a large number of different systems. These include the JDBC driver, which can connect to most databases that have a JDBC interface available. There is the LDAP driver that connects to a large number of LDAP systems, There is the SOAP driver which can connect in principle to any SOAP (SPML or DSML) based web service. There are the Delimited Text driver, for when all else fails, as long as you can you get a text file (CSV perhaps) dump of the data you can use this driver. You could always write your own, using the Java API to connect to the target systems native interfaces, or when available you can use the Scripting driver to do the same, but in a simpler fashion.

Between all these options you should be good to connect most systems as needed.

Once you have a nice system set up, troubleshooting is always entertaining. One criticism I have of the Novell provided documentation is that it does not have sufficient documentation of possible error codes that can occur for each driver. Now to be fair, this is a pretty hard task, as many of the error codes are not actually generated on the Novell Identity Manager engine side, and rather are often system specific to the connected system. Nonetheless I think it would be very beneficial to include such error codes and cases. I have noticed that the troubleshooting section of the driver docs have been getting filled in, with at least some minor amount of details in the latest revisions of the documentation, which is a very good thing. But as always, I encourage more content from the writers.

Rather than just be a blow hard and annoy people, I decided to try and work on this issue myself.

I have a series of articles along this train of thought. For the JDBC driver I wrote:

These articles list the error message, showing trace, and the error itself, and where I knew an answer, explain what went wrong, and how to fix it. My hope is that the next person who encounters this error in trace, will copy and paste into a Google search, and whammo, find at least one possible suggestion on how to resolve it. Should it not be the exact error case, needing a different solution, at least they will have a hint on how it affected someone else and a thought on how to proceed.

I did the same for the Active Directory driver in this series of articles:

For the eDirectory driver there is:
Error Codes of the eDirectory Driver for Identity Manager - Part 1

I have more content stored up, that I need to finish up writing, like this article on the SAP HR driver error codes. I have a bunch of error messages from the GroupWise, and eDirectory drivers I need to write about.

I highly recommend that when you work with a new driver, keep a nice text editor open, and as each error occurs, paste it into a continuing file, and as soon as you figure out the problem, write down a couple of lines explaining the issue. Then when you have some time, write it up in this sort of format and submit it to Cool Solutions. This way everyone benefits!

Additionally, if you are not aware of it, the Novell Support Forums do an excellent job of supporting Novell products for free. The community works to answer the questions, and Novell does encourage people by selecting some of the more common contributors to be Novell Knowledge Partners. I personally follow the Identity Manager forums and lightly watch some of the others. You can find the Novell Support Forums at http://forums.novell.com or you can use NNTP in a news reader like Thunderbird or the GroupWise client as well. I personally prefer NNTP over a web interface, but the web interface is more easily searched and indexed, so different strokes for different folks.

Before we get to the error codes, a few quick points about the SAP HR driver, SAP as a system is HUGE! It has many many components and systems, and lots of people involved in getting it going. There are actually several SAP drivers from Novell. The original two are the SAP UM, and HR modules. The SAP UM lets you synchronize users in and out of SAP, as users of the system. That is, as users in SAP, who can log in to a module and do work. Alas, the way passwords work in SAP, they are per module. There is a specific SAP module called the CUA (Central User Agent?) which the SAP UM driver can connect too, and push users into, and then the CUA within SAP can push those users to other modules, but it does not push passwords. In which case, you might have a standard SAP UM driver connecting to the CUA to synchronize users, but need additional drivers for all the systems you wish to synchronize passwords too, that only sync passwords, and no other attributes.

Novell has since released a couple of SAP drivers for GRC, and Netweaver that I have not had time to look at, but sound interesting.

The SAP HR driver is meant to synchronize employees out of the HR system, as they are hired, transferred, and withdrawn (nobody is terminated in SAP, they are withdrawn). These are used to create user objects in the Identity Vault to populate and control users in all the other systems.

This article is just about the SAP HR driver. This driver is somewhat peculiar in at least three ways.

There are two different communication methods. The Publisher channel (events from SAP HR coming to IDdentity Manager) uses iDOCs to transmit the information. For more information about iDOC's in this context, you can read my previous articles on the topic.
- Troubleshooting iDOC Issues in the SAP HR Driver for Identity Manager
- Decoding iDOCs with the IDM SAP Driver
The Subscriber channel uses BAPI, which needs the Jconnect libraries, (here are a couple of articles related to that:
- IDM SAP HR Driver: Where to Get the JConnect Libraries
- Using the SAP HR Driver for IDM on a 64-bit OS
When an event comes through the Publisher channel as an iDOC the data in the iDOC is all you can query. While you can try and query back to SAP HR, really all you are looking at is the iDOC in memory. This is a very frustrating limitation.
RELATIONSHIPS: SAP HR uses a relatively complex system for managing reporting structure and relationships. Ask an SAP guy and he will say, but of course you do it like this, it is simple. To every one else it is very confusing. The RELATIONSHIP data is only available during the life span of the iDOC in memory in the Remote Loader (aka while it is being processed) and there is no way to query back for it either.

Now on to the errors:

Error on bad iDOC format:

DirXML: [10/02/08 14:01:10.232]: TRACE:  ParseIDoc: IDoc file opened successfully.
DirXML: [10/02/08 14:01:10.258]: TRACE:  ParseIDoc: Exception in IDoc Parsing: java.lang.NumberFormatException: For input string: "        ".  File processing terminated.
DirXML: [10/02/08 14:01:10.268]: TRACE:  ParseIDoc: File '/idm/idocs/O_400_00000000115871645' renamed to '/idm/idocs/O_400_00000000115871645.proc' successfully.

DirXML: [10/02/08 14:01:10.750]: TRACE:  SAPPublicationShim: Setting 'success' status on eventObject 'P+00011188'
DirXML: [10/02/08 14:01:10.751]: TRACE:  ParseIDoc: Status of Published document 'O_400_00000000115871645' is 'bad'
DirXML: [10/02/08 14:01:10.763]: TRACE:  ParseIDoc: File 'O_400_00000000115871645.proc' renamed to '/idm/idocs/O_400_00000000115871645.bad' successfully.

As I was learning about iDOCs I tried editing one myself, to cut it into pieces and make it smaller, into a single event I could follow and troubleshoot. Well looks like I made a typo.

The formatting is VERY constrained and non tolerant of errors. Extra carriage returns or line feeds are a definite no no, and with everything being based on placement of the character in a very long line, you can imagine all the possible errors inadvertent editing could cause.

The good news is that vi or vim are great editors as they do not line wrap on you, munge carriage returns or line feeds, or otherwise do things that many Windows based editors will do to your file.

There are actually two different classes of this kind of error. This first one, just gets caught by the shim and treated as an error.

Then next class of error is much more fatal to the driver shim (whether it be running local or in a remote loader. Though as always, a remote loader is probably a better idea).

Fatal iDOC processing error:

This is the remote loader side trace of the event, as it reads the file, and sees an error. Finally it decides it is a fatal error.

DirXML: [10/01/09 10:49:06.017]: TRACE:  ParseIDoc: No Character Set Encoding specified.  Using default encoding: ISO8859_1
DirXML: [10/01/09 10:49:06.018]: TRACE:  ParseIDoc: IDoc file opened successfully.
DirXML: [10/01/09 10:49:06.019]: TRACE:  ParseIDoc: IDoc to parse: /idm/idocs/O_400_0000000019079301a
DirXML: [10/01/09 10:49:06.020]: TRACE:  ParseIDoc: Segment EDI_DC40
DirXML: [10/01/09 10:49:06.025]: TRACE:  ParseIDoc: Unable to read specified byte count from array.  Bad line in input file.
DirXML: [10/01/09 10:49:06.027]: TRACE:  ParseIDoc: Segment E2PLOGI
DirXML: [10/01/09 10:49:06.028]: TRACE:  ParseIDoc: Object type S found in filter.
DirXML: [10/01/09 10:49:06.028]: TRACE:  ParseIDoc: Parsing object type S segment
DirXML: [10/01/09 10:49:06.029]: TRACE:  ParseIDoc: Object identifier: 00030928
DirXML: [10/01/09 10:49:06.029]: TRACE:  ParseIDoc: Operation: I
DirXML: [10/01/09 10:49:06.030]: TRACE:  ParseIDoc: E2PITYP found
DirXML: [10/01/09 10:49:06.030]: TRACE:  ParseIDoc: Parsing infotype: 1000, subtype:     
DirXML: [10/01/09 10:49:06.031]: TRACE:  ParseIDoc: GSA segment 'E2P1000001'
DirXML: [10/01/09 10:49:06.032]: TRACE:  ParseIDoc: Skipping history item - Type: P1000, timestamp: 20050418-20050801
DirXML: [10/01/09 10:49:06.033]: TRACE:  ParseIDoc: E2PITYP found
DirXML: [10/01/09 10:49:06.033]: TRACE:  ParseIDoc: Parsing infotype: 1001, subtype: A003
DirXML: [10/01/09 10:49:06.035]: TRACE:  ParseIDoc: GSA segment 'E2P1001001'
DirXML: [10/01/09 10:49:06.035]: TRACE:  ParseIDoc: Skipping history item - Type: P1001, timestamp: 20050418-20050801
DirXML: [10/01/09 10:49:06.036]: TRACE:  ParseIDoc: Unable to read specified byte count from array.  Bad line in input file.
DirXML: [10/01/09 10:49:06.037]: 
DirXML Log Event -------------------
    Driver  = \ACME-EDIR\acme\services\idm\IDMSet\SAP-HR351
    Thread  = Publisher
    Level   = error
    Message = Exception caused by PublicationShim.start()
java.lang.NullPointerException
        at com.novell.nds.dirxml.driver.SAPShim.ParseIDoc.startparse(ParseIDoc.java(Compiled Code))
        at com.novell.nds.dirxml.driver.SAPShim.SAPPublicationShim.getOutboundIDoc(SAPPublicationShim.java(Compiled Code))
        at com.novell.nds.dirxml.driver.SAPShim.SAPPublicationShim.start(SAPPublicationShim.java(Compiled Code))
        at com.novell.nds.dirxml.remote.loader.Driver.run(Driver.java:851)
        at java.lang.Thread.run(Thread.java:570)
DirXML: [10/01/09 10:49:06.049]: 
DirXML Log Event -------------------
    Driver  = \ACME-EDIR\acme\services\idm\IDMSet\SAP-HR351
    Thread  = Publisher
    Level   = fatal
    Message = Exception caused by PublicationShim.start()
java.lang.NullPointerException
        at com.novell.nds.dirxml.driver.SAPShim.ParseIDoc.startparse(ParseIDoc.java(Compiled Code))
        at com.novell.nds.dirxml.driver.SAPShim.SAPPublicationShim.getOutboundIDoc(SAPPublicationShim.java(Compiled Code))
        at com.novell.nds.dirxml.driver.SAPShim.SAPPublicationShim.start(SAPPublicationShim.java(Compiled Code))
        at com.novell.nds.dirxml.remote.loader.Driver.run(Driver.java:851)
        at java.lang.Thread.run(Thread.java:570)

You can see the Log Event, that it is a Fatal type event, meaning the driver will shut down.

Then you see the driver shut down event. Nice that it ended so gracefully in this case.

DirXML: [10/01/09 10:49:06.068]: TRACE:  <nds dtdversion="3.5" ndsversion="8.x">
        <input>
                <status event-id="report status" level="warning" type="remoteloader">Remote driver stopped</status>
                <init-params event-id="write-state"/>
        </input>
</nds>

Watching trace on the engine side shows, the following error, basically the same error the Remote Loader reported, bubbled back to the engine side.

This is useful, as often you may not have easy access to the Remote Loader side to watch the trace, nor might you be running it with trace enabled, as it can eat up disk space quickly, and reduce performance.

[10/01/09 10:49:06.061]:SAP-HR351 :Remote Interface Driver: Received.
[10/01/09 10:49:06.061]:SAP-HR351 :
<nds dtdversion="3.5" ndsversion="8.x">
  <input>
    <status event-id="report status" level="fatal" type="remoteloader">Exception caused by PublicationShim.start()
java.lang.NullPointerException
        at com.novell.nds.dirxml.driver.SAPShim.ParseIDoc.startparse(ParseIDoc.java(Compiled Code))
        at com.novell.nds.dirxml.driver.SAPShim.SAPPublicationShim.getOutboundIDoc(SAPPublicationShim.java(Compiled Code))
        at com.novell.nds.dirxml.driver.SAPShim.SAPPublicationShim.start(SAPPublicationShim.java(Compiled Code))
        at com.novell.nds.dirxml.remote.loader.Driver.run(Driver.java:851)
        at java.lang.Thread.run(Thread.java:570)
</status>
    <init-params event-id="write-state"/>
  </input>
</nds>

In the first case of a bad iDOC file, I no longer remember what and how I edited the file to cause the non fatal error, that was nicely handled by the shim. In the case of the fatal error, I am pretty sure all I did was leave a trailing empty line, carriage return in VI. Sort of like when you paste in, and the cursor does not have a ~ on the last line, rather there is a blank line in the view in VI. A simple dd to delete the line clears it up, but leaving it behind seems to have caused the issue.

The actual error, on the Remote Loader side that seems to get us in trouble here is: Unable to read specified byte count from array. Bad line in input file.

I wonder if this is a bug versus normal behavior, as it would seem this would be an excellent case, where the shim should quietly handle it as an error, rename the file to .bad as it does above in the previous example and get on with its life. But for some reason, this specific error is fatal. I don't really have the time to report it as a bug, but it leaves me wondering.

Bad Password for account in SAP:

<nds dtdversion="1.0" ndsversion="8.5">
  <source>
    <product build="20070918_0739 " instance="SAP-HR351" version="3.5.1">DirXML Driver for SAP/HR</product>
    <contact>Novell, Inc.</contact>
  </source>
  <output>
    <status level="fatal" type="app-authentication">
      <description>Error authenticating to SAP host: RFC_ERROR_LOGON_FAILURE</description>
    </status>
    <init-params event-id="write-state"/>
  </output>
</nds>
[10/08/08 12:58:47.684]:SAP-HR351 PT:Applying schema mapping policies to input.
[10/08/08 12:58:47.685]:SAP-HR351 PT:Applying policy: %+C%14CMapping+Policy%-C.
[10/08/08 12:58:47.682]:SAP-HR351 PT:Resolving association references.
[10/08/08 12:58:47.687]:SAP-HR351 PT:
DirXML Log Event -------------------
     Driver:   \ACME-DEV\acme\services\idm\IDMSet\SAP-HR351
     Channel:  Publisher
     Status:   Fatal
     Message:  <description>Error authenticating to SAP host: RFC_ERROR_LOGON_FAILURE</description>

This is a nice clear error. Logon Failure. Yay. The best kind. The password was wrong for the account in SAP, used on the Subscriber channel for the BAPI calls. This happened to me, as they refreshed our QC environment and lost my SAP account, recreated it with a different password. I am curious, is this normal for SAP people to do? They did this to me about 4 times over the course of the project. It has to be the most annoying thing to come on site and find out that nothing is working. Takes about half a day to find out what they changed without mentioning it to me, and then to fix it.

Oh, did we not tell you we refreshed that machine, and upgraded it, and threw away any changes we made for your lab? Do I sound bitter? It truly was annoying, especially when it kept happening!

Anyway, easy enough to detect and figure out once you know what to look for.

I think that is enough for now, stay tuned for part 2 where we tackle a bunch more errors that may help you out when deploying this driver in your environment.

As always, I highly recommend that you try to do this sort of article yourself and publish any error codes you might find. The more we get into Google the easier it will be for others searching for help on the topic to help themselves!

Cool Solutions

Free GroupWise Interactive Quick Start Cards

Messaging Architects — Thu, 19 Nov 2009 17:37:35 -0700

Go Green with GroupWise Quick Start Cards from Messaging Architects

Fewer printed materials mean a greener environment for all of us. We at Messaging Architects had an overwhemling number of requests for our GroupWise Quick Start cards so, as part of our initiative to reduce our environmental footprint, we're making these handy interactive reference cards available online for free.

Versions Available:

Key GroupWise 8 Topics Include:

Basic Navigation
Managing Contacts
Creating Recurrng Appointments, Tasks, and Notes
Sharing Folders
Granting Proxy Access
… and more!

Download your GroupWise QuickStart Cards today. They're good for your users – and the environment!

Cool Solutions

New: Merchandise Transfer Form in the Teaming Library

Teaming_Library — Thu, 19 Nov 2009 17:33:18 -0700

New Form in the Teaming Library: Merchandise Transfer

The Library is excited to add a form that will not only aid efficiency but add security control to your business process. The Merchandise Transfer form is a simple way to request, record, and respond to transfer of goods inside of your organization.

To Download the form visit the Library by clicking HERE.

Cool Solutions

Webinar: Upgrading to GroupWise 8 Tips & Tricks

Messaging Architects — Thu, 19 Nov 2009 17:07:42 -0700

Start: 1 Dec 2009 - 3:00pm

Timezone: US/Eastern

GroupWise 8 offers a superior end-user experience and cool, new collaboration features. So, let's get that migration started!

Is your organization planning on upgrading to GW8? Do you have questions or concerns? Do you want to learn some tricks for upgrading and how best to prepare your system? Join this Webinar from Messaging Architects' GroupWise Best Practices Webinar Series. You'll learn about tested best practices and get other valuable tips to consider when you upgrade to GroupWise 8. GroupWise Expert Gregg A. Hinchman will answer specific questions related to your messaging environment and help to ensure your upgrade is as smooth as possible.

Register Today!
Date: Tuesday, December 1, 2009
Time: 3:00 pm Eastern Standard Time (New York, GMT-05:00)

Duration: 1 hour

Cool Solutions

Hosting — It's an Issue of Trust

Messaging Architects — Thu, 19 Nov 2009 17:01:21 -0700

Thoughts on Eating Our Own Caviar … and Boomerang Mail Servers

For many IT departments, the decision to go or not go with hosted email hinges on trust. Can the organization trust the provider to give the system the same care an attention that internal IT people would? I didn't want Messaging Architects to offer a hosted email service unless I could be confident the answer would be, "Yes — and more."

The first step in that direction was for Messaging Architects to move our own GroupWise system onto the same virtualized infrastructure in the same datacenter (Rackspace) that we would be providing to our hosted clients. I am pleased to announce that this transition is done and we are running all our systems on a custom-designed, fully virtualized server stack at Rackspace. We learned a ton in the process, and these lessons have been integrated into a smart, simple, and secure migration methodology. The goal is to make it easy for clients to move from a physical onsite infrastructure to a virtualized and hosted one.

Contrary to other hosted email providers, we decided that each client would get their own 100%-dedicated, virtualized server stack. Unlike other hosting providers that mix your email with that of hundreds of other companies on mega servers, we provide each client a fully isolated server. This service is not meant for small companies (we start at 350 users and up) and it is certainly a more expensive approach for us, but we think it is the right way to build a flexible service.

I always had a problem with providers that hold your data hostage and make it difficult or impossible to get it back. We chose to make it very easy to take your servers back in-house. Since each client has their own set of virtual machines, we can return the entire virtual framework — data and apps — back over to your organization in less than 24 hours. Just like you hope your adult children will never move back in, we hope you’ll never want your servers back … but if you do, then we make it both possible and painless.

Our unique approach also means the system can still leverage your organization's rich, on-site infrastructure such as eDirectory and Identity Manager (enabling provisioning, single sign-on, etc.). There's also transparent integration with enterprise apps that either remain onsite (CRM, HR) or might be hosted elsewhere (Salesforce). None of this is possible with other hosted offerings.

Of course, with Rackspace as a partner, we have access to world-class, 24 x 7 protection against power failure and equipment malfunction, as well as built-in redundancies on servers and network connectivity. Our custom engineering is designed to spread the virtual stack across multiple physical servers, each with point-in-time backup and restore. All data transmissions are over secure HTTPS connections.

We are accepting our first round of clients now, and we will go big in early 2010. The service includes GroupWise 8 and the combination of M+Archive and M+Guardian. The result is a risk-free, zero-effort email infrastructure that can grow with your organization — and perhaps even move back in one day ;-).

— Frederic Bourget, Messaging Architects VP of Technology & Solutions

Cool Solutions

Webinar: GWAVA Reload 3.1 - Save Time with Reload

GWAVAComm — Thu, 19 Nov 2009 15:05:41 -0700

Start: 10 Dec 2009 - 3:00pm

End: 10 Dec 2009 - 4:00pm

Timezone: US/Eastern

GWAVA is pleased to invite you to a special 1 hour webinar on December 10th, 2009 at 3pm EST. Stephen Cohen will demonstrate step-by-step how the new migration features in Reload can save you time.

If you are thinking of migrating your GroupWise post office to Linux, you don't want to miss this free webinar!

Presenter: Stephen Cohen, GWAVA Systems Engineer

To sign up, click HERE.

Workgroup Cool Solutions

GWAVACon 2010 Call for Speakers

GWAVAComm — Thu, 19 Nov 2009 15:00:04 -0700

GWAVACon 2010 is shaping up to be an exciting event with lots of interesting technical sessions and keynotes. There are still a few session slots available. Let us know if you would like to speak or if you know of a session topic or speaker that would interest GWAVACon attendees.

Contact gregj@gwava.com if you would like to participate.

GWAVACon is the Novell GroupWise technology conference where you can enjoy face-to-face interaction with the top experts on GroupWise and other Novell technology. GWAVACon will be held January 24-26, 2010 in Las Vegas, Nevada.

For more information on GWAVACon or to register, go to www.gwavacon.com.

Cool Solutions