971-270-0003 info@singlecoilinc.com

cURL – Troubleshooting Quirks and Caveats

cURL is a versatile, multi-platform command-line tool and system library for networking, which supports a wide variety of network protocols and options. Although powerful, it can also be fickle to get working properly in certain cases. Today I will discuss some quirks that I have encountered while using cURL, in hopes of saving others the time it took for me to resolve them.

Issue: Following a successfully-posted HTTP request by the client, the remote host resets the connection before it can send back a successful response.

This was a major issue I encountered while attempting to fetch some XML data via SOAP request from an NTLM authenticated server. This request looked something like the following in cURL:

curl -X POST -d @soap.xml -H 'Content-type: text/xml;charset="utf-8"' -H 'Accept: text/xml' -H 'Cache-Control: no-cache' -H 'Pragma: no-cache' -H 'SOAPAction: "http://tempuri.org/ProductService/find"' -H 'Content-length: 623' -H 'Connection: Keep-Alive' --anyauth -u myUserName:myPassword http://someurl.org/

The server would successfully process the SOAP message on the backend, but right at the moment when it should have sent back the XML response, it would instead terminate the connection with a TCP RST packet.

My cURL client would then exit with the following error message:

curl: (56) Recv failure: Connection reset by peer

Such a message is fairly ambiguous as to the exact source of the problem, which can often lie somewhere in the network or firewall configuration. In my case, however, the issue lied within cURL itself:

/* We got no data, we attempted to re-use a connection and yet we want a
        "body". This might happen if the connection was left alive when we were
        done using it before, but that was closed when we wanted to read from
        it again. Bad luck. Retry the same request on a fresh connect! */
     infof(conn->data, "Connection died, retrying a fresh connect\n");
     *url = strdup(conn->data->change.url);
     if(!*url)
       return CURLE_OUT_OF_MEMORY;

     conn->bits.close = TRUE; /* close this connection */
     conn->bits.retry = TRUE; /* mark this as a connection we're about
                                 to retry. Marking it this way should
                                 prevent i.e HTTP transfers to return
                                 error just because nothing has been
                                 transferred! */

Here cURL tried to reuse the NTLM connection, but the server took too long to respond and so cURL assumed that the connection was closed. To workaround this issue, I found that the solution is to use chunked transfer encoding, like so:

curl -X POST -d @soap.xml -H 'Content-type: text/xml;charset="utf-8"' -H 'Accept: text/xml' -H 'Cache-Control: no-cache' -H 'Pragma: no-cache' -H 'SOAPAction: "http://tempuri.org/ProductService/find"' -H 'Transfer-Encoding: chunked' --anyauth -u myUserName:myPassword http://someurl.org/

Issue: A request which uses authentication fails with “401 Unauthorized”, even though the credentials are valid.

Another problem I ran into with cURL was when trying to use NTLM authentication in an HTTP request:

curl -X POST -d @soap.xml -H 'Content-type: text/xml;charset="utf-8"' -H 'Accept: text/xml' -H 'Cache-Control: no-cache' -H 'Pragma: no-cache' -H 'SOAPAction: "http://tempuri.org/ProductService/find"' -H 'Transfer-Encoding: chunked' -H 'Connection: Keep-Alive' --anyauth -u myUserName:myPassword http://someurl.org/

Although this request worked without issue on my MacBook, it failed when I tried it from my server. I would receive continuous “401 Unauthorized” responses from the host, until the connection was closed.

To fix this, I had to explicitly tell cURL to use NTLM for its authentication method, like so:

curl -X POST -d @soap.xml -H 'Content-type: text/xml;charset="utf-8"' -H 'Accept: text/xml' -H 'Cache-Control: no-cache' -H 'Pragma: no-cache' -H 'SOAPAction: "http://tempuri.org/ProductService/find"' -H 'Transfer-Encoding: chunked' -H 'Connection: Keep-Alive' --ntlm -u myUserName:myPassword http://someurl.org/

A note on different versions of cURL:

I have had issues with versions of cURL < 7.30.0, such as the fixes described above not working at all.

Thus, if you are using cURL as a library that is integrated with a scripting language such as PHP, be sure to note which version of the library your scripting environment uses. PHP in particular has some cURL-related bugs that should be kept in mind when troubleshooting a script.

You can check the version of cURL that PHP is using via the command:

php --info | grep cURL

This should give you output which looks something like this:

cURL support => enabled

cURL Information => 7.30.0

This information can also be found in the output of PHP’s phpinfo() function.

Installing PHP with a working version of cURL on OS X Mountian Lion

Mac OS X 10.8 Mountain Lion Server includes PHP 5.3, which utilizes libcurl v7.21. As mentioned in the previous section, this version of cURL is problematic. I will provide brief instructions on how to install the Macports PHP 5.4 with libcurl 7.36. The Macports PHP is installed side-by-side with OS X server’s default PHP.

1.) If Macports is not installed, run the installer, available from http://www.macports.org/install.php

2.) Once Macports has been installed, open a Terminal window and run this command to install PHP 5.4, plus its necessary libraries:

sudo port install php54 php54-apache2handler php54-openssl php54-curl php54-pear pear-HTTP_Request2

3.) Next, setup Apache to use the newly-installed PHP library, copy over the new PHP extension (mod_php54.so), backup the original PHP extension (libphp5.so), and rename the new extension to the old extension’s name:

  1. cd /opt/local/apache2/modules
  2. sudo /opt/local/apache2/bin/apxs -a -e -n php5 mod_php54.so
  3. sudo cp mod_php54.so /usr/libexec/apache2
  4. cd /usr/libexec/apache2
  5. sudo mv libphp5.so libphp5.so.old
  6. sudo mv mod_php54.so libphp5.so

4.) Lastly, PHP needs to be configured with a timezone in order to function correctly. The PHP configuration file must be created and then modified accordingly:

  1. Copy /opt/local/etc/php54/php.ini-production to /opt/local/etc/php54/php.ini
  2. Edit the php.ini file and set the “date.timezone” value to your desired timezone. Refer to PHP’s List of Supported Timezones for more details.

Conclusion

Since many applications and libraries – PHP or otherwise – are built on top of cURL, it is useful to know at least the basics of how this software functions. When troubleshooting or debugging a program, a seemingly higher-level network error could turn out to be an issue with the underlying cURL library, and so it is often helpful to test out specific network requests first on the command-line in order to eliminate other confounding factors.

I hope this post proved useful and informative, and if you have any other cURL caveats and their solutions that you want to discuss, go ahead and share your knowledge in the comments section below.

Quickly regenerate Default Data Import Export Framework (DIXF) Target Entity Mapping in Microsoft Dynamics AX 2012

As I have been heavily involved at a Customer’s Data Migration the last few months, I have become intimately familiar with the new Microsoft Dynamics AX 2012 Data Import Export Framework (DIXF), formerly known as the Data Migration Framework (DMF). While it is not a perfect tool, it has been tremendously helpful in migrating data and I highly recommend anyone working through data migration into AX 2012 to fully take advantage of the DIXF.

Today I want to talk about a quick way to regenerate all Target Mapping to the default AX values.

Out of the box the only possible way to regenerate the Target Mappings is to navigate to
Data import export framework –> Setup –> Target Entities

DMFSetup

Select a Specific Entity –> Modify Target Mapping

DMFModifyTargetMapping

Mapping Details –> Generate Mapping

this can get very time consuming for all the (to date) 147 entities, not to mention any custom entities created.  Also, sometimes it is not possible to regenerate the target mapping for a particular entity as clicking on “Modify target mapping” generates an error.   This error typically reads along the lines of “Duplicate assignment found for the following target field <some field name>”

DMFDuplicateAssignment

A way to get around this is via an X++ job that generates all the entity associations using a built in method of the DMFTargetXMLToEntityMap table.

Below is a quick and simple job that deletes the complete mapping and regenerates them based on the dynamic association in AX.  The job can be modified to also work for only specific entities by including a where clause.

Regenerate All DMF Target Mapping

static void regenerateDMFTargetMapping(Args _args)
{
    DMFTargetXMLToEntityMap     dmfTargetXMLToEntityMap;
    DMFEntity                   dmfEntity;

    // delete all target Mappings
    ttsBegin;
        delete_from dmfTargetXMLToEntityMap;
    ttsCommit;

    // Regenerate Target Mapping for all Entities
    while select dmfEntity
    {
        DMFTargetXMLToEntityMap::generateMapping(dmfEntity);
    }

    info("All Done!");
}

Regenerate Specific Target Mapping

static void regenerateDMFTargetMapping(Args _args)
{
    DMFTargetXMLToEntityMap     dmfTargetXMLToEntityMap;
    DMFEntity                   dmfEntity;

    // delete specified target Mapping
    ttsBegin;
        delete_from dmfTargetXMLToEntityMap where dmfTargetXMLToEntityMap.Entity = <EntityName>;
    ttsCommit;

    // Regenerate Target Mapping for specified Entity
    while select dmfEntity where dmfEntity.EntityName = <EntityName>
    {
        DMFTargetXMLToEntityMap::generateMapping(dmfEntity);
    }

    info("All Done!");
}

That’s it for today’s post, hope this helps in your Microsoft Dynamics AX 2012 Data Migration efforts!

Adding a youtube video to an iOS app in 5 easy steps.

Adding a youtube video to an iPhone or iPad should be easy–and it is!  The most difficult part in finding how to do this is sorting through the search results to find what is the current recommended approach and what has been deprecated.  Searching for “embed youtube in iOS app” may lead you to this page which starts with a warning that the information is out-of-date but also points you to this page which contains the information you want but is a bit TL; DR. Here’s the simple solution for iOS 6 & 7:

1. Add an HTML file to your iOS project, making sure it is also added to the desired target. Here is the HTML (in the iFrame link, be sure to replace YOUR_VIDEO_ID with your video ID from youtube.):

<!DOCTYPE html>
<html>
<head>
<style>
body{
margin: 0 0 0 0;
background-color: #000;
}

iframe{
position:absolute;
height: 100%;
width: 100%;
background-color: #000;
}
</style>
</head>
<body>
<iframe id="player" type="text/html"   
src="http://www.youtube.com/embed/YOUR_VIDEO_ID?enablejsapi=1&fs=1&playsinline=0&rel=0&autoplay=1&origin=http://www.example.com"
frameborder="0"></iframe>

<script>
// 1. This code loads the IFrame Player API code asynchronously.
var tag = document.createElement('script');
tag.src = "https://www.youtube.com/iframe_api";
var firstScriptTag = document.getElementsByTagName('script')[0];
firstScriptTag.parentNode.insertBefore(tag, firstScriptTag);

// 2. This function creates an <iframe> (and YouTube player)
//    after the API code downloads.
//http://youtu.be/rzl9v_u43g4
var player;
function onYouTubeIframeAPIReady() {
  player = new YT.Player('player');
}

</script>
</body>
</html>

2. Drag a UIWebView to your storyboard. For iPad, you can make the UIWebView the size you want the video to be. For iPhone/iPod, the video will play full screen automatically, so the UIWebView will only need to be as big as the clickable area you want to start the video.

3. In your view controller, add a property for the web view.

@property (weak, atomic) IBOutlet UIWebView* videoView;

4. In the storyboard, link the UIWebView to the property.

5. In the view controller’s .m file, load your local HTML into the web view:

-(void)viewDidAppear:(BOOL)animated
{
    [self.videoView loadRequest:[NSURLRequest 
                                requestWithURL:[NSURL fileURLWithPath:[[NSBundle mainBundle]         
                                                                        pathForResource:@"YOUR_LOCAL_HTML_FILE_NAME" 
                                                                        ofType:@"html"]
                                                      isDirectory:NO]]];
}

That should do it.

Bonus section: To make the experience better, there are things that can be done in the app as well as through youtube’s API.

1. Detect when the video finishes. When a video is done playing, a notification is broadcasted.  If you listen for it, you’ll know what to do.  Add the following to the view controller’s .m file:

//in the viewDidLoad event
    [[NSNotificationCenter defaultCenter] addObserver:self
                                             selector:@selector(playerWillExitFullscreen:)
                                                 name:@"UIMoviePlayerControllerWillExitFullscreenNotification"
                                               object:nil];
...
//add this to respond to the notification:
- (void)playerWillExitFullscreen:(NSNotification *)notification
{
    //code to run when video is done.
}

2. Add parameters to youtube’s player. The interface youtube uses to play the video is the HTML5 player. There are many parameters you can use to customize controls, when the video starts, when it ends, how long it plays, etc. These parameters can be found here. They can be added to the src tag in the iFrame of your local HTML file (they are & separated.) 3. Use youtube’s javascript API to detect other events. The HTML supplied above automatically download’s youtube’s javascript library for detecting events.  This link shows how to listen for them.