Objective Sharpie

Automated Binding Definitions using Objective Sharpie

This article provides an introduction to Objective Sharpie, Xamarin's command line tool used to automate the process of creating a binding to a Objective-C Library

Overview

Objective Sharpie is a command line tool to help bootstrap the first pass of a binding. It works by parsing the header files of a native library to map the public API into the binding definition (a process that previously was manually done).

Objective Sharpie uses Clang parse header files, so the binding is as exact and thorough as possible. This can greatly reduce the time and effort it takes to produce a quality binding.

WARNING: Objective Sharpie is a tool for experienced Xamarin developers with advanced knowledge of Objective-C (and by extension, C). Before attempting to bind an Objective-C library you should have solid knowledge of how to build the native library on the command line (and a good understanding of how the native library works).

History

We have been evolving and using the Objective Sharpie internally at Xamarin for the last two years. As a testament to the power of Objective Sharpie, APIs introduced in iOS 8 and Mac OS X 10.10 were bootstrapped entirely with Objective Sharpie.

Objective Sharpie is a very advanced tool that requires advanced knowledge of Objective-C and C, how to use the clang compiler on the command line, and generally how native libraries are put together. Because of this high bar, we felt that having a GUI wizard sets the wrong expectations, and as such, Objective Sharpie is only available as a command line tool.

Installing Objective Sharpie

Objective Sharpie is currently a standalone command line tool for Mac OS X 10.9 and newer, and is not a fully supported Xamarin product. It should only be used by advanced developers to assist in creating a binding project to a 3rd party Objective-C library.

Objective Sharpie can be downloaded as a standard OS X package installer. Run the installer and follow all of the on-screen prompts from the installation wizard:

Tip: use the sharpie update command to update to the latest version.

Basic Walkthrough

Objective Sharpie is a command line tool provided by Xamarin that assists in creating the definitions required to bind a 3rd party Objective-C library to C#. Even when using Objective Sharpie, the developer will need to modify the generated files after Objective Sharpie finishes to address any issues that could not be automatically handled by the tool.

Where possible, Objective Sharpie will annotate APIs with which it has some doubt on how to properly bind (many constructs in the native code are ambiguous). These annotations will appear as [Verify] attributes.

In this section, we'll take a brief look at using Objective Sharpie to create the initial ApiDefinition.cs and StructsAndEnums.cs for an Objective-C library.

Objective Sharpie comes with one major rule for proper usage: you must absolutely pass it the correct clang compiler command line arguments in order to ensure proper parsing. This is because the Objective Sharpie parsing phase is simply a tool implemented against the clang libtooling API.

This means that Objective Sharpie has the full power of Clang (the C/Objective-C/C++ compiler that actually compiles the native library you would bind) and all of its internal knowledge of the header files for binding. Instead of translating the parsed AST to object code, Objective Sharpie translates the AST to a C# binding "scaffold" suitable for input to the bmac and btouch Xamarin binding tools.

If Objective Sharpie errors out during parsing, it means that clang errored out during its parsing phase trying to construct the AST, and you need to figure out why.

Unfortunately that's not always an easy task. The quick advice is to always build the native library you are trying to bind, and then build a simple native program against that native library. The compiler flags you use to build the simple native program are what (in some form) you need to pass to Objective Sharpie. Often you can simply run xcodebuild and discern what's required. However, this is a manual process, and one that does require a bit of familiarity with compiling native code on the command line.

After we have Objective Sharpie successfully installed, let's start the Terminal app and enter the following command to get help on all of the tools that it provides to assist in binding:

sharpie -help

If we execute the above command, the following output will be generated:

$ sharpie -help
usage: sharpie [OPTIONS] TOOL [TOOL_OPTIONS]

Options:
  -h, -help                Show detailed help
  -v, -version             Show version information

Telemetry Options:
  -tlm-about               Show a detailed overview of what usage and binding
                             information will be submitted to Xamarin by
                             default when using Objective Sharpie.
  -tlm-do-not-submit       Do not submit any usage or binding information to
                             Xamarin. Run 'sharpie -tml-about' for more
                             information.
  -tlm-do-not-identify     Do not submit Xamarin account information when
                             submitting usage or binding information to Xamarin
                             for analysis. Binding attempts and usage data will
                             be submitted anonymously if this option is
                             specified.

Available Tools:
  xcode         Get information about Xcode installations and available SDKs.
  bind          Create a Xamarin C# binding to Objective-C APIs
  update        Update to the latest release of Objective Sharpie
  verify-docs   Show cross reference documentation for [Verify] attributes
  docs          Open the Objective Sharpie online documentation

Objective Sharpie provides the following tools:

ToolDescription
xcodeprovides information about the current Xcode installation and the versions of iOS and Mac SDKs that are have available. We will be using this information later when we generate our bindings.
bindparses the header files (*.h) in the Objective-C library into the initial ApiDefinition.cs and StructsAndEnums.cs files.
updatechecks for newer versions of Objective Sharpie and downloads and launches the installer if one is available
verify-docsshows detailed information about [Verify] attributes.
docsnavigates to this document in your default web browser.

To get help on a specific Objective Sharpie tool, enter the name of the tool and the -help option. For example, sharpie xcode -help returns the following output:

$ sharpie xcode -help
usage: sharpie xcode [OPTIONS]

Options:
  -h, -help        Show detailed help
  -v, -verbose     Be verbose with output

Xcode Options:
  -sdks            List all available Xcode SDKs. Pass -verbose for more
                     details.

Before we can start the binding process, we need to get information about our current installed SDKs by entering the following command into the Terminal sharpie xcode -sdks -verbose. Your output may differ depending on which version(s) of Xcode you have installed. Objective Sharpie looks for SDKs installed in any Xcode*.app under the /Applications directory:

$ sharpie xcode -sdks
sdk: iphoneos8.1    arch: arm64   armv7
sdk: iphoneos7.1    arch: arm64   armv7
sdk: macosx10.10    arch: x86_64  i386
sdk: macosx10.9     arch: x86_64  i386

From the above, we can see that we have the iphoneos8.1 SDK installed on our machine and it has arm64 architecture support. We will be using these values for the rest of this document. With this information in place, we are ready to parse the Objective-C library header files into the initial ApiDefinition.cs and StructsAndEnums.cs for the Binding project.

Read through the header files for the target project to find the "top-level" header file to use with Objective Sharpie. Well designed libraries make this pretty easy, for example: LibFoo will probably have a LibFoo.h that has #import or #include directives to bring the rest of the library API into scope for parsing. Therefore only LibFoo.h should need to be specified. If the output from Objective Sharpie isn't what you expect, re-examine the header files and include only what you need to input to the tool.

Real-World Example

This example uses the POP library from Facebook as found on GitHub. Before using Objective Sharpie, we must first clone and build the library. Building the library before binding is typically a useful and important step before binding (although you may not always be able to build the library, especially in the case of a proprietary binary). Always read through any documentation that may be associated with the library regarding building, compiling, or linking against the library.

$ git clone https://github.com/facebook/pop.git
Cloning into 'pop'...
remote: Counting objects: 908, done.
remote: Total 908 (delta 0), reused 0 (delta 0)
Receiving objects: 100% (908/908), 427.35 KiB | 0 bytes/s, done.
Resolving deltas: 100% (588/588), done.
Checking connectivity... done.

$ cd pop

Because the POP library has an Xcode project (pop.xcodeproj), we can just use xcodebuild to build POP. This process may in turn generate header files that Objective Sharpie may need to parse. This is why building before binding is important. When building via xcodebuild ensure you pass the same SDK identifier and architecture that you intend to pass to Objective Sharpie:

$ xcodebuild -sdk iphoneos8.1 -arch arm64

Build settings from command line:
    ARCHS = arm64
    SDKROOT = iphoneos8.1

=== BUILD TARGET pop OF PROJECT pop WITH THE DEFAULT CONFIGURATION (Release) ===

...

CpHeader pop/POPAnimationTracer.h build/Headers/POP/POPAnimationTracer.h
    cd /Users/aaron/src/sharpie/pop
    export PATH="/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/usr/bin:/Applications/Xcode.app/Contents/Developer/usr/bin:/Users/aaron/bin::/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin:/usr/local/git/bin:/Users/aaron/.rvm/bin"
    builtin-copy -exclude .DS_Store -exclude CVS -exclude .svn -exclude .git -exclude .hg -strip-debug-symbols -strip-tool /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/strip -resolve-src-symlinks /Users/aaron/src/sharpie/pop/pop/POPAnimationTracer.h /Users/aaron/src/sharpie/pop/build/Headers/POP

...

** BUILD SUCCEEDED **

There will be a lot of build information output in the console as part of xcodebuild. As displayed above, we can see that a "CpHeader" target was run wherein header files were copied to a build output directory. This is often the case, and makes binding easier: as part of the native library's build, header files are often copied into a "publicly" consumable location which can make parsing easier for binding. In this case, we know that POP's header files are in the build/Headers directory.

We are now ready to bind POP. We know that we want to build for SDK iphoneos8.1 with the arm64 architecture, and that the header files we care about are in build/Headers under the POP git checkout. If we look in the build/Headers directory, we'll see a number of header files:

$ ls build/Headers/POP/*
POP.h                    POPAnimationTracer.h     POPDefines.h
POPAnimatableProperty.h  POPAnimator.h            POPGeometry.h
POPAnimation.h           POPAnimatorPrivate.h     POPLayerExtras.h
POPAnimationEvent.h      POPBasicAnimation.h      POPPropertyAnimation.h
POPAnimationExtras.h     POPCustomAnimation.h     POPSpringAnimation.h
POPAnimationPrivate.h    POPDecayAnimation.h

If we look at POP.h, we can see it is the library's main top-level header file that #imports other files. Because of this, we only need to pass POP.h to Objective Sharpie, and clang will do the rest behind the scenes:

$ sharpie bind -output Binding -sdk iphoneos8.1 \
    -scope build/Headers build/Headers/POP/POP.h \
    -c -Ibuild/Headers -arch arm64

Parsing Native Code...

Binding...
  [write] ApiDefinitions.cs
  [write] StructsAndEnums.cs

Binding Analysis:
  Automated binding is complete, but there are a few APIs which have
  been flagged with [Verify] attributes. While the entire binding
  should be audited for best API design practices, look more closely
  at APIs with the following Verify attribute hints:

  ConstantsInterfaceAssociation (1 instance):
    There's no fool-proof way to determine with which Objective-C
    interface an extern variable declaration may be associated.
    Instances of these are bound as [Field] properties in a partial
    interface into a near-by concrete interface to produce a more
    intuitive API, possibly eliminating the 'Constants' interface
    altogether.

  StronglyTypedNSArray (4 instances):
    A native NSArray* was bound as NSObject[]. It might be possible
    to more strongly type the array in the binding based on
    expectations set through API documentation (e.g. comments in the
    header file) or by examining the array contents through testing.
    For example, an NSArray* containing only NSNumber* instances can
    be bound as NSNumber[] instead of NSObject[].

  MethodToProperty (2 instances):
    An Objective-C method was bound as a C# property due to
    convention such as taking no parameters and returning a value (
    non-void return). Often methods like these should be bound as
    properties to surface a nicer API, but sometimes false-positives
    can occur and the binding should actually be a method.

  Once you have verified a Verify attribute, you should remove it
  from the binding source code. The presence of Verify attributes
  intentionally cause build failures.

  For more information about the Verify attribute hints above,
  consult the Objective Sharpie documentation by running 'sharpie
  docs' or visiting the following URL:

    http://xmn.io/sharpie-docs

Submitting usage data to Xamarin...
  Submitted - thank you for helping to improve Objective Sharpie!

Done.

You will notice that we passed a -scope build/Headers argument to Objective Sharpie. Because C and Objective-C libraries must #import or #include other header files that are implementation details of the library and not API you wish to bind, the -scope argument tells Objective Sharpie to ignore any API that is not defined in a file somewhere within the -scope directory.

You will find the -scope argument is often optional for cleanly implemented libraries, however there is no harm in explicitly providing it.

Additionally, we specified -c -Ibuild/headers. Firstly, the -c argument tells Objective Sharpie to stop interpreting command line arguments and pass any subsequent arguments directly to the clang compiler. Therefore, -Ibuild/Headers is a clang compiler argument that instructs clang to search for includes under build/Headers, which is where the POP headers live. Without this argument, clang would not know where to locate the files that POP.h is #importing. Almost all "issues" with using Objective Sharpie boil down to figuring out what to pass to clang.

Completing the Binding

Objective Sharpie has now generated Binding/ApiDefinitions.cs and Binding/StructsAndEnums.cs. These two files can now be added to a binding project in Xamarin Studio or be passed directly to the btouch or bmac tools to produce the final binding.

Objective Sharpie will provide a basic first pass at the binding, and in a few cases it might be all you need. As stated above however, the developer will usually need to manually modify the generated files after Objective Sharpie finishes to fix any issues that could not be automatically handled by the tool.

Some of the next steps include:

  • Adjusting Names: Sometimes you will want to adjust the names of methods and classes to match the .NET Framework Design Guidelines.
  • Methods or Properties: The heuristics used by Objective Sharpie sometimes will pick a method to be turned into a property. At this point, you could decide whether this is the intended behavior or not.
  • Hook up Events: You could link your classes with your delegate classes and automatically generate events for those.
  • Hook up Notifications: It is not possible to extract the API contract of notifications from the pure header files, this will require a trip to the API documentation. If you want strongly typed notifications, you will need to update the result.
  • API Curation: At this point, you might choose to provide extra constructors, Add methods (to allow for C# initialize-on-construction syntax), operator overloading and implement your own interfaces on the extra definitions file.

For a complete walkthrough of the binding process, please see our Walkthrough - Binding an Objective-C Library document.

Binding Native Frameworks

Sometimes a native library is distributed as a framework. Objective Sharpie provides a convenience feature for binding properly defined frameworks through the -framework option.

For example, binding the Adobe Creative SDK Frameworks for iOS is straightforward:

$ sharpie bind \
    -framework AdobeCreativeSDKFoundation.framework \
    -sdk iphoneos8.1

In some cases, a framework will specify an Info.plist which indicates against which SDK the framework should be compiled. If this information exists and no explicit -sdk option is passed, Objective Sharpie will infer it from the framework's Info.plist (either the DTSDKName key or a combination of the DTPlatformName and DTPlatformVersion keys).

The -framework option does not allow explicit header files to be passed. The umbrella header file is chosen by convention based on the framework name. If an umbrella header cannot be found, Objective Sharpie will not attempt to bind the framework and you must manually perform the binding by providing the correct umbrella header file(s) to parse, along with any framework arguments for clang (such as the -F framework search path option).

Under the hood, specifying -framework is just a shortcut. The following bind arguments are identical to the -framework shorthand above. Of special importance is the -F . framework search path provided to clang.

$ sharpie bind \
    -sdk iphoneos8.1 \
    AdobeCreativeSDKFoundation.framework/Headers/AdobeCreativeSDKFoundation.h \
    -scope AdobeCreativeSDKFoundation.framework/Headers \
    -c -F .

Verify Attributes

You will often find that bindings produced by Objective Sharpie will be annotated with the [Verify] attribute. These attributes indicate that you should verify that Objective Sharpie did the correct thing by comparing the binding with the original C/Objective-C declaration (which will be provided in a comment above the bound declaration).

Verification is recommended for all bound declarations, but is most likely required for declarations annotated with the [Verify] attribute. This is because in many situations, there is not enough metadata in the original native source code to infer how to best produce a binding. You may need to reference documentation or code comments inside the header files to make the best binding decision.

Once you have verified that the binding is correct or have fixed it to be correct, remove the [Verify] attribute from the binding. [Verify] attributes intentionally cause C# compilation errors so that you are forced to verify the binding.

Verify Hints Reference

The hint argument supplied to the attribute can be cross referenced with documentation below. Documentation for any produced [Verify] attributes will be provided on the console as well after the binding has completed.

You can also quickly receive documentation for a hint using the sharpie verify-docs tool (e.g. sharpie verify-docs InferredFromPreceedingTypedef).

Verify Hint Description
InferredFromPreceedingTypedef The name of this declaration was inferred by common convention from the immediately preceeding typedef in the original native source code. Verify that the inferred name is correct as this convention is ambiguous.
ConstantsInterfaceAssociation There's no fool-proof way to determine with which Objective-C interface an extern variable declaration may be associated. Instances of these are bound as [Field] properties in a partial interface into a near-by concrete interface to produce a more intuitive API, possibly eliminating the 'Constants' interface altogether.
MethodToProperty An Objective-C method was bound as a C# property due to convention such as taking no parameters and returning a value (non-void return). Often methods like these should be bound as properties to surface a nicer API, but sometimes false-positives can occur and the binding should actually be a method.
StronglyTypedNSArray A native NSArray* was bound as NSObject[]. It might be possible to more strongly type the array in the binding based on expectations set through API documentation (e.g. comments in the header file) or by examining the array contents through testing. For example, an NSArray* containing only NSNumber* instancescan be bound as NSNumber[] instead of NSObject[].

Telemetry Submission

Objective Sharpie by default submits information back to Xamarin about each binding attempt as Xamarin is very actively trying to improve the quality and reliability of Objective Sharpie.

The following data will be submitted to Xamarin for analysis on each run of the `sharpie bind' tool:

  • all command line arguments passed to sharpie bind
  • the name of the current working directory
  • the output of the xcodebuild -showBuildSettings command as run against the current working directory
  • unhandled exceptions
  • your Xamarin account information if logged in to Xamarin Studio:

No source code, libraries, or binaries will be submitted to Xamarin.

The following telemetry options can be used to disable or limit data to be sent to Xamarin for analysis. These options must be specified on each run of the sharpie bind tool. Options below should be specified before the bind tool name (e.g. sharpie -tlm-do-not-submit bind Foo.h). All data mentioned above is submitted by default to Xamarin.

-tlm-about               Show a detailed overview of what usage and binding
                         information will be submitted to Xamarin by
                         default when using Objective Sharpie.

-tlm-do-not-submit       Do not submit any usage or binding information to
                         Xamarin. Run 'sharpie -tml-about' for more
                         information.

-tlm-do-not-identify     Do not submit Xamarin account information when
                         submitting usage or binding information to Xamarin
                         for analysis. Binding attempts and usage data will
                         be submitted anonymously if this option is
                         specified.

Release History

2.1.6 (March 17, 2015)

  • Fixed binary operator expression binding: the left-hand side of the expression was incorrectly swapped with the right-hand (e.g. 1 << 0 was incorrectly bound as 0 << 1). Thanks to Adam Kemp for noticing this!
  • Fixed an issue with NSInteger and NSUInteger being bound as int and uint instead of nint and nuint on i386; -DNS_BUILD_32_LIKE_64 is now passed to Clang to make parsing objc/NSObjCRuntime.h work as expected on i386.
  • The default architecture for Mac OS X SDKs (e.g. -sdk macosx10.10) is now x86_64 instead of i386, so -arch can be omitted unless overriding the default is desired.

2.1.0 (March 15, 2015)

  • bxc#27849: Ensure using ObjCRuntime; is produced when ArgumentSemantic is used.
  • bxc#27850: Ensure using System.Runtime.InteropServices; is produced when DllImport is used.
  • bxc#27852: Default DllImport to loading symbols from __Internal.
  • bxc#27848: Skip forward-declared Objective-C container declarations.
  • bxc#27846: Bind protocol types with a single qualification as concrete interfaces (id<Foo> as Foo instead of Foundation.NSObject<Foo>).
  • bxc#28037: Bind UInt32, UInt64, and Int64 literals as Int32 to drop the u and/or uL suffixes when the values can safely fit into Int32.
  • bxc#28038: Fix enum name mapping when original native name starts with a k prefix.
  • sizeof C expressions whose argument type does not map to a C# primitive type will be evaluated in Clang and bound as an integer literal to avoid generating invalid C#.
  • Fix Objective-C syntax for properties whose type is a block (Objective-C code appears in comments above bound declarations).
  • Bind decayed types as their original type (int[] decays to int* during semantic analysis in Clang, but bind it as the original as-written int[] instead).

Thanks very much to Dave Dunkin for reporting many of the bugs fixed in this point release!

2.0.0: March 9, 2015

Objective Sharpie 2.0 is a major release that features an improved Clang-based driver and parser and a new NRefactory-based binding engine. These improved components provide for much better bindings, particularly around type binding. Many other improvements have been made that are internal to Objective Sharpie which will yield many user-visible features in future releases.

Objective Sharpie 2.0 is based on Clang 3.6.1.

Type binding improvements

  • Objective-C blocks are now supported. This includes anonymous/inline blocks and blocks named via typedef. Anonymous blocks will be bound as System.Action or System.Func delegates, while named blocks will be bound as strongly named delegate types.

  • There is an improved naming heuristic for anonymous enums that are immediately preceded by a typedef resolving to a builtin integral type such as long or int.

  • C pointers are now bound as C# unsafe pointers instead of System.IntPtr. This results in more clarity in the binding for when you may wish to turn pointer parameters into out or ref parameters. It is not possible to always infer whether a parameter should be out or ref, so the pointer is retained in the binding to allow for easier auditing.

  • An exception to the above pointer binding is when a 2-rank pointer to an Objective-C object is encountered as a parameter. In these cases, convention is predominant and the parameter will be bound as out (e.g. NSError **errorout NSError error).

Verify attribute

You will often find that bindings produced by Objective Sharpie will now be annotated with the [Verify] attribute. These attributes indicate that you should verify that Objective Sharpie did the correct thing by comparing the binding with the original C/Objective-C declaration (which will be provided in a comment above the bound declaration).

Verification is recommended for all bound declarations, but is most likely required for declarations annotated with the [Verify] attribute. This is because in many situations, there is not enough metadata in the original native source code to infer how to best produce a binding. You may need to reference documentation or code comments inside the header files to make the best binding decision.

See the Verify Attributes documentation for more details.

Other notable improvements

  • using statements are now generated based on types bound. For instance, if an NSURL was bound, a using Foundation; statement will be generated as well.

  • struct and union declarations will now be bound, using the [FieldOffset] trick for unions.

  • Enum values with constant expression initializers will now be properly bound; the full expression is translated to C#.

  • Variadic methods and blocks are now bound.

  • Frameworks are now supported via the -framework option. See the documentation on Binding Native Frameworks for more details.

  • Objective-C source code will be auto-detected now, which should eliminate the need to pass -ObjC or -xobjective-c to Clang manually.

  • Clang module usage (@import) is now auto-detected, which should eliminate the need to pass -fmodules to Clang manually for libraries which use the new module support in Clang.

  • The Xamarin Unified API is now the default binding target; use the -classic option to target the 32-bit only Classic API.

Notable bug fixes

  • Fix instancetype binding when used in an Objective-C category
  • Fully name Objective-C categories
  • Prefix Objective-C protocols with I (e.g. INSCopying instead of NSCopying)

1.1.35: December 21, 2014

Minor bug fixes.

1.1.1: December 15, 2014

1.1.1 was the first major release after 1.5 years of internal use and development at Xamarin following the initial preview of Objective Sharpie in April 2013. This release is the first to be generally considered stable and usable for a wide variety of native libraries, featuring a new Clang backend.

Summary

This article has provided a brief introduction to Objective Sharpie and show how it can be used to help automate the process of creating a binding to a 3rd party Objective-C Library.