This article has multiple issues. Please help improve it or discuss these issues on the talk page. (Learn how and when to remove these template messages) This article may rely excessively on sources too closely associated with the subject, potentially preventing the article from being verifiable and neutral. Please help improve it by replacing them with more appropriate citations to reliable, independent, third-party sources. (August 2015) (Learn how and when to remove this template message) This article relies largely or entirely on a single source. Relevant discussion may be found on the talk page. Please help improve this article by introducing citations to additional sources.Find sources: "OpenHMPP" – news · newspapers · books · scholar · JSTOR (June 2014) This article may contain an excessive amount of intricate detail that may interest only a particular audience. Please help by spinning off or relocating any relevant information, and removing excessive detail that may be against Wikipedia's inclusion policy. (June 2014) (Learn how and when to remove this template message) (Learn how and when to remove this template message)

OpenHMPP (HMPP[1] for Hybrid Multicore Parallel Programming) - programming standard for heterogeneous computing. Based on a set of compiler directives, standard is a programming model designed to handle hardware accelerators without the complexity associated with GPU programming. This approach based on directives has been implemented because they enable a loose relationship between an application code and the use of a hardware accelerator (HWA).


The OpenHMPP directive-based programming model offers a syntax to offload computations on hardware accelerators and to optimize data movement to/from the hardware memory.

The model is based on works initialized by CAPS (Compiler and Architecture for Embedded and Superscalar Processors), a common project from INRIA, CNRS, the University of Rennes 1 and the INSA of Rennes.

OpenHMPP concept

OpenHMPP is based on the concept of codelets, functions that can be remotely executed on HWAs.

The OpenHMPP codelet concept

A codelet has the following properties:

  1. It is a pure function.
    • It does not contain static or volatile variable declarations nor refer to any global variables except if these have been declared by a HMPP directive “resident”
    • It does not contain any function calls with an invisible body (that cannot be inlined). This includes the use of libraries and system functions such as malloc, printf, ...
    • Every function call must refer to a static pure function (no function pointers).
  2. It does not return any value (void function in C or a subroutine in Fortran).
  3. The number of arguments should be fixed (i.e. it can not be a variadic function as in stdarg.h in C).
  4. It is not recursive.
  5. Its parameters are assumed to be non-aliased (see Aliasing (computing) and Pointer aliasing).
  6. It does not contain callsite directives (i.e. RPC to another codelet) or other HMPP directives.

These properties ensure that a codelet RPC can be remotely executed by a HWA. This RPC and its associated data transfers can be asynchronous.

Codelet RPCs

HMPP provides synchronous and asynchronous RPC. Implementation of asynchronous operation is hardware dependent.

Synchronous versus asynchronous RPC
Synchronous versus asynchronous RPC

HMPP Memory Model

HMPP considers two address spaces: the host processor one and the HWA memory.

HMPPP memory Model
HMPPP memory Model

Directives concept

The OpenHMPP directives may be seen as “meta-information” added in the application source code. They are safe meta-information i.e. they do not change the original code behavior. They address the remote execution (RPC) of a function as well as the transfers of data to/from the HWA memory.

The table below introduces the OpenHMPP directives. OpenHMPP directives address different needs: some of them are dedicated to declarations and others are dedicated to the management of the execution.

Control flow instructions Directives for data management
Declarations codelet
Operational Directives callsite

Concept of set of directives

One of the fundamental points of the HMPP approach is the concept of directives and their associated labels which makes it possible to expose a coherent structure on a whole set of directives disseminated in an application.

There are two kinds of labels:

OpenHMPP Directives Syntax

In order to simplify the notations, regular expressions will be used to describe the syntax of the HMPP directives.

The color convention below is used for the description of syntax directives:

General syntax

The general syntax of OpenHMPP directives is:

#pragma hmpp <grp_label> [codelet_label]? directive_type [,directive_parameters]* [&]
!$hmpp <grp_label> [codelet_label]? directive_type [,directive_parameters]* [&]


Directive parameters

The parameters associated to a directive may be of different types. Below are the directive parameters defined in OpenHMPP:

OpenHMPP directives

Directives for declaring and executing a codelet

A codelet directive declares a computation to be remotely executed on a hardware accelerator. For the codelet directive:

The syntax of the directive is:

#pragma hmpp <grp_label> codelet_label codelet 
                            [, version = major.minor[.micro]?]?
                            [, args[arg_items].io=[[in|out|inout]]*
                            [, args[arg_items].size={dimsize[,dimsize]*}]*
                            [, args[arg_items].const=true]*
                            [, cond = "expr"]
                            [, target=target_name[:target_name]*]

More than one codelet directive can be added to a function in order to specify different uses or different execution contexts. However, there can be only one codelet directive for a given call site label.

The callsite directive specifies how the use a codelet at a given point in the program.

The syntax of the directive is:

#pragma hmpp <grp_label> codelet_label callsite
                     [, asynchronous]?
                     [, args[arg_items].size={dimsize[,dimsize]*}]*
                     [, args[arg_items].advancedload=[[true|false]]*
                     [, args[arg_items].addr="expr"]*
                     [, args[arg_items].noupdate=true]*

An example is shown here :

 /* declaration of the codelet */
 #pragma hmpp simple1 codelet, args[outv].io=inout, target=CUDA
 static void matvec(int sn, int sm, float inv[sm], float inm[sn][sm], float *outv){
     int i, j;
     for (i = 0 ; i < sm ; i++) {
       float temp = outv[i];
       for (j = 0 ; j < sn ; j++) {
         temp += inv[j] * inm[i][ j];
    outv[i] = temp;
  int main(int argc, char **argv) {
    int n;
  /* codelet use */
  #pragma hmpp simple1 callsite, args[outv].size={n}
  matvec(n, m, myinc, inm, myoutv);

In some cases, a specific management of the data throughout the application is required (CPU/GPU data movements optimization, shared variables...).

The group directive allows the declaration of a group of codelets. The parameters defined in this directive are applied to all codelets belonging to the group. The syntax of the directive is:

#pragma hmpp <grp_label> group 
                          [, version = <major>.<minor>[.<micro>]?]? 
                          [, target = target_name[:target_name]*]]? 
                          [, cond  = “expr]?

Data transfers directives to optimize communication overhead

When using a HWA, the main bottleneck is often the data transfers between the HWA and the main processor.
To limit the communication overhead, data transfers can be overlapped with successive executions of the same codelet by using the asynchronous property of the HWA.

The allocate directive locks the HWA and allocates the needed amount of memory.

#pragma hmpp <grp_label> allocate [,args[arg_items].size={dimsize[,dimsize]*}]*

The release directive specifies when to release the HWA for a group or a stand-alone codelet.

#pragma hmpp <grp_label> release

The advancedload directive prefetches data before the remote execution of the codelet.

#pragma hmpp <grp_label> [codelet_label]? advancedload

The delegatedstore directive is a synchronization barrier to wait for an asynchronous codelet execution to complete and to then download the results.

#pragma hmpp <grp_label> [codelet_label]? delegatedstore 

The synchronize directive specifies to wait until the completion of an asynchronous callsite execution. For the synchronize directive, the codelet label is always mandatory and the group label is required if the codelet belongs to a group.

#pragma hmpp <grp_label> codelet_label synchronize

In the following example, the device initialization, memory allocation and upload of the input data are done only once outside the loop and not in each iteration of the loop.

The synchronize directive allows to wait for the asynchronous execution of the codelet to complete before launching another iteration. Finally the delegatedstore directive outside the loop uploads the sgemm result.

 int main(int argc, char **argv) {
 #pragma hmpp sgemm allocate, args[vin1;vin2;vout].size={size,size}
 #pragma hmpp sgemm advancedload, args[vin1;vin2;vout], args[m,n,k,alpha,beta]
 for ( j = 0 ; j < 2 ; j ++) {
    #pragma hmpp sgemm callsite, asynchronous, args[vin1;vin2;vout].advancedload=true, args[m,n,k,alpha,beta].advancedload=true
    sgemm (size, size, size, alpha, vin1, vin2, beta, vout);
    #pragma hmpp sgemm  synchronize
 #pragma hmpp sgemm delegatedstore, args[vout]
 #pragma hmpp sgemm release

Sharing data between codelets

Those directives map together all the arguments sharing the given name for all the group.

The types and dimensions of all mapped arguments must be identical.

The map directive maps several arguments on the device.

#pragma hmpp <grp_label>  map, args[arg_items]

This directive is quite similar as the map directive except that the arguments to be mapped are directly specified by their name. The mapbyname directive is equivalent to multiple map directives.

#pragma hmpp <grp_label> mapbyname [,variableName]+

Global variable

The resident directive declares some variables as global within a group. Those variables can then be directly accessed from any codelet belonging to the group. This directive applies to the declaration statement just following it in the source code.

The syntax of this directive is:

#pragma hmpp <grp_label> resident 
               [, args[::var_name].io=[[in|out|inout]]*
               [, args[::var_name].size={dimsize[,dimsize]*}]*
               [, args[::var_name].addr="expr"]*
               [, args[::var_name].const=true]*

The notation ::var_name with the prefix ::, indicates an application's variable declared as resident.

Acceleration of regions

A region is a merge of the codelet/callsite directives. The goal is to avoid code restructuration to build the codelet. Therefore, all the attributes available for codelet or callsite directives can be used on regions directives.

In C language:

#pragma hmpp [<MyGroup>] [label] region         
                           [, args[arg_items].io=[[in|out|inout]]*
                           [, cond = "expr"]<
                           [, args[arg_items].const=true]*
                           [, target=target_name[:target_name]*]
                           [, args[arg_items].size={dimsize[,dimsize]*}]*
                           [, args[arg_items].advancedload=[[true|false]]*
                           [, args[arg_items].addr="expr"]*
                           [, args[arg_items].noupdate=true]*
                           [, asynchronous]?
                           [, private=[arg_items]]*


The OpenHMPP Open Standard is based on HMPP Version 2.3 (May 2009, CAPS entreprise).

The OpenHMPP directive-based programming model is implemented in:

OpenHMPP is used by HPC actors[who?] in Oil & Gas,[citation needed] Energy,[citation needed] Manufacturing,[citation needed] Finance,[citation needed] Education & Research.[citation needed]

See also


  1. ^ Dolbeau, Romain; Bihan, Stéphane; Bodin, François (4 October 2007). HMPP: A Hybrid Multi-core Parallel Programming Environment (PDF). Workshop on General Purpose Processing on Graphics Processing Units. Archived from the original (PDF) on 16 January 2014. Retrieved 14 January 2014.