Enduro/X is an open-sourcemiddleware platform for distributed transaction processing. It is built on proven APIs such as X/Open group's XATMI and XA. The platform is designed for building real-time microservices based applications with a clusterization option. Enduro/X functions as an extended drop-in replacement for Oracle Tuxedo. The platform uses in-memory POSIX Kernel queues which insures high interprocess communication throughput.
Services
The platform allows runtime system patching, without service interruption. The system monitors running processes, and restarts them after a crash or hang. System configuration can be dynamically reloaded and it provides a cluster facility, allowing two or more instances to be joined in peer-to-peer manner. Enduro/X is written in C and provides native APIs for C/C++. Mavimax provides bindings are provided for Go, Java programming languages. Community provides bindings for Python, PHP and Perl. Thus effectively Enduro/X may be used as language interoperability tool between supported programming languages.
Technology
Currently Enduro/X works on Linux kernels starting from 2.6.12. The most critical Linux feature is epoll() on POSIX queues. This Linux functionality provides the mechanism for one queue - multiple servers, where Enduro/X servers are polling over the queues. Enduro/X developers pushed the EPOLLEXCLUSIVE flag inclusion into Linux kernel mainline (starting from version 4.5).[1] The patch improved the Enduro/X response in case if hundreds of executables are providing the same service. Particularly kernel is not doing thundering herd type behavior anymore by waking up all binaries waiting on queue. Now only single receiver of the service call is woken. In some workloads the Enduro/X applications gets significant boost. The test case for the patch increased the performance of the workload from the 860 seconds to 24 sec.[2]
Starting with Enduro/X version 3.1.2 production grade support for FreeBSD, IBM AIX (versions 6.1 and 7.1) and Oracle Solaris V11 is added. V3.1.2 introduced experimental support for Apple macOS and Cygwin. Under Unix platforms where Posix queues are available, they are employed so that for every ATMI server/service pair there is separate queue open and caller service chooses queue in round robin mode. As for MacOS there is no support for Posix queues in operating system, Enduro/X uses emulated Posix queues which are based on memory mapped files in tandem with Posix thread's process shared mutexes. For FreeBSD Posix queues can be polled too, thus Enduro/X on FreeBSD works in the say way as Linux with one queue-multiple servers. The polling on FreeBSD is done with help of kqueue.
With version 6.0 System V queues are preferred IPC mechanisms for Unix platforms other than FreeBSD and Linux. The System V queues also gives single queue - multiple servers (SQMS) message distribution approach for effective load balancing. To load balance in SQMS mode with System V queues, XATMI server must be configured to use request address (logical channel/queue) on which all XATMI servers advertises same services. This Queue mode is available for Oracle Solaris and IBM AIX. For MacOS still emulated Posix queue are used, due to fact that it is not possible to adjust queue capacity on particular operating system.
Communication types - Synchronous, Asynchronous, Conversational, Publish/subscribe
Typed buffers
UBF (Unified Buffer Format) which provides emulation of Tuxedo's FML/FML32 format. UBF if high performance binary protocol buffer format. Buffer is indexed by binary search on fixed data types. As from version v8.0, support for recursive UBF buffers, PTR and VIEW fields has been added.
STRING buffer format.
CARRAY (byte array) buffer format.
JSON buffer format, automatic conversion between JSON and UBF available.
VIEW buffer (starting from version 5.0+). This offer C structure sending between processes in cross platform way. Also this allows to map UBF fields to VIEW fields, thus helping developer quicker to develop applications, by combining UBF and VIEW buffers.
Event broker (also called publish and subscribe messaging)
System process monitoring and self healing (pings and restarts)
SOA Service cache. XATMI services can be cached to LMDB database. Resulting that the next call to service from any local client receives results directly from cache (mainly from direct memory read).
Dynamic re-configuration
Custom server polling extensions
XATMI sub-system is able to work without main application server daemon (ndrxd)
Main application server daemon (ndrxd) can be restarted (if crashed). When started back it enters in learning mode for some period of time, in which in gathers information about system, what services are running, etc. After learning period, it starts to do normal operations
Available tpforward() call
ATMI server threads may become clients, and can do tpcall()
ATMI call timeout configuration via tptoutset() API call
Extensive logging & debugging. Enduro/X logging can be configured per binary with different log levels. As ATMI servers can be started outside of appserver, it is possible to debug them from programming IDE or with tools like valgrind.
For quality assurance project uses automated unit-testing and integration-testing
Built in ATMI service profiling.
Environment variables can be updated for XATMI server processes without full application reboot.
Generic client process monitor (cpm). Subsystem allows to start/stop/monitor client executables. At client process crashes, cpm will start it back. Massive client administration feature allows easy definition of many similar executables (with sub-section ranges) and in the same way it is very easy to manage them all together by regexp masks.
Starting with version 5.2 Enduro/X provides configuration data encryption feature, so that software which is built on top of Enduro/X may comply with Payment Card Industry Data Security Standard (PCI/DSS).
Starting with version 6.0.17 for XATMI server and client processes max resident and virtual memory limit can be set. Thus when limit is reached, the processes are gracefully restarted. This particular feature is welcome in environments where vendor have delivered binaries with memory leaks, thus in production environments these limits may help temporarily cope with the situation until resolved by developer.
TMIB API for system information gathering.
NetXMS agent for monitoring client and server processes, queues, services, machine and cluster.
Full support of build tools (buildserver, buildclient, buildtms).[3]
XATMI service dispatch threading.
Data Dependent Routing support for UBF buffers.
Automatic transactions for XATMI servers.
Service priority handling.
On the fly log setting changing and log rotation for any Enduro/X related program.
Latent Command Framework for custom real-time command publishing to processes.
Performance
Several benchmarks have been made with the Enduro/X middleware and it shows that it competes with Oracle Tuxedo and ZeroMQ quite well. Enduro/X was faster than Tuxedo 30% in 56 KB data block range.[4] Enduro/X was significantly faster than ZeroMQ in 10-20KB range.[5] Also for test performed with RabbitMQ Enduro/X showed its strength and was ahead of the competitor. For the RPC calls, Enduro/X was several times ahead, for send only calls, RabbitMQ was bit closer to Enduro/X.[6]
Subsystems
Enduro/X contains different sub-modules which extends the application server and middleware functionality.
Queuing subsystem
Enduro/X provides a queuing subsystem called TMQ (Transactional Message Queue). This facility provides persistent queues that allows applications to explicitly enqueue and dequeue messages from named queues. Queues can be ordered by message en-queue time in LIFO or FIFO order. Queues are managed by an XA compliant resource manager allowing queue operations to participate in distributed transactions. An automated queue forwarding feature is provided that will remove entries from a queue and invoke an associated Enduro/X ATMI services, placing the reply message on an associated reply queue and failed messages to failure queue.
Configuration services
Enduro/X offers applications to use ini file based configuration reading via special XATMI server, provided in package, named "cconfsrv". Basically via standard tpcall() user application is able to read configuration files (can be separate ini file or folder with them). Technique allows to use the sub-sections with key/value inheritance from main section.
Trace logging services
Enduro/X introduced new set of API function named tplog. This allows user to use the same C based high-performance debug logging facilities which Enduro/X uses internally. It is possible to redirect all logging by thread bases files. Also there are APIs for "request logging", which means that user and Enduro/X can do logging in separate files for each request (e.g. log by session id, log by username, etc.)
Enduro/X Connect package
Enduro/X Connect package provides generic way for exposing XATMI services for outer world. Thus service can be invoked as REST normal services. The package provides functions for converting message internal messages to JSON format and vice versa. Web service module is extensible and provides several options for data conversion between formats. Module additionally supports file uploading, static content serving and orchestration of global transactions over the web service interface.
Enduro/X Connect package also provides module for accessing outer world web services via internal XATMI service call
Third connectivity component is in Enduro/X Connect package is generic TCP/IP connector which provides bi-directional XATMI service API to the TCP/IP sockets. Such functionality simplifies application development for IoT and Banking industries, where the data for different reasons needs to be sent in custom binary protocols, such as ISO8583.[7]
Application Server for Go (ASG)
ASG is bindings project on top of the Enduro/X. Authors considers these binding as application server for go, because Golang projects are linked statically and by default there is no such thing as components. Enduro/X allows to run separate executable binaries, which represents stateless services. These services can be reloaded on the fly without service interruption. Also platform allows to run distributed transaction processing by using Golang. Developers provides modified Oracle DB OCI8 drivers for Go, which supports XA transactions. Enduro/X ASG provides first ever distributed transaction facility for Golang..
Enduro/X Java
Enduro/X version 7.0 offers binding for Java. Full features XATMI servers and client processes may be written Java. Framework is based on multi-processing principle, so that XATMI servers and clients have its own copy of JVM. Requirements for XATMI server/service classes are minimal, thus framework may be considered as POJO application server. XA two phase commit can be used too with standard JDBC drivers. Transactions are managed by standard Enduro/X Core transaction manager to which JDBC drivers are loaded for transaction orchestration. Binding package also offer Java Linker process called exjld. It offers option to link java objects (jar files) to standard executable process with dependency to libjvm and libjava. Thus all jars may be packed in executable process, making Java software delivery simpler, now each process may contain all its dependencies, thus allowing different executables to use different versions of common jar dependencies.
As server process both java executable with classpath or linked java binaries are supported, ndrxd daemon can boot both of them.
Enduro/X-PHP
Full client bindings are available for PHP5. Client PHP processes might compile a loadable module or compile the Enduro/X process inside the PHP. When PHP process is running as client, it must be present in Enduro/X Application environment, that could be done when application is hosted under the same user where PHP app runs (Apache or so) or for the PHP user creating a micro environment which further connects to main application server via tpbridge process. Module is available here [4].
Enduro/X-Perl
Client and server bindings are available for Perl scripting language. Perl module is available here [5].
Acquisition
The Enduro/X was acquired from ATR Baltic, Ltd by Mavimax Ltd on February, 2017 with full rights to the source code and current agreements.[8]
History
Enduro/X was developed as closed source project by ATR Baltic. The development begun in December 2009 and first production customers (mainly banks) where put on the system in late 2012, at the time Enduro/X reached version 1.0. The project was open sourced in 2015, with version v2.3.2. From that event, all the development is made publicly available. Constantly new features are added with every major version release. Till version 6.0 Enduro/X was available under GNUGPLv2 license, starting from Enduro/X 6.0, project is moved to GNU AGPLv3 license. Starting with version 7.0, license is eased for middleware usage from Java and Go programming languages. The statement is that AGPL used by Enduro/X does not affect the Java or Go code which uses Enduro/X. Thanks to these changes, now Enduro/X Java and Enduro/X Go (ASG) modules now are released under LGPL v3, thus Java and Go projects may use middleware commercially with any extra license, while the support is provided commercially.
