As smartphone usage continues to grow, the demand for applications increases as well. Recent studies show that Android and iOS are the only relevant platforms left (Richter, 2018). And as shown in Figure 1, the number of applications available nowadays has doubled in just three years. Although these numbers in both platforms are quite high, Figure 1 shows that there is still a gap of at least 600,000 applications that are not present in iOS.

Figure 1. Number of Applications Available in App Stores (Statista and Apple, 2017; Statista et al., 2017)

Developers who do not distribute their applications for both platforms lose a big portion of the market-share of potential users. Developing for both platforms natively is the most optimal way to have an ideal application, but, at the same time, most small development teams cannot afford to start and maintain two projects for a single application. The most common approach to this scenario is to develop a web application, which uses HTML (Hyper Text Markup Language), CSS (Cascading Style Sheets), JavaScript (Together these are the building block of a modern website) and a back-end language. Doing so will create a system that can run in any platform that have a browser and internet access.

On the other hand, the web approach has some disadvantages, such as problems accessing native APIs, performance drops related to the internet connection, lack of the platform look and feel and the impossibility of launching the application into the platform's application store (Bernardes & Myiake, 2016; Charkaoui et al., 2014; Heitkötter et al., 2012; Raj & Tolety, 2012). Even with solutions such as the use of HTML5 (the most recent version of HTML), which provides access to some native APIs, there are still compatibility issues, thus not being the ideal solution to most types of applications. The alternative is to delve into cross-platform development frameworks, which uses a variety of approaches to develop applications with just one code base.

There are five major approaches to cross-platform development (Gaouar et al., 2015; Latif et al., 2016; Raj & Tolety, 2012; Xanthopoulos & Xinogalos, 2013). The first is the web approach, that was previously discussed. The next one is the hybrid approach, which basically transforms a web application into a mobile application, by embedding a small web browser into a native application. The interpreted approach uses an interpreter to transform code into native actions during runtime. The generative approach, that converts code into a native project. And the Model Driven approach, which uses modeling languages, models, meta-models and model transformations to generate native projects.

All these approaches have pros and cons, and all have a best usage scenario, where the advantages are more prominent (Bernardes & Myiake, 2016). But recent trends in the state of industry suggest that the generative approach is taking the lead in popularity for the cross-platform development frameworks. Figure 2 shows the interest of some frameworks in Google search from 2009 to 2018. Xamarin and React Native use the generative approach, PhoneGap and Ionic uses the hybrid approach, and AppCelerator uses the interpreted approach. Other frameworks, such as NativeScript, Weex, Applause, AppMobi, Cannapi, LiveCode, Marmalade, MoSync, Rhomobile and Sencha Touch, were not included in the comparison due to low overall interest compared to the included ones.

Figure 2. Most Popular Frameworks Interest from Each Cross-Platform Development Approach on Google Searches through 2009 to 2018.

Based on this trend, the goal of this work is to compare some of the most popular frameworks for the generative approach. This comparison is not based only on technical points of view, such as what language it uses or the IDE (Integrated Development Environment) that should be used, but also on a non-technical point of view, that includes long-term feasibility, installation efforts and ease of development. In order to create this comparison, a practical study is made, using the context of an application that uses camera and geographic location. This context generative more criteria which is used to evaluate the produced applications.

The search for cross-platform approaches characteristics is not a new topic. Therefore acknowledging them beforehand is a fundamental step to understand the aim of this work. Starting at the foundation of the field, (Heitkötter et al., 2012) is responsible for raising the major criteria for comparison between approaches and frameworks. In his work, he makes a deep comparison between Web, Native, Hybrid (using PhoneGap) and Interpreted (using Titanium). The authors concluded that PhoneGap the preferred choice, if native look and feel can be neglected.

(Dalmasso et al., 2013) were the first ones to develop a comparison using decision factors, facilitating the process of developers making an accurate decision on which approach should be used. Their work also had a focus on finding the general requirements of cross-platform frameworks and its general architecture. They had identified that these requirements are: Multiple mobile platform support, rich user interface, back-end communication, security, support for app-extensions, power consumption, access to built-in features and open source. They made a power consumption, memory and CPU (Central Processor Unity) usage study on PhoneGap and Titanium frameworks, concluding the most resource efficient framework is PhoneGap, and PhoneGap with Sencha Touch 2.0 have a better UI (User Interface) at cost of a bit more resource consumption.

Xanthopoulos and Xinogalos (2013) made a comparative analysis of the approaches and elaborated a case study, developing an application using the Interpreted approach. The analysis shows that the most promising approach is that of generative apps, but they could not find a non-commercial development environment supporting this approach for building generic apps. In the other hand, interpreted was proven to be a solid approach to build cross-platform apps. The author also mentions that both HTML5 and Generated approach evolution will play a major role in the future.

Bernardes and Myiake (2016) gathered the advantages and disadvantages of each approach through a systematic review. Their work also reveals an enlightenment of the modern state-of-the-art, listing most of the relevant collaborations produced. They observed that web approach has a fast and easy development, but do not have the native look and feel, neither can be distributed in application markets. Hybrid applications have a slightly more complicated development and maintain its web look and feel. Interpreted apps have the native look and feel but have performance limitations. And generative apps have a better performance than interpreted ones.

All the reviewed studies, except for Gaouar et al. (2015) have a focus on comparing different frameworks between all approaches, or approaches in general. But in current state of industry, the frameworks have much more identity, making them more suited for different purposes, and different types of developers. Therefore, the need to zoom in and have a more focused comparison becomes a necessity.

First, to select the adequate frameworks to apply in this comparison, a revision on the related works was made with the objective to find generative frameworks. Then more modern frameworks that were not mentioned in the papers were searched, by looking into developer blogs and development-related web pages. After that, all these names were compared by interest in the last 5 years using Google Trends in Figure 2. Finally, the frameworks with the most interest currently were ReactNative, Xamarin, NativeScript and Weex.

In order to dwell into the frameworks and compare them thoroughly, a practical study was made. The study consisted of the development, by the authors of this paper, of a cross-platform application using all the proposed frameworks. As the native API is a frequent concern to crossplatform development approaches, it was decided to implement an application that makes use of it. With that in mind, it was decided that it will use the geolocation and the camera. Also, this research wanted to test (Latif et al., 2016) affirmation that the generative approach cannot use these specific features. To allow for a steep learning curve, the installation and development process were guided using the official documentation available from the tool website. The camera and geolocation usage were done following the documentation recommendations as well.

The criteria to be compared approximates the most common criteria found in the studies, with the addition of domain specific problems that were generative by the practical study.

The criteria, and a question to explain how they were evaluated are shown in Table 1.

In the following section more details about the generative approach are presented.

Table 1. Chosen Criteria for Generative Approach Development

2.1 Generative Approach Frameworks

The generative approach compiles its application like a native application, resulting in platform-specific versions for each target platform as Figure 3 shows (Xanthopoulos & Xinogalos, 2013). This delivers more performance for the application, a true native look and feel and the possibility of extending the code natively if needed, although not recommended. Modern generative frameworks have evolved in the last years, and some problems that are mentioned in literature may be outdated, as will be further discussed.

Figure 3. Generative Development Approach Concept

The related works mention the following frameworks belongs to the generative approach: Applause; Canappi; Codename One; iPhonical; Marmalade; Mono; MoSync; Neomade; Xamarin; e XMLVM. Through development blogs and websites, more frameworks that belong to the generative approach were identified: Haxe; Kotlin/Native; Qt; React Native; NativeScript; Vue.js; Weex; and Xamarin.

React Native (2015) is a framework for developing native Android and iOS applications with JavaScript, generating a complete usable project to be further extended natively using Java or Swift. It works by transforming the code using a connector that transforms the project into a native project. A portion of the developed code in React Native can be reused in a web application with ReactJS, as it is a similar JavaScript code. Building the application with React Native requires the React Native Camera, a module that includes photographs, videos and bar-code scanning. This is a React Native Community extension, therefore it should be installed through its repository page. For the location, React Native already has an integrated solution, therefore no additional module is required.

NativeScript is another framework for natively developing iOS and Android applications with JavaScript. But differently from React Native, it resembles even more native development with an XML (eXtensive Markup Language) for UI, and JS (JavaScript) code behind (Vantoll, 2017). The UI management can be further extended through native code. The framework can be expanded using NativeScript specific plug-ins, Node Package Manager (npm), Gradle and CocoaPods plugins (NativeScript, 2014). Camera and geolocation access can be achieved using NativeScript's specific plugins, following the recommendations for each plug-in in its documentation.

Weex is a Chinese native framework that resembles standard Web Development with JavaScript/CSS/HTML, and can deliver iOS, Android and web software using the same code rendering its views with Vue.js (Weex, 2017). It has support to multiple mobile native only functions, such as camera, through official plug-ins that can be easily integrated into the project. All the plug-ins can be found in the Nat.js website. To access camera and geolocation in Weex, it is necessary to download the official geolocation and camera plug-ins, both using the Nat.js, and follow the official documentation.

Xamarin is the solution to develop cross-platform mobile applications using C#. It uses Visual Studio and was created by Miguel de Icaza (Xamarin, 2017). The main concern of Xamarin is to share the logic, or logic and user interface, between all mobile platforms, and at the same time keeping them native. In Xamarin you can choose between a native user interface and a shared one using Xamarin. Forms, a technology that allows user interface development with XAML (eXtensible Application Markup Language), the main language for Microsoft's C#, UI development, to be shared with all targeted platforms.

For the code sharing strategy, it is possible to choose between a shared project folder, portable class libraries (code sharing through libraries) and Xamarin Forms.

Camera and geolocation usage can be achieved using the Media Plug-in, that allows camera access in the shared C# application logic, with the PCL (Portable Class Libraries) approach, which is one of the development approaches available to Xamarin. Device location can be requested naturally through Xamarin Location Provider.

The Development process take place within a 2 months period, resulting in one of the authors generating a total of 274 lines of code. Specifically: 72 for React Native, 52 for NativeScript, 55 for Weex and 94 for Xamarin. That resulted in 4 versions of the app, each of them runs on atleast 2 platforms (Android and iOS) each, Xamarin's version have support for Windows Phone as well. All these versions can be utilized and meets the requirements elaborated in section 3, being equivalent for each framework.

The overall look of the app is the same for Xamarin and Weex as presented of Figure 4 for NativeScript. React having this difference due do the camera access method.

Figure 4. NativeScript and React Native User Interfaces Comparison

In React Native app, when the button to capture is pressed, the image freezes, storing the photo and displaying the coordinates to the user. For the other versions, a template image is shown at the middle of the screen, until the user presses the button to take a picture, then the built-in camera app is displayed, allowing the user to capture an image. After confirming the picture, the application returns to its initial state, with the template image replaced by the taken photo, and showing the coordinates above it.

The analysis was made from the authors point of view, with the intention to evaluate the criteria previously elaborated. The results for them were cautiously reviewed in order to prevent them from being biased.

This section will discuss the development experience gathered and compare each one relatively to one another.

Starting with camera access, except for React Native, all the frameworks behaved the same. This means that the camera cannot be implemented directly into the application, instead, when you click to take a picture, the native camera application is brought into foreground within the developed application and used to take the picture. This approach has the advantage of using the ability of the device to manually configure focus, exposure and other camera configurations. Having direct camera access means the application view holds an element that shows the camera in real time, and when the button is pressed, the image is captured. This allows for more possibilities for the developer to create interactions with how the image will be used. The Figure 4 shows how this behavior of React Native compared to Native Script.

Weex's lack of camera options customizations only implies that the image will be stored at full resolution, and have the default name, which is composed of the date and hour of the capture.

The installation process was very straightforward for all options. Xamarin uses Visual Studio, so it is a matter of pressing buttons and waiting for the IDE to do the rest. React Native, NativeScript and Weex are installed via Node, so there is not much room for error. NativeScript have a slight disadvantage due to having further configuration after the npm statements.

For user interface development, Xamarin is the only one that features a visualizer that does not require building and deploying the application. React Native and NativeScript have technologies that let you see alterations on the code in real time, so there is no need to a UI visualization tool.

Both React Native and NativeScript have a solid documentation for beginners, making the learning process faster. Xamarin's documentation is a bit overwhelming at first but proves to fulfill most advanced users' questions. Weex documentations are not fully available in English, so for some advanced questions, the developers need to search in the Chinese documentation.

In terms of IDE, Xamarin takes the lead due to Visual Studio having tools for everything. React Native and NativeScript are slightly better than Weex mostly because of the fast deployment technologies, as all of them have basic tools that a developer need.

For the long term feasibility, React Native takes the lead, as it has one of the most active communities, along with frequent update schedules and bug-fixing. Xamarin is a solid tool that is under constant maintenance and have a big community as well. NativeScript and Weex do not have that much popularity but are on par with the maintenance. Weex stands slightly above due to its popularity in China, therefore having a more active community.

The geolocation worked practically the same on all frameworks, having similar implementations. They all also can customize precision and track the position in real time.

In terms of performance, all the resulting applications have the same feel, the buttons responded immediately and the startup took less than a second. Even with the geolocation requests executed after the picture was taken, there were no stutter at all. For an integral benchmark on this subject, further research is needed, with the proper use of a controlled environment and more resource intensive applications.

Regarding maintainability, Xamarin have a slight advantage over other frameworks. This is mainly from the possibility of choosing an architecture that suits best the needs of your project. For more complex software, the possibility of using PCL enables development to be more modularized, facilitating concurrent development and enhancing code clarity.

Table 2 is the overall comparison of the results represented in using the enumeration proposed in Table 1. In this table, the last three criteria were compared relatively to each other, with values from 1 to 4, being 1 the most well performing framework to the criteria.

Table 2. Overall Comparison of Resultant Applications

In this practical study, which involved an application that uses both camera and geolocation, React Native was the preferred framework, for performing better overall in most criteria. But this result does not mean much, as different application requirements may lead to better choices. For example, if an application requires perfectly calibrated photographs, React Native may not be the most indicated, as it relies on the raw image captured by the device alone. Using the native camera application may produce better results in these cases.

Xamarin may be just behind, especially if the developer needs to create a Windows Phone application, which is not supported by other tools. The collected criteria for this domain were camera access and camera options customization. Both have critical importance for developers that wants to incorporate camera into their cross-platform applications.

Another valid point for choosing an adequate framework, is based on how it works. Web developers may be more familiarized with React Native, which resembles ReactJS, and Weex, which works just like traditional web development HTML, CSS and JavaScript. People coming from a traditional mobile development background might be more acquainted to try NativeScript, which resembles native Android development, and Xamarin, which uses XAML for User Interfaces and code behind in C#.

This study attempted to evaluate cross-platform code generation frameworks, in order to analyze the most wellknown concerns and criteria found in the literature. In this study, it was observed that the literature is not always updated, as this is an area of constant evolution and growth. For example, this experiment was able to successfully implement access to the camera and geolocation, despite (Latif et al., 2016) statement that this is not possible.

For future work, further research is needed to elaborate a more complete guide for decision-making when selecting an approach and framework to develop. Such guide should be less focused on tools and technologies, but rather be focused on more timeless concepts and approaches. It should also take into consideration both academic and industrial points of view, surveying and interviewing professionals who work with cross-platform mobile development. The next goal is to gather all these data to show their opinions and advices, providing a more complete picture of the process.

Along with the interview and surveys, more practical studies like this need to be elaborated. These practical studies should have the objective of refining the criteria that were previously gathered from the related work. As shown in this paper, the context of the study reflects directly on its criteria. For a developer, these peculiarities are a fundamental part of their development process. Therefore having an extensive list of criteria is an important task to achieve a complete guide to decision making.

Acknowledgements

The research work reported in this paper received financial support from São Paulo Research Foundation (FAPESP) - Brazil, grant number 2015 / 24429-1.