Search results

This paper tests whether Bayesian A/B testing yields better decisions that traditional Neyman-Pearson hypothesis testing. It proposes a model and tests it using a large, multiyear Google Analytics (GA) dataset.

This paper is an empirical study. Competing A/B testing models were used to analyze a large, multiyear dataset of GA dataset for a firm that relies entirely on their website and online transactions for customer engagement and sales.

Bayesian A/B tests of the data not only yielded a clear delineation of the timing and impact of the intellectual property fraud, but calculated the loss of sales dollars, traffic and time on the firm’s website, with precise confidence limits. Frequentist A/B testing identified fraud in bounce rate at 5% significance, and bounces at 10% significance, but was unable to ascertain fraud at the standard significance cutoffs for scientific studies.

None within the scope of the research plan.

Bayesian A/B tests of the data not only yielded a clear delineation of the timing and impact of the IP fraud, but calculated the loss of sales dollars, traffic and time on the firm’s website, with precise confidence limits.

Bayesian A/B testing can derive economically meaningful statistics, whereas frequentist A/B testing only provide p-value’s whose meaning may be hard to grasp, and where misuse is widespread and has been a major topic in metascience. While misuse of p-values in scholarly articles may simply be grist for academic debate, the uncertainty surrounding the meaning of p-values in business analytics actually can cost firms money.

There is very little empirical research in e-commerce that uses Bayesian A/B testing. Almost all corporate testing is done via frequentist Neyman-Pearson methods.

The aim of the paper is to test and demonstrate the potential benefits in applying reinforcement learning instead of traditional methods to optimize the content of a company's mobile application to best help travellers finding their ideal flights. To this end, two approaches were considered and compared via simulation: standard randomized experiments or A/B testing and multi-armed bandits.

The simulation of the two approaches to optimize the content of its mobile application and, consequently, increase flights conversions is illustrated as applied by Skyscanner, using R software.

The first results are about the comparison between the two approaches – A/B testing and multi-armed bandits – to identify the best one to achieve better results for the company. The second one is to gain experiences and suggestion in the application of the two approaches useful for other industries/companies.

The case study demonstrated, via simulation, the potential benefits to apply the reinforcement learning in a company. Finally, the multi-armed bandit was implemented in the company, but the period of the available data was limited, and due to its strategic relevance, the company cannot show all the findings.

The right algorithm can change according to the situation and industry but would bring great benefits to the company's ability to surface content that is more relevant to users and help improving the experience for travellers. The study shows how to manage complexity and data to achieve good results.

The paper describes the approach used by an European leading company operating in the travel sector in understanding how to adapt reinforcement learning to its strategic goals. It presents a real case study and the simulation of the application of A/B testing and multi-armed bandit in Skyscanner; moreover, it highlights practical suggestion useful to other companies.

Uncertainty in the early development of digital business startups can benefit from data-driven testing of hypotheses. Startups face uncertainty not only in product development, but also over the structure of the business model and the nature of the customer or market to address. The authors conceptualize a new model, the Lean Discovery Process (LDP), which focuses on market-based testing from the early business idea through to fully realized product stages of an innovation. The purpose of this paper is to highlight a methodology to help digital business reduce uncertainty and apply lean principles as early as possible in the development of a business concept.

Examining literature in lean startups, lean user experience and lean software development, the authors highlight underlying assumptions of existing lean models. The authors then examine the LDP using the case of raiserve, a social entrepreneurship startup that focuses on the management of cause-based voluntary service.

Existing literature focuses on product development against an assumed customer base. Early hypothesis testing can be applied to business concept development to substantially reduce cost and time to market.

Further investigation of different forms of uncertainty in digital startups can open up opportunities to further apply lean methodologies. A more extensive empirical study to measure the potential impact is warranted.

The authors conceptualize the minimum viable customer and support early testing with concepts from market research and collective intelligence. The authors demonstrate early opportunities to apply lean principles and rigorous hypothesis testing in an LDP that results in significant reductions in time and expense of product development.

Any business that opts to adopt a recommender engine (RE) for various potential benefits must choose from the candidate solutions, by matching to the task of interest and domain. The purpose of this paper is to choose RE that fits best from a set of candidate solutions using rule-based automated machine learning (ML) approach. The objective is to draw trustworthy conclusion, which results in brand building, and establishing a reliable relation with customers and undeniably to grow the business.

An experimental quantitative research method was conducted in which the ML model was evaluated with diversified performance metrics and five RE algorithms by combining offline evaluation on historical and simulated movie data set, and the online evaluation on business-alike near-real-time data set to uncover the best-fitting RE.

The rule-based automated evaluation of RE has changed the testing landscape, with the removal of longer duration of manual testing and not being comprehensive. It leads to minimal manual effort with high-quality results and can possibly bring a new revolution in the testing practice to start a service line “Machine Learning Testing as a service” (MLTaaS) and the possibility of integrating with DevOps that can specifically help agile team to ship a fail-safe RE evaluation product targeting SaaS (software as a service) or cloud deployment.

A small data set was considered for A/B phase study and was captured for ten movies from three theaters operating in a single location in India, and simulation phase study was captured for two movies from three theaters operating from the same location in India. The research was limited to Bollywood and Ollywood movies for A/B phase, and Ollywood movies for simulation phase.

The best-fitting RE facilitates the business to make personalized recommendations, long-term customer loyalty forecasting, predicting the company's future performance, introducing customers to new products/services and shaping customer's future preferences and behaviors.

The proposed rule-based ML approach named “2-stage locking evaluation” is self-learned, automated by design and largely produces time-bound conclusive result and improved decision-making process. It is the first of a kind to examine the business domain and task of interest. In each stage of the evaluation, low-performer REs are excluded which leads to time-optimized and cost-optimized solution. Additionally, the combination of offline and online evaluation methods offer benefits, such as improved quality with self-learning algorithm, faster time to decision-making by significantly reducing manual efforts with end-to-end test coverage, cognitive aiding for early feedback and unattended evaluation and traceability by identifying the missing test metrics coverage.

This study aims to explore the use of the conversion rate metric as well as A/B testing which is complimentary to that measurement for the improvement of exposing and increasing usage of library digital services.

This column is a viewpoint piece. A literature search was performed as well as ideas from cited books.

There are no findings to speak of.

The use of certain marketing techniques such as the conversion rate and the use of user experience testing such as A/B analysis has the potential to increase the ability of libraries to objectively measure the impact of their online services and increase the efficacy of those services.

Two recently graduated MBA students are tasked with developing an ad-serving learning algorithm for a mobile ad-serving company. The case illustrates the way in which hypotheses can be tested in an A/B format or “horse race” in order to establish customer preferences and superior profitability. The case was written for a course elective covering hypothesis testing.

Scatter in the outcome of repeated experiments is unavoidable due to measurement errors in addition to the non-linear nature of the output responses with unknown influential input parameters. It is a standard practice to select an orthogonal array in the Taguchi approach for tracing optimum input parameters by conducting a few number of experiments and confirm them through additional experimentation (if necessary). The purpose of this paper is to present a simple methodology and its validation with existing test results in finding the expected range of the output response by suggesting modifications in the Taguchi method.

The modified Taguchi approach is proposed to find the optimum process parameters and the expected range of the output response.

This paper presents a simple methodology and its validation with existing test results in finding the expected range of the output response by suggesting modifications in the Taguchi method.

Adequacy of this methodology should be examined by considering the test data on different materials and structures.

The introduction of Chauvenet’s criterion and opposing the signal-to-noise ratio transformation on repeated experiments of each test run will provide fruitful results and less computation burden.

This paper aims to demonstrate that Facebook likes affect outcomes in nonprofit settings. Specifically, Facebook likes influence affinity to nonprofits, which, in turn, affects fundraising outcomes.

The authors report three studies that establish that relationship. To examine social contagion, Study 1 – an auction field study – relies on selling artwork created by underprivileged youth. To isolate signaling, Study 2 manipulates the number of total Facebook likes on a page. To isolate commitment escalation, Study 3 manipulates whether a participant clicks a Facebook like.

The results show that Facebook likes increase willingness to contribute in nonprofit settings and that the process goes through affinity, as well as through Facebook impressions and bidding intensity. The total number of Facebook likes has a direct signaling effect and an indirect social contagion effect.

The effectiveness of the proposed mechanisms is limited to nonprofit settings and only applies to short-term effects.

Facebook likes serve as both a quality signal and a commitment mechanism. The magnitude of commitment escalation is larger, and the relationship is moderated by familiarity with the organization. Managers should target Facebook likes at those less familiar with the organization and should prioritize getting a potential donor to leave a like as a step leading to donation, in essence mapping a donor journey from prospective to active, where Facebook likes play an essential role in the journey. In a charity auction setting, the donor journey involves an additional step of bidder intensity.

The approach the authors study is shown effective in nonprofit settings but does not appear to extend to corporate social responsibility more broadly.

To the best of the authors’ knowledge, this study is the first investigation to map Facebook likes to a seller’s journey through signals and commitment, as well as the only investigation to map Facebook likes to charity auctions and show the effectiveness of this in the field.

The purpose of this study was to develop a correlation between the properties of acrylonitrile butadiene styrene parts 3D printed by material extrusion (MEX) process.

The two MEX parameters and their values have been selected by design of experiment method. Three properties of MEX parts, i.e. strength (tensile and three-point bending), surface roughness and the dimensional accuracy, are studied at different build speeds (35 mm/s, 45 mm/s and 55 mm/s) and the layer heights (0.06 mm, 0.10 mm and 0.15 mm).

The results show that tensile strength and three-point bending strength both increase with the decrease in build speed and the layer height. The artifact selected for dimensional accuracy test shows higher accuracy of the features when 3D printed with 0.06 mm layer height at 35 mm/s build speed as compared to those of higher layer heights and build speeds. The optical images of the 3D-printed specimen reveal that lower build speed and the layer height promote higher inter-layer diffusion that has the effect of strong bonding between the layers and, as a result, higher strength of the specimen. The surface roughness values also have direct relation with the build speed and the layer height.

The whole experiments demonstrate that the part quality, surface roughness and the mechanical strength are correlated and depend on the build speed and the layer height.