Bayesian Additive Regression Networks (BARN): A Machine Learning Algorithm and Scientific Software Library

Abstract

We apply Bayesian Additive Regression Tree (BART) principles to training an ensemble of small neural networks for regression tasks. Using Markov Chain Monte Carlo, we sample from the posterior distribution of neural networks that have a single hidden layer. To create an ensemble of these, we apply Gibbs’ sampling to update each network against the residual target value (i.e. subtracting the effect of the other networks). We demonstrate the effectiveness of this technique on several benchmark regression problems, comparing it to equivalent shallow neural networks, BART, and ordinary least squares. Our Bayesian Additive Regression Networks (BARN) provide more consistent and often more accurate results. On test data benchmarks, BARN averaged between 5\% to 20\% lower root mean square error. This error performance does come at the cost, however, of greater computation time. BARN sometimes takes on the order of a minute where competing methods take a second or less. But, BARN without cross-validated hyperparameter tuning takes about the same amount of computation time as tuned other methods. Yet BARN is still typically more accurate. Additionally, we make BARN available as a Python package, `barmpy`, with documentation at `https://dvbuntu.github.io/barmpy/` for general machine learning practitioners. Our object-oriented design is compatible with SciKit-Learn, allowing usage of their tools like cross-validation. To ease learning to use `barmpy`, we produce a companion tutorial that expands on reference information in the documentation. Any interested user can `pip install barmpy` from the official PyPi repository. Finally, we expand BARN beyond regression to classification tasks. Much like in regression, we evaluate it against competing classification methods on benchmark data sets, real and synthetic. Here, we find that BARN is relatively best in specific scenarios, and even then by only a few percentage points as measured by test accuracy. This is, however, consistent with analysis of BART results in earlier research, where other methods were just as, if not more, accurate. BARN is a new method in machine learning with promising evidence for wide applicability. In addition to demonstrating its effectiveness in different situations, we outline how to apply similar principles to model types other than shallow neural networks. With rigorous statistical support and friendly software, BARN should be considered for new general purpose modeling problems.