Presuming that businesses and organizations will increasingly choose to participate directly in the Bitcoin ecosystem and will need to interact with the Bitcoin and Lightning networks at an enterprise scale, it stands to reason that they will need to manage instances of Bitcoin and Lightning nodes including their configurations and connections to other nodes (channels in the case of Lightning nodes). Kubernetes is quickly becoming the global standard cloud-native container orchestration and configuration management engine for modern enterprise software. Using a Kubernetes operator pattern, I intend to explore the utility of Kubernetes as a platform for a hosting and managing private Bitcoin and Lightning infrastructure.
Since these two technical domains do not yet often intersect, I expect any audience to this post series may be familiar with one or the other domain, but not often both. So, for additional context, here are some preliminary answers to what I imagine to be some key questions.
For the k8s community: Why would enterprises manage Bitcoin and Lightning nodes?
When dealing with money transactions in traditional software, it is common to use third-party payment processors. These services provide access to card payment networks, etc. However, the Bitcoin and Lightning networks do not require any special permissions to participate, so a third-party service provider is not absolutely necessary. Arguably, anything other than direct participation in these networks countradicts some of the intentions of the protocol design.
So, any organization that needs to transact on the Bitcoin and/or Lightning networks may choose to do so directly and bypass the cost of a network access provider. However, direct network access must be supported with private network infrastructure, specifically Lightning and/or Bitcoin nodes. Organizations must accept the responsbility of managing this payment infrastructure if they wish to natively transact without trusting a third-party. A standard set of management practices packaged as a Kubernetes operator could help allieviate this operational burden.
For the crypto community: How is Kubernetes useful?
For the purposes of this exploration, think of Kubernetes as a continuous state reconciliation loop. Reconciliation is a process of determining the actual state of the environment and taking actions based on the difference between the actual and the expected states. This design simplifies the input required to manage and operate software. Developers can focus on representing desired environment state, instead of providing the operational procedures to acheive that desired state.
However, the goal for this experiment is to manage Bitcoin and Lightning infrastructure, not just any generic containers, so I will need to extend Kubernetes and "teach" the cluster some things about how to manage Bitcoin and Lightning nodes. To accomplish this, I am developing using a Kubernetes operator pattern. Operators extend the Kubernetes reconciliation loop and with the help of CustomResourceDefinitions we can introduce new domains to the management scope of the Kubernetes engine.
Getting started on the kiln operator
I've developed the beginnings of an operator implementation. At this time, the operator is only capable of managing simple Bitcoin and Lightning node deployments. A Lightning Node requires a connection to a Bitcoin Node, so my intial goal was to support the configuration pertaining to this particular connection.
With the kiln operator running and connected to a Kubernetes cluster, I can apply a BitcoinNode resource with parameters that configure its RPC server.
apiVersion: bitcoin.kiln-fired.github.io/v1alpha1
kind: BitcoinNode
metadata:
name: btcd
namespace: kiln
spec:
rpcServer:
certSecret: btcd-rpc-tls
user: node-user
password: st4cks4ts
In the above respresentation, I provide three configuration parameters for the RPC server. First, the name of a Kubernetes secret that contains a TLS data. I am managing that certificate data using an operator called cert-manager. I'll provide more details about this certificate management technique in a later post, for now, note that I am providing a TLS certificate and private key to secure the endpoint. Next, I provide a set of static credentials for a user that is authorized to use connect to the node.
With the help of the kiln operator, the Kubernetes reconciliation engine detects the new BitcoinNode resource and reconciles the desired state by deploying a basic Bitcoin node instance. Pod logs show the Bitcoin node is running and listening for client connections.
Command: btcd --simnet --debuglevel=info --rpcuser=node-user --rpcpass=st4cks4ts --datadir=/data --logdir=/data --rpccert=/rpc/rpc.cert --rpckey=/rpc/rpc.key --rpclisten=0.0.0.0 --txindex
2022-02-07 21:04:43.597 [INF] BTCD: Version 0.20.1-beta
2022-02-07 21:04:43.598 [INF] BTCD: Loading block database from '/data/simnet/blocks_ffldb'
2022-02-07 21:04:44.491 [INF] BCDB: Detected unclean shutdown - Repairing...
2022-02-07 21:04:44.522 [INF] BCDB: Database sync complete
2022-02-07 21:04:44.524 [INF] BTCD: Block database loaded
2022-02-07 21:04:44.775 [INF] INDX: Transaction index is enabled
2022-02-07 21:04:44.775 [INF] INDX: Committed filter index is enabled
2022-02-07 21:04:45.128 [INF] INDX: Catching up indexes from height -1 to 0
2022-02-07 21:04:45.159 [INF] INDX: Indexes caught up to height 0
2022-02-07 21:04:45.183 [INF] CHAN: Chain state (height 0, hash 683e86bd5c6d110d91b94b97137ba6bfe02dbbdb8e3dff722a669b5d69d77af6, totaltx 1, work 2)
2022-02-07 21:04:45.183 [INF] RPCS: RPC server listening on 0.0.0.0:18556
2022-02-07 21:04:45.262 [INF] AMGR: Loaded 0 addresses from file '/data/simnet/peers.json'
2022-02-07 21:04:45.262 [INF] CMGR: Server listening on 0.0.0.0:18555
2022-02-07 21:04:45.262 [INF] CMGR: Server listening on [::]:18555
Next, I want to fire up a Lightning node with a client connection to the Bitcoin node's RPC server.
apiVersion: bitcoin.kiln-fired.github.io/v1alpha1
kind: LightningNode
metadata:
name: lnd
namespace: kiln
spec:
bitcoinConnection:
host: btcd
network: simnet
certSecret: btcd-rpc-tls
user: node-user
password: st4cks4ts
By providing the RPC connection details in the CustomResource instance, I express that the desired state is a Lightning node which is connected to the previously deployed Bitcoin node instance. Notice the simnet network is specified, meaning that for now, we will not interact with the live public Bitcoin network, but rather operate in our own private simulated network for the purposes of experimentation.
Once the kiln operator reconciles the LightningNode resource. A pod is deployed which logs a successful connection.
2022-02-07 21:04:45.968 [WRN] LTND: open /.lnd/lnd.conf: no such file or directory
2022-02-07 21:04:45.968 [INF] LTND: Version: 0.14.1-beta commit=v0.14.1-beta-2-g03a038c7, build=production, logging=default, debuglevel=info
2022-02-07 21:04:45.968 [INF] LTND: Active chain: Bitcoin (network=simnet)
2022-02-07 21:04:45.971 [INF] RPCS: RPC server listening on 0.0.0.0:10009
2022-02-07 21:04:45.982 [INF] RPCS: gRPC proxy started at 127.0.0.1:8080
2022-02-07 21:04:45.983 [INF] LTND: Opening the main database, this might take a few minutes...
2022-02-07 21:04:45.983 [INF] LTND: Opening bbolt database, sync_freelist=false, auto_compact=false
2022-02-07 21:04:45.999 [INF] LTND: Creating local graph and channel state DB instances
2022-02-07 21:04:46.230 [INF] CHDB: Checking for schema update: latest_version=24, db_version=24
2022-02-07 21:04:46.230 [INF] LTND: Database(s) now open (time_to_open=247.188329ms)!
2022-02-07 21:04:47.308 [INF] CHRE: Primary chain is set to: bitcoin
2022-02-07 21:04:57.249 [INF] LNWL: Opened wallet
2022-02-07 21:04:59.014 [INF] LNWL: The wallet has been unlocked without a time limit
2022-02-07 21:05:13.220 [INF] CHRE: LightningWallet opened
2022-02-07 21:05:13.252 [INF] HSWC: Cleaning circuits from disk for closed channels
2022-02-07 21:05:13.252 [INF] HSWC: Finished cleaning: no closed channels found, no actions taken.
2022-02-07 21:05:13.252 [INF] HSWC: Restoring in-memory circuit state from disk
2022-02-07 21:05:13.258 [INF] HSWC: Payment circuits loaded: num_pending=0, num_open=0
2022-02-07 21:05:13.274 [INF] LNWL: Started rescan from block 683e86bd5c6d110d91b94b97137ba6bfe02dbbdb8e3dff722a669b5d69d77af6 (height 0) for 0 addresses
2022-02-07 21:05:13.278 [INF] LNWL: Catching up block hashes to height 0, this might take a while
2022-02-07 21:05:13.283 [INF] LNWL: Done catching up block hashes
2022-02-07 21:05:13.283 [INF] LNWL: Finished rescan for 0 addresses (synced to block 683e86bd5c6d110d91b94b97137ba6bfe02dbbdb8e3dff722a669b5d69d77af6, height 0)
2022-02-07 21:05:13.364 [INF] LTND: Channel backup proxy channel notifier starting
2022-02-07 21:05:13.364 [INF] ATPL: Instantiating autopilot with active=false, max_channels=5, allocation=0.600000, min_chan_size=20000, max_chan_size=16777215, private=false, min_confs=1, conf_target=3
2022-02-07 21:05:13.364 [INF] LTND: We're not running within systemd
2022-02-07 21:05:13.365 [INF] LTND: Waiting for chain backend to finish sync, start_height=0
And this connection is reflected in the Bitcoin node logs as well.
2022-02-07 21:05:13.191 [INF] RPCS: New websocket client 10.217.0.45:50334
2022-02-07 21:05:13.277 [INF] RPCS: Beginning rescan for 0 addresses
2022-02-07 21:05:13.278 [INF] RPCS: Skipping rescan as client has no addrs/utxos
2022-02-07 21:05:13.278 [INF] RPCS: Finished rescan
Continue reading the next post in this series.