Presuming that businesses and organizations will increasingly choose to participate directly in the Bitcoin ecosystem and will need to interact with the Bitcoin and Lightning networks at an enterprise scale, it stands to reason that they will need to manage instances of Bitcoin and Lightning nodes including their configurations and connections to other nodes (channels in the case of Lightning nodes). Kubernetes is quickly becoming the global standard cloud-native container orchestration and configuration management engine for modern enterprise software. Using a Kubernetes operator pattern, I intend to explore the utility of Kubernetes as a platform for a hosting and managing private Bitcoin and Lightning infrastructure.
Since these two technical domains do not yet often intersect, I expect any audience to this post series may be familiar with one or the other domain, but not often both. So, for additional context, here are some preliminary answers to what I imagine to be some key questions.
For the k8s community: Why would enterprises manage Bitcoin and Lightning nodes?
When dealing with money transactions in traditional software, it is common to use third-party payment processors. These services provide access to card payment networks, etc. However, the Bitcoin and Lightning networks do not require any special permissions to participate, so a third-party service provider is not absolutely necessary. Arguably, anything other than direct participation in these networks countradicts some of the intentions of the protocol design.
So, any organization that needs to transact on the Bitcoin and/or Lightning networks may choose to do so directly and bypass the cost of a network access provider. However, direct network access must be supported with private network infrastructure, specifically Lightning and/or Bitcoin nodes. Organizations must accept the responsbility of managing this payment infrastructure if they wish to natively transact without trusting a third-party. A standard set of management practices packaged as a Kubernetes operator could help allieviate this operational burden.
For the crypto community: How is Kubernetes useful?
For the purposes of this exploration, think of Kubernetes as a continuous state reconciliation loop. Reconciliation is a process of determining the actual state of the environment and taking actions based on the difference between the actual and the expected states. This design simplifies the input required to manage and operate software. Developers can focus on representing desired environment state, instead of providing the operational procedures to acheive that desired state.
However, the goal for this experiment is to manage Bitcoin and Lightning infrastructure, not just any generic containers, so I will need to extend Kubernetes and "teach" the cluster some things about how to manage Bitcoin and Lightning nodes. To accomplish this, I am developing using a Kubernetes operator pattern. Operators extend the Kubernetes reconciliation loop and with the help of
CustomResourceDefinitions we can introduce new domains to the management scope of the Kubernetes engine.
Getting started on the
I've developed the beginnings of an operator implementation. At this time, the operator is only capable of managing simple Bitcoin and Lightning node deployments. A Lightning Node requires a connection to a Bitcoin Node, so my intial goal was to support the configuration pertaining to this particular connection.
kiln operator running and connected to a Kubernetes cluster, I can apply a
BitcoinNode resource with parameters that configure its RPC server.
apiVersion: bitcoin.kiln-fired.github.io/v1alpha1 kind: BitcoinNode metadata: name: btcd namespace: kiln spec: rpcServer: certSecret: btcd-rpc-tls user: node-user password: st4cks4ts
In the above respresentation, I provide three configuration parameters for the RPC server. First, the name of a Kubernetes secret that contains a TLS data. I am managing that certificate data using an operator called
cert-manager. I'll provide more details about this certificate management technique in a later post, for now, note that I am providing a TLS certificate and private key to secure the endpoint. Next, I provide a set of static credentials for a user that is authorized to use connect to the node.
With the help of the
kiln operator, the Kubernetes reconciliation engine detects the new
BitcoinNode resource and reconciles the desired state by deploying a basic Bitcoin node instance. Pod logs show the Bitcoin node is running and listening for client connections.
Command: btcd --simnet --debuglevel=info --rpcuser=node-user --rpcpass=st4cks4ts --datadir=/data --logdir=/data --rpccert=/rpc/rpc.cert --rpckey=/rpc/rpc.key --rpclisten=0.0.0.0 --txindex 2022-02-07 21:04:43.597 [INF] BTCD: Version 0.20.1-beta 2022-02-07 21:04:43.598 [INF] BTCD: Loading block database from '/data/simnet/blocks_ffldb' 2022-02-07 21:04:44.491 [INF] BCDB: Detected unclean shutdown - Repairing... 2022-02-07 21:04:44.522 [INF] BCDB: Database sync complete 2022-02-07 21:04:44.524 [INF] BTCD: Block database loaded 2022-02-07 21:04:44.775 [INF] INDX: Transaction index is enabled 2022-02-07 21:04:44.775 [INF] INDX: Committed filter index is enabled 2022-02-07 21:04:45.128 [INF] INDX: Catching up indexes from height -1 to 0 2022-02-07 21:04:45.159 [INF] INDX: Indexes caught up to height 0 2022-02-07 21:04:45.183 [INF] CHAN: Chain state (height 0, hash 683e86bd5c6d110d91b94b97137ba6bfe02dbbdb8e3dff722a669b5d69d77af6, totaltx 1, work 2) 2022-02-07 21:04:45.183 [INF] RPCS: RPC server listening on 0.0.0.0:18556 2022-02-07 21:04:45.262 [INF] AMGR: Loaded 0 addresses from file '/data/simnet/peers.json' 2022-02-07 21:04:45.262 [INF] CMGR: Server listening on 0.0.0.0:18555 2022-02-07 21:04:45.262 [INF] CMGR: Server listening on [::]:18555
Next, I want to fire up a Lightning node with a client connection to the Bitcoin node's RPC server.
apiVersion: bitcoin.kiln-fired.github.io/v1alpha1 kind: LightningNode metadata: name: lnd namespace: kiln spec: bitcoinConnection: host: btcd network: simnet certSecret: btcd-rpc-tls user: node-user password: st4cks4ts
By providing the RPC connection details in the
CustomResource instance, I express that the desired state is a Lightning node which is connected to the previously deployed Bitcoin node instance. Notice the
simnet network is specified, meaning that for now, we will not interact with the live public Bitcoin network, but rather operate in our own private simulated network for the purposes of experimentation.
kiln operator reconciles the
LightningNode resource. A pod is deployed which logs a successful connection.
2022-02-07 21:04:45.968 [WRN] LTND: open /.lnd/lnd.conf: no such file or directory 2022-02-07 21:04:45.968 [INF] LTND: Version: 0.14.1-beta commit=v0.14.1-beta-2-g03a038c7, build=production, logging=default, debuglevel=info 2022-02-07 21:04:45.968 [INF] LTND: Active chain: Bitcoin (network=simnet) 2022-02-07 21:04:45.971 [INF] RPCS: RPC server listening on 0.0.0.0:10009 2022-02-07 21:04:45.982 [INF] RPCS: gRPC proxy started at 127.0.0.1:8080 2022-02-07 21:04:45.983 [INF] LTND: Opening the main database, this might take a few minutes... 2022-02-07 21:04:45.983 [INF] LTND: Opening bbolt database, sync_freelist=false, auto_compact=false 2022-02-07 21:04:45.999 [INF] LTND: Creating local graph and channel state DB instances 2022-02-07 21:04:46.230 [INF] CHDB: Checking for schema update: latest_version=24, db_version=24 2022-02-07 21:04:46.230 [INF] LTND: Database(s) now open (time_to_open=247.188329ms)! 2022-02-07 21:04:47.308 [INF] CHRE: Primary chain is set to: bitcoin 2022-02-07 21:04:57.249 [INF] LNWL: Opened wallet 2022-02-07 21:04:59.014 [INF] LNWL: The wallet has been unlocked without a time limit 2022-02-07 21:05:13.220 [INF] CHRE: LightningWallet opened 2022-02-07 21:05:13.252 [INF] HSWC: Cleaning circuits from disk for closed channels 2022-02-07 21:05:13.252 [INF] HSWC: Finished cleaning: no closed channels found, no actions taken. 2022-02-07 21:05:13.252 [INF] HSWC: Restoring in-memory circuit state from disk 2022-02-07 21:05:13.258 [INF] HSWC: Payment circuits loaded: num_pending=0, num_open=0 2022-02-07 21:05:13.274 [INF] LNWL: Started rescan from block 683e86bd5c6d110d91b94b97137ba6bfe02dbbdb8e3dff722a669b5d69d77af6 (height 0) for 0 addresses 2022-02-07 21:05:13.278 [INF] LNWL: Catching up block hashes to height 0, this might take a while 2022-02-07 21:05:13.283 [INF] LNWL: Done catching up block hashes 2022-02-07 21:05:13.283 [INF] LNWL: Finished rescan for 0 addresses (synced to block 683e86bd5c6d110d91b94b97137ba6bfe02dbbdb8e3dff722a669b5d69d77af6, height 0) 2022-02-07 21:05:13.364 [INF] LTND: Channel backup proxy channel notifier starting 2022-02-07 21:05:13.364 [INF] ATPL: Instantiating autopilot with active=false, max_channels=5, allocation=0.600000, min_chan_size=20000, max_chan_size=16777215, private=false, min_confs=1, conf_target=3 2022-02-07 21:05:13.364 [INF] LTND: We're not running within systemd 2022-02-07 21:05:13.365 [INF] LTND: Waiting for chain backend to finish sync, start_height=0
And this connection is reflected in the Bitcoin node logs as well.
2022-02-07 21:05:13.191 [INF] RPCS: New websocket client 10.217.0.45:50334 2022-02-07 21:05:13.277 [INF] RPCS: Beginning rescan for 0 addresses 2022-02-07 21:05:13.278 [INF] RPCS: Skipping rescan as client has no addrs/utxos 2022-02-07 21:05:13.278 [INF] RPCS: Finished rescan
Continue reading the next post in this series.