Starting from epoch 25 the rewards scheme is fair-share!
Performance focused - GROOT - Mina Staking-as-a-Service
A tree for all the epochs that GROOT staking node is running.
+1 tree/epoch>0 minted blocks, +2/epoch>20
GROOT Staking as a Service is a resilient set of nodes for Mina blockchain.
Staking-as-a-Service GROOT is performance oriented staking node for the Mina project. Having a bigger or smaller forest planted solely from nodes profit will be a great gift.
What rewards do I offer? Fair-share, any block produced is distributed proportionate to your stake.
Starting from epoch 25 (March 24th) the rewards will be distributed in a fair-share system.
That means: when the node has successfully produced a block your share will be X percent from the block value, X is your share from the total staked amount on the node.
Feel free to use the contact form below for any information or ideas.
What hardware am I using? i9 & Ryzen
At least three physically dedicated servers covering Europe and North America with a good combination of Tier1 and T2 peering/routes. I had servers also in Asia (SG, JP, HK) but were proven to underperform due to global distribution of nodes.
The specs are: 6 to 32 cores and 32 to 128 GB RAM. The "newest-november" acquisition is an Intel Core i9-12900K with DDR5.
Running SNARKS.
I will avoid the use of AWS, GC and Azure infrastructure for Mina network. In fact I avoid these three in all of the crypto projects in which I'm involved, in my opinion is ethically wrong to provide a decentralization service using a...monopolistic centralized system.
Also, from time to time I use various low and high end specs for testing purposes.
Means of communications Via this website contact form. There will be an email subscription for any Staking-as-a-Service related queries.
Mina delegators and Staking-as-a-Service Operators may need to use some of the below resources at one point. All the websites whoing metrics displays informations taken from Mina blockchain explorer however there can differences between the information format. Check and use the one you like.
Useful metrics about Mina staking nodes for delegators and node operators in MINA Network
Mina blockchain technically related content
Telegram channels for Mina Blockchain, Nodes Operator and Mina Developers
1. What about decentralization? There are over 650 Genesis Founding Members and any of them can run nodes. Nodes can be any or all of these: Block Producers, SNARK-ers, Archive. Over 500 Validators are active on the network at 12th of June 2021, I have a positive feeling about decentralization considering the mainnet launch on March.
2. Mina depends and use Google Cloud for Archive nodes? It doesn't depends or rely on GC, they use it as well as using AWS. GC & AWS are easy to manage and scale, personally I don't use the big three, but if I was O1 Labs creating such a wonderful blockchain I would like to spend as less time as possible on settings servers but focus on the protocol development.
3. Why TPS is low? Is common sense and a logical decision to focus on what's important and what makes the protocol great instead of what's important for mythology. TPS can be increased fairly easy but it won't benefit the chain yet.
4. Transaction procesing time? A transaction average processing time is not more than 4 minutes. A block is created at approximately every 3 minutes, if you've made a transaction 30 seconds before the block production time, 30 seconds will be the processing time. However, when using exchanges we can wait for about 1 hour to deposit. Kraken as an example needs 15 confirmations for deposit, that mean 15 blocks which takes around 50 minutes, withdrawals are done in about 5 minutes from my experience.
medium.com/minaprotocol A great explanation to most common questions and what's next, published on 12th of June 2021
For any cold operations use a computer which was not, and will never be connected to the internet.
That's your effort but keys and funds remain secure.
Note: Have Money - Love Story! No Money - I'm Sorry!
When we install any server: Dedicated, VPS or a Raspberry PI in the bedroom don’t let him running around before securing it as fast as possible after installation/deployment. Turn it off if in a rush but don't let it naked.
If SSH is not in place, we need to either create a new pair on the new computer. Or copy them via rsync, ssh-copy-id, WinSCP or whatever you like.
Test the keys are working: without closing the current session, open a new SSH, if not successful take another try as something is missing.
# 1. Permissions when you copy chmod 600 ~/.ssh/authorized_keys chmod 700 ~/.ssh # 2. SSHD restart for Debian/Ubuntu systemctl restart ssh.service # 3. SSHD edit nano /etc/ssh/sshd_config # 4. Port, X11 and Pwd Auth Changes # Check through the entire page # duplicate lines can be at the bottom Port 836 PasswordAuthentication no #no for bellow if you don't use root PermitRootLogin prohibit-password X11Forwarding no # 5. 1000 most scanned ports 1,3-4,6-7,9,13,17,19-26,30,32-33,37,42-43,49,53,70,79-85,88-90,99-100,106,109-111,113,119,125,135,139,143-144,146,161,163,179,199,211-212,222,254-256,259,264,280,301,306,311,340,366,389,406-407,416-417,425,427,443-445,458,464-465,481,497,500,512-515,524,541,543-545,548,554-555,563,587,593,616-617,625,631,636,646,648,666-668,683,687,691,700,705,711,714,720,722,726,749,765,777,783,787,800-801,808,843,873,880,888,898,900-903,911-912,981,987,990,992-993,995,999-1002,1007,1009-1011,1021-1100,1102,1104-1108,1110-1114,1117,1119,1121-1124,1126,1130-1132,1137-1138,1141,1145,1147-1149,1151-1152,1154,1163-1166,1169,1174-1175,1183,1185-1187,1192,1198-1199,1201,1213,1216-1218,1233-1234,1236,1244,1247-1248,1259,1271-1272,1277,1287,1296,1300-1301,1309-1311,1322,1328,1334,1352,1417,1433-1434,1443,1455,1461,1494,1500-1501,1503,1521,1524,1533,1556,1580,1583,1594,1600,1641,1658,1666,1687-1688,1700,1717-1721,1723,1755,1761,1782-1783,1801,1805,1812,1839-1840,1862-1864,1875,1900,1914,1935,1947,1971-1972,1974,1984,1998-2010,2013,2020-2022,2030,2033-2035,2038,2040-2043,2045-2049,2065,2068,2099-2100,2103,2105-2107,2111,2119,2121,2126,2135,2144,2160-2161,2170,2179,2190-2191,2196,2200,2222,2251,2260,2288,2301,2323,2366,2381-2383,2393-2394,2399,2401,2492,2500,2522,2525,2557,2601-2602,2604-2605,2607-2608,2638,2701-2702,2710,2717-2718,2725,2800,2809,2811,2869,2875,2909-2910,2920,2967-2968,2998,3000-3001,3003,3005-3007,3011,3013,3017,3030-3031,3052,3071,3077,3128,3168,3211,3221,3260-3261,3268-3269,3283,3300-3301,3306,3322-3325,3333,3351,3367,3369-3372,3389-3390,3404,3476,3493,3517,3527,3546,3551,3580,3659,3689-3690,3703,3737,3766,3784,3800-3801,3809,3814,3826-3828,3851,3869,3871,3878,3880,3889,3905,3914,3918,3920,3945,3971,3986,3995,3998,4000-4006,4045,4111,4125-4126,4129,4224,4242,4279,4321,4343,4443-4446,4449,4550,4567,4662,4848,4899-4900,4998,5000-5004,5009,5030,5033,5050-5051,5054,5060-5061,5080,5087,5100-5102,5120,5190,5200,5214,5221-5222,5225-5226,5269,5280,5298,5357,5405,5414,5431-5432,5440,5500,5510,5544,5550,5555,5560,5566,5631,5633,5666,5678-5679,5718,5730,5800-5802,5810-5811,5815,5822,5825,5850,5859,5862,5877,5900-5904,5906-5907,5910-5911,5915,5922,5925,5950,5952,5959-5963,5987-5989,5998-6007,6009,6025,6059,6100-6101,6106,6112,6123,6129,6156,6346,6389,6502,6510,6543,6547,6565-6567,6580,6646,6666-6669,6689,6692,6699,6779,6788-6789,6792,6839,6881,6901,6969,7000-7002,7004,7007,7019,7025,7070,7100,7103,7106,7200-7201,7402,7435,7443,7496,7512,7625,7627,7676,7741,7777-7778,7800,7911,7920-7921,7937-7938,7999-8002,8007-8011,8021-8022,8031,8042,8045,8080-8090,8093,8099-8100,8180-8181,8192-8194,8200,8222,8254,8290-8292,8300,8333,8383,8400,8402,8443,8500,8600,8649,8651-8652,8654,8701,8800,8873,8888,8899,8994,9000-9003,9009-9011,9040,9050,9071,9080-9081,9090-9091,9099-9103,9110-9111,9200,9207,9220,9290,9415,9418,9485,9500,9502-9503,9535,9575,9593-9595,9618,9666,9876-9878,9898,9900,9917,9929,9943-9944,9968,9998-10004,10009-10010,10012,10024-10025,10082,10180,10215,10243,10566,10616-10617,10621,10626,10628-10629,10778,11110-11111,11967,12000,12174,12265,12345,13456,13722,13782-13783,14000,14238,14441-14442,15000,15002-15004,15660,15742,16000-16001,16012,16016,16018,16080,16113,16992-16993,17877,17988,18040,18101,18988,19101,19283,19315,19350,19780,19801,19842,20000,20005,20031,20221-20222,20828,21571,22939,23502,24444,24800,25734-25735,26214,27000,27352-27353,27355-27356,27715,28201,30000,30718,30951,31038,31337,32768-32785,33354,33899,34571-34573,35500,38292,40193,40911,41511,42510,44176,44442-44443,44501,45100,48080,49152-49161,49163,49165,49167,49175-49176,49400,49999-50003,50006,50300,50389,50500,50636,50800,51103,51493,52673,52822,52848,52869,54045,54328,55055-55056,55555,55600,56737-56738,57294,57797,58080,60020,60443,61532,61900,62078,63331,64623,64680,65000,65129,65389
If you can’t choose what port to use, check over this list with the most scanned 1000 ports and avoid them. Also, on iana.org you may find a detailed description of the chosen port, if in use by someone.
SSHD restart and confirm the changes.
Before the upgrade we can set the time zone and language. Can save you time on some future updates. If you do it, when generating the en_US language or something you prefer, check it! As some of the providers only use en_GB or some substitute, change the 2nd line accordingly. Set the time zone to UTC.
When doing the updates you can be asked to keep some currently installed configuration files or replace them, one can be sshd_config, keep current version if you don’t know what are the changes. Implicitly, new configuration is required when replaced.
# 6. Before the upgrade #timezone dpkg-reconfigure tzdata #language locale-gen en_US.UTF-8 update-locale LANG=en_US.UTF-8 # 7. Updates and housekeeping apt-get update -y apt-get upgrade -y apt-get autoremove apt-get autoclean #Unattended-updates apt-get install unattended-upgrades dpkg-reconfigure -plow unattended-upgrades
After the updates reboot! This reboot is mandatory and most of the times is recommended to re-run the upgrades again as some packages might be pretty ancient, skipping this step can affect the performance of the system. The unattended updates package consist of stable versions only but if you don't trust it just drop it.
The following part can be ignored but I think limiting the connections to your server is beneficial. Limits per Mina node port and the number of global in-connections. For this we have to add the rules on before.rules configuration UFW file, as it is before rules (before the user rules) your whitelisted IP should also be present here. Using before rules in order to also have a summary look inside the rules and we don't want to add rules that will be subsequently ignored.
Two important facts are: 1.if we write a rule in to the UFW rules configuration files and we mess something the UFW will become and stay disabled when reloading, with a message about which line is broken or some error. 2. the rules are read in order.
If first rule is to allow on port 8302 after that a denied IP, it won’t block the IP access if is connecting to port 8302. In order to deny an IP you have to use ufw insert 2 deny from 129.134.29.123, use insert 2 or 3 (assuming you have at least 2 or 3 rules) for easy remove/check after, avoid position 1 as you may want to use it for critical allows.
We’ll use the masking, basic masking is very easy to understand for UFW allow/block IPv4, advanced masking might be a bit confusing.
The basics are like that: 8>.16>.24>.32=
Allowing or denying with mask 8: for the above example is any IP starting with 129 will be allowed/denied.
# 8. UFW install apt-get install ufw # 9. Add your IP, node port and enable ufw allow from xxx.xx.xx.xxx ufw allow 8302/tcp ufw enable #10. Add your IP in before.rules # Limit connections coming in nano /etc/ufw/before.rules -I ufw-before-input -s 129.134.29.123 -j ACCEPT -A ufw-before-input -p tcp --syn -m connlimit --connlimit-above 3 --connlimit-mask 32 -j DROP -A ufw-before-input -p tcp --syn --dport 8302 -m connlimit --connlimit-above 100 -j DROP -A ufw-before-input -p tcp --syn -m connlimit --connlimit-above 50 --connlimit-mask 24 -j DROP -A ufw-before-input -p tcp --syn -m connlimit --connlimit-above 400 --connlimit-mask 0 -j DROP
1st rule is to allow our IP.
Maximum 3 connections per IP, whatever the port. The node needs only one per IP, let’s say two if other parties have difficulties connecting and their server is initiating a new connection without firstly closing this one. So, I set 3 because I feel cute.
Maximum 100 connection coming via port 8302, the Mina node port. This is the third rule (or 2nd for block category) to avoid the probable unintended mess that can be caused by a legit but in trouble node who's trying to connect chaotically. We have first rule no more than 3 connections and then if is Mina node related is free for 100. Having this rule after the one with 400 global could make our node useless for Mina related new connections, in the face of a concerted random attack. We don't forget the main scope why we are here: To run the MINA.
Maximum 50 out of 255 potential IPs with 24 mask, note that the number of computers behind those 254 can be larger.
Maximum 500 TCP IN total connections to our node by using mask 0, we don’t need 500 but let's be selfish.
Considering the above rules the maximum needed is 300 for the node to be operative and some more if we update or we watch a movie in the same time. This type of configuration is some sort of protection in case something hard to anticipate is occuring and we still want to remain useful as a Mina Staking-as-a-Service, keep the protocol running but in the same time don't get punched out.
Keep an eye on the Mina updates and protocol needs. Feel free to adjust the numbers however you want or you consider reasonable. No mask 16 and 8 used as the IP distribution around the world is not even. There are no benefits in configuring useless rules, in order to really have useful rules in here there's some work to do and the efficacy will still remain questionable.
After you add any rules straight to *.rules make sure ufw reload.
When on before.rules you can check and alter for your needs: the 5th set of rules is relating to your server ping response, we can DROP it from here or from sysctl. This one is better as you and the allowed IPs above will be able to ping the node for various monitoring/checks but nobody else can. On the sysctl variant is disabled for everyone.
UFW also provide us with maximum initiated attempts limit in a defined period. The default is 6times/30seconds when enabled, the command is:
ufw limit 836/tcp this will deny any IP who tried more than 6times/30sec/port836.
All of the rules that you input by command are added for IPv4 and IPv6. If you have to actively use the UFW adding/removing rules features and only need the IPv4 ones, changing the IPv6 to no in the first row on rules here nano /etc/default/ufw will block all the IPv6 connections and only allow the loopback communication, also no new rules for IPv6 will be added and the rules/status will be easier to read.
When no static IP and your ISP can’t hand you over one the simplest and safest way is to add the entire IP range. With mask 8 allowed we are roughly 200 times less exposed without locking ourselves out.
#11. Extra UFW before.rules checks #DROP ping reply from UFW firewall not sysctl # ok icmp codes for INPUT -A ufw-before-input -p icmp --icmp-type destination-unreachable -j DROP -A ufw-before-input -p icmp --icmp-type time-exceeded -j DROP -A ufw-before-input -p icmp --icmp-type parameter-problem -j DROP -A ufw-before-input -p icmp --icmp-type echo-request -j DROP #12. UFW allow class for dynamic IPs # if you want to use UFW ALLOW user rules ufw insert 1 allow from 129.134.29.123/8 # 129.0.0.0/8 have the same effect
Allowing 10-20 IPs with mask 8 rather than allowing the SSH port is still around 200 times less exposure, so don’t lock yourself out, use whichever mask you are positive about meeting your needs. If definitely know that only the last set of digits from your IP will be changed use mask 24. This rule should be the first one added (see above UFW discussion) in the before.rules right after the "End of required lines" or ufw insert 1 allow from if you use command line for user.rules
If there is no fixed IP, class of IPs or we have to connect from an unknown location we can use simple port knocking, you can go for advanced ones but they can be more annoying.
#13. Port knocking nano /etc/ufw/before.rules -A ufw-before-input -m state --state NEW -m tcp -p tcp --dport 836 -m recent --rcheck --name SSH -j ACCEPT -A ufw-before-input -m state --state NEW -m tcp -p tcp --dport 31747 -m recent --name SSH --remove -j DROP -A ufw-before-input -m state --state NEW -m tcp -p tcp --dport 31748 -m recent --name SSH --set -j DROP -A ufw-before-input -m state --state NEW -m tcp -p tcp --dport 31749 -m recent --name SSH --remove -j DROP
The knocking looks like that: before being able to connect to our SSH port 836 we need to first initiate any connection (can use Putty and change port, or any mean of sending a packet of data) to port 31748 in our example; nothing happens as the packet is dropped but our needed port 836 will be opened for us now, so we can use SSH and do the spring ploughing, after we finished we need to initiate a connection to 31747 or 31749 to close the listening on SSH. *7 and *9 are used to avoid our port opening without closing it straight away by random port scanners.
Very important 7: do not use NTP servers from Ubuntu, Fedora and Google; they aren't accurate! Google was good some time ago, currently is at best a non-sense with low Stratum and excellent latency but ridiculous time. Note: time.nist.gov is a reliable source in many US locations.
Very important 9: use minsources 3 (as global setting), it compares the time of minimum 3 different sources before deciding, it automatically reject the bad one time. Without this enabled the accuracy can get affected if there are NTP's very close but inaccurate and some other far away. Chrony will synchronize with the closest as it considers the other affected by latency.
The easy way to configure is to use pools from ntp.org, choose 3/2/1.country.pool.ntp.org and 2/1/0.continent.pool.ntp.org or whatever you’d like, iburst need to be present on all of the servers and pools, for the pools I go with maxsources between 4 and 8.
#14. Chrony fundamental settings apt-get install chrony nano /etc/chrony/chrony.conf pool 1.sg.pool.ntp.org iburst maxsources 7 maxpoll 8 pool 1.asia.pool.ntp.org iburst maxsources 7 maxpoll 8 server ntp.xtom.com.hk iburst maxpoll 8 maxupdateskew 5.0 rtcsync makestep 0.1 -1 leapsectz right/UTC local stratum 10 minsources 3
We don’t use minpoll and the maxpoll is 8, not less than 6 anyway. Poll 8 means that at maximum 256 seconds we'll ask for new synch with the NTP servers. Poll 7 means 128 seconds, poll 6 is 64 and so on. There are some compulsive disordered myths about using maxpoll 2, 3 or even 0 or below.
If we find ourselves in a position of asking more than 1 time per minute that mean something else in our structure is really unfit for purpose. You may use it when you own a set of 3 NTP servers.
Couple of reasons why it doesn't make sense:
The recommended way is to look for Stratum 1 & 2 servers near Mina node location and use it. If your Google search is currently broken there's another solution:
#15. Start Chrony with IPv4 only (add -4) nano /etc/systemd/system/chronyd.service ExecStart= /usr/lib/systemd/scripts/chronyd-starter.sh -4
Familiar issues: giving IPv6 or listening on but IPv6 disabled, edit chronyd service and start with -4 option, IPv4 only; none of the servers are working, check the firewall and then with your provider to see if they use their own servers.
For improved security and performance, I’ve crafted a set of rules that can be added to any Mina node. Look at the last part if using multiple virtual nodes on one host as some of the marked settings can interfere with the virtualized hosts, however running a single instance of Linux all of the settings should be copy/paste.
#16. Sysctl configuration nano /etc/sysctl.conf #add following lines # Ratio tcp/app net.ipv4.tcp_adv_win_scale = 2 # Latency vs Throughput net.ipv4.tcp_low_latency = 1 # Long live the King net.ipv4.tcp_slow_start_after_idle = 0 # No route saving net.ipv4.tcp_no_metrics_save = 1 # Send with first packet net.ipv4.tcp_fastopen = 1 # IPv6 disabling net.ipv6.conf.all.disable_ipv6 = 1 net.ipv6.conf.default.disable_ipv6 = 1 net.ipv6.conf.lo.disable_ipv6 = 1 # BBR net.core.default_qdisc = fq net.ipv4.tcp_congestion_control = bbr # Gretel and Hansel net.ipv4.conf.all.rp_filter = 1 net.ipv4.conf.default.rp_filter = 1 # ASLR kernel.randomize_va_space = 1 # SYN attacks net.ipv4.tcp_max_syn_backlog = 4096 net.ipv4.tcp_syn_retries = 4 net.ipv4.tcp_synack_retries = 2 net.ipv4.tcp_syncookies = 1 # Ignore ICMP broadcast request and bogus response net.ipv4.icmp_echo_ignore_broadcasts = 1 net.ipv4.icmp_ignore_bogus_error_responses = 1 # Reboot 10 seconds after Kernel mortality, avoid downtime kernel.panic = 10 vm.panic_on_oom = 1 # Using swap vm.swappiness = 10 # Check the below set when running multiple machines on one host # No routing and redirects net.ipv4.ip_forward = 0 net.ipv4.conf.all.accept_redirects = 0 net.ipv4.conf.all.accept_source_route = 0 net.ipv4.conf.all.secure_redirects = 0 net.ipv4.conf.all.send_redirects = 0 net.ipv4.conf.default.accept_redirects = 0 net.ipv4.conf.default.accept_source_route = 0 net.ipv4.conf.default.secure_redirects = 0 net.ipv4.conf.default.send_redirects = 0 net.ipv6.conf.all.accept_redirects = 0 net.ipv6.conf.all.accept_source_route = 0 net.ipv6.conf.default.accept_redirects = 0 net.ipv6.conf.default.accept_source_route = 0 # Martians (If you are interested to look on the logs) net.ipv4.conf.all.log_martians = 1 net.ipv4.conf.default.log_martians = 1 # No pongs to pings, don't enable if you use ping net.ipv4.icmp_echo_ignore_all = 1 net.ipv6.icmp.echo_ignore_all = 1
Will be counterproductive for all of us to describe each of the rules but there is a small description before them, feel free to search for if interested.
The response to ping is the last rule, as we previously added our IP to before.rules and set the others requests to DROP is not necessary to activate this option as this will stop any reply. However if not needed go for it.
IPv6 is faster and better than v4 mostly because lacking in NAT and gateway. In 1-2 years when providers will be more prepared, we’ll have a noticeable improvement in latencies, however at the moment we keep it disabled when running only the Mina node.
Regarding the security, there are some flying poetries around narrating the flaws and other epic fails, my suggestion is to make your research and don't forget that both IPv4 & IPv6 are transport protocols and both need to be protected.
Sysctl might not be fully executed if some of the modules are not loaded at the time of sysctl launching. In this case an easy option is to add a cronjob 33 seconds after restart, you can tweak the time as not more than 5 seconds are needed in 99% of the situations.
#17. Delays and synchronizations #Delay the mina-node service with 60 seconds nano /etc/systemd/system/mina.service #add the below line before ExecStart ExecStartPre = /bin/sleep 60 #Edit crontab and add these lines sudo crontab -e @reboot sleep 33; /sbin/sysctl --load=/etc/sysctl.conf @reboot sleep 35; chronyc -a 'burst 4/4' @reboot sleep 53; chronyc -a makestep @reboot sleep 55; /sbin/hwclock --utc –systohc
These two are together as we need to edit the same file fstab in order to add both of the entries. A swap file is better to be there even when the amount of usable memory is more than enough. Securing the shared memory means that no programs can be executed from there.
#18. Swap file creation and shared memory security #Check if swap is on and happy with size swapon -s #Disable if not happy sudo swapoff -a #Set size (8G=8GB) sudo fallocate -l 8G /swapfile #Secure the swap sudo chown root:root /swapfile sudo chmod 0600 /swapfile #Make it sudo mkswap /swapfile #Enable it sudo swapon /swapfile #Make sure is there swapon -s #Add the following lines in fstab sudo nano /etc/fstab #1st is swap, 2nd is shared memory security /swapfile none swap sw 0 0 tmpfs /run/shm tmpfs defaults,noexec,nosuid 0 0
Download all of the snippets in one TXT file.
With these settings our server is ready to run a Mina Staking-as-a-Service node secure and very efficiently.
In regards to security there are two valid statements:
In regards to resiliency and life in general there is one simple thing, is not about IF but WHEN. Don’t keep all your eggs in one basket and be prepared to lose the battle but win the war...IF is worthy!
Your server, cold wallet, EEPROMs, biometrics, etc. can be stolen, hacked, destroyed at any point. Facts like some mofos having five factors authentication, guesting themselves on their servers to protect some imaginary roots and using some quantum-photonic encryption in order to be imba are mostly BS.
Any layer is prone to failure or penetration, is a matter of time and/or who's on the other side. Assuming that one of the distinctive advantage of the other side is not bending knees backwards but they're passionate about hacking into 100 dollars servers.
Each layer adds complexity to a system, which doesn't automatically convert to security. The weakest link and human factor will remain as it is but the easiness of restoring/replacing a simple system won't. Focus and achieve the right balance for you.
My suggestion is to never use more layers than you can handle and if you believe that something is not safe it isn't. Take it easy, get more knowledge, understand what's happening and if your time, effort or investment for those specific circumstances have logic. If in doubt don't do it!
Keep your bloody cold keys always COLD and have backups.
Copy/paste those lines of code, talk with Jesus, use some sorceries but keep them fucking COLD.
Those keys are everything that matters on our case here; everything else like losing a server or all of them; your nodes hacked by some IRC fans, another set of lunatics sending some PayPal phishing from your SNARK-er…trivial details.
Feel free to ask everything about anything.
I'll get back to you not later than 5 days!