Here’s my play by play for installing a Thales nShield Connect network-based HSM into a CentOS linux environment:
- Edit your ~/.bash_profile to include the following additional directory in your path and then re-exec bash: /opt/nfast/bin
Or you could also
# echo 'PATH=$PATH:/opt/nfast/bin' > /etc/profile.d/nfast.sh # exec bash
- Get the Thales nShield client software from Thales support. They should end up providing you with a zip file that has an iso file inside it, which you then mount to get at tar files. It will be named something like SecWorld-linux64-user-12.10.01.zip with SecWorld-linux64-user-12.10.01.iso.
- Mount the ISO locally, such as:
mount -o loop /root/SecWorld-linux64-user-12.10.01.iso /mnt
- This next part is weird. You need to be in the true root directory, aka /, because the tar files have the expectation you’ll be extracting them from that point so the files end up in the correct places. You will likely need to extract more than one tar file depending on which components you need. Consult the nShileld Connect Installation Guide for the list of the components. I’m doing OpenSSL CHIL interface, but also want to use the random number generator (requires PKCS#11), so I’m going to extract the following, relative to my mount point of the ISO.
2023 edit – the files you extract change across versions so here are the two versions I am familiar with.
Version 12.40.2 and older:
/mnt/linux/libc6_11/amd64/nfast/ctls/agg.tar /mnt/linux/libc6_11/amd64/nfast/dsserv/user.tar /mnt/linux/libc6_11/amd64/nfast/hwcrhk/gnupg.tar /mnt/linux/libc6_11/amd64/nfast/hwcrhk/user.tar /mnt/linux/libc6_11/amd64/nfast/hwsp/agg.tar /mnt/linux/libc6_11/amd64/nfast/javasp/agg.tar /mnt/linux/libc6_11/amd64/nfast/jcecsp/user.tar /mnt/linux/libc6_11/amd64/nfast/nhfw/agg.tar /mnt/linux/libc6_11/amd64/nfast/pkcs11/user.tar /mnt/linux/libc6_11/amd64/nfast/ratls/agg.tar /mnt/linux/libc6_11/amd64/nfast/snmp/agg.tar /mnt/linux/libc6_11/amd64/nfast/version.txt
Version 12.60 through at least 12.80.4:
/mnt/linux/amd64/ctd.tar.gz /mnt/linux/amd64/ctls.tar.gz /mnt/linux/amd64/devref.tar.gz /mnt/linux/amd64/hwsp.tar.gz /mnt/linux/amd64/javasp.tar.gz /mnt/linux/amd64/jd.tar.gz /mnt/linux/amd64/ncsnmp.tar.gz /mnt/linux/amd64/raserv.tar.gz
- Switch to /opt/nfast/sbin and run ./install
I was expecting this to be a disaster as the experience has been a little kludgy but it ran without issue:
/opt/nfast/sbin# ./install ---- Stopping any nCipher servers ---- No nCipher init scripts installed. ---- Cleaning up any old install ---- No nCipher components requiring cleanup found. ---- Installing ---- -- Running install fragment 10 nfastug Checking for user 'nfast' in group 'nfast' Creating nfast group. Creating nfast user. useradd: warning: the home directory already exists. Not copying any file from skel directory into it. Checking user 'nfast' is in correct group 'nfast' users created correctly -- Running install fragment 15 makefiles Setting up directories. Making default config file. Making default cardlist file -- Running install fragment 45 drivers Unloading old nCipher PCI nfp driver. Checking for PCI nfp hardware. Warning: No suitable pre-built PCI driver available. No nCipher PCI nfp devices found. Installing startup scripts for 'drivers'. Not linking in init scripts or loading drivers. -- Running install fragment 50 hardserver Making privconn setuid and root. Configuring hardserver privileges. cap_net_bind_service capability set on hardserver file. ls: cannot access /dev/nfastpci*: No such file or directory Installing startup scripts for 'hardserver'. Linking in init scripts Warning: Installed, but no directly attached hardware was found. If you have an nCipher PCI card , re-run 'install' script with hardware attached, or with '-d' option, or consult nCipher support. Starting nCipher 'hardserver' server process. waiting for nCipher server to become operational ... nCipher server now running -- Running install fragment 60 cmdadp -- Running install fragment 70 edgecfg -- Running install fragment 90 ncsnmpd Checking for user 'ncsnmpd' in group 'ncsnmpd' Creating ncsnmpd group. Creating ncsnmpd user. useradd: warning: the home directory already exists. Not copying any file from skel directory into it. Checking user 'ncsnmpd' is in correct group 'ncsnmpd' users created correctly Installing startup scripts for 'ncsnmpd'. Creating wait script for 'ncsnmpd' Linking in init scripts Starting nCipher 'ncsnmpd' server process. 'ncsnmpd' server now running ---- Installation complete ----
- You should be able to run the ‘enquiry’ command now, which will be in your new path:
Server: enquiry reply flags none enquiry reply level Six serial number 1234-abcd-1234 mode operational version 12.80.4 speed index 4512 rec. queue 422..622 level one flags Hardware HasTokens SupportsCommandState version string 12.80.4-274-813026a, 12.80.2-140-b43b1b3, 12.80.4-0-f6c0b92 checked in 00000000000aaaaa Tue Nov 9 16:43:56 2021 level two flags none max. write size 8192 level three flags KeyStorage level four flags OrderlyClearUnit HasRTC HasNVRAM HasNSOPermsCmd ServerHasPollCmds FastPollSlotList HasSEE HasKLF HasShareACL HasFeatureEnable HasFileOp HasPCIPush HasLongJobs ServerHasLongJobs AESModuleKeys NTokenCmds JobFragmentation LongJobsPreferred Type2Smartcard ServerHasCreateClient HasInitialiseUnitEx Type3Smartcard HasKLF2 module type code 0 product name nFast server device name EnquirySix version 8 impath kx groups feature ctrl flags none features enabled none version serial 0 level six flags none remote port (IPv4) 9004 kneti hash 0000000000000000000000000000000000000000 rec. LongJobs queue 0 SEE machine type None supported KML types active modes none remote port (IPv6) 9004
- The Thales nShield Connect needs a Remote File System (RFS) server where the nShield will store and access critical files. It can also use the RFS as an optional target for syslog data. The RFS can be a client that is also participating in transaction processing, or is an HSM client, but that is not mandatory. I personally recommend a third option. Have an additional system that is configured identically to one of your transactional servers, with full nShield client and RFS, but is not sent any transactions. The reason for this is that if a transactional client is also the RFS for your nShield Connects, and even if you have redundant nShield Connects, you will still take a service affecting outage when upgrading firmware. If you use a third system that is running the full client install, you can upgrade your standby nShield Connect first, push config changes to your active clients, they begin using it, then upgrade what is now the standby nShield Connect, before switching back. You can’t do this if your RFS is also processing transactions, nor can you do this is you have a system being solely the RFS without the normal full client install. The system we just installed the full client software on is set to be an RFS as-is.
- After the above install, your system is now listening for connections on all interfaces to port 9004. I recommend adjusting the relevant firewall settings on the server to limit that to just the IP addresses of nShield Connects.
- You should be able to hook a USB keyboard to your nShield Connect now, unless you’re a sadist and want to enter all the settings using four menu buttons. You’ll want to enter the relevant IPv4/IPv6 addresses, gateway, subnet mask.
- From the client side, if your network is secure between the client and the nShield Connect, run the ‘anonkneti’ utility against the IP of your nShield Connect to obtain its electronic serial number (ESN, which differs from the physical serial); for example:
anonkneti 192.0.2.1
The output will look like the following:
A285-4F5A-7500 2418ec85c86027eb2d5959fef35edc5e1b3b698f
- Now you’ll use the rfs-setup utility to initialize the RFS on this specific client and permit your nShield to connect to it:
# rfs-setup 192.0.2.1 A285-4F5A-7500 2418ec85c86027eb2d5959fef35edc5e1b3b698f Adding read-only remote_file_system entries Ensuring the directory /opt/nfast/kmdata/local exists Adding new writable remote_file_system entries Ensuring the directory /opt/nfast/kmdata/hsm-A285-4F5A-7500 exists Ensuring the directory /opt/nfast/femcerts exists Ensuring the directory /opt/nfast/kmdata/hsm-A285-4F5A-7500/features exists Ensuring the directory /opt/nfast/kmdata/hsm-A285-4F5A-7500/config exists Ensuring the directory /opt/nfast/log/hsm-A285-4F5A-7500 exists Saving the new config file and configuring the hardserver Done
- Enroll this client to the HSM with privileged command access since this is the RFS and client we’ll use for pushing firmware and config to the HSM(s):
# nethsmenroll -p 192.0.2.1 A285-4F5A-7500 2418ec85c86027eb2d5959fef35edc5e1b3b698f
OK configuring hardserver's nethsm imports
If this were being performed on a regular transactional system, you’d leave the -p flag off so it does not get privileged access to the HSM(s).
Now, just for the sake of mentioning it, you need to be very very sure you always have a backup copy of the /opt/nfast/kmdata/ directory structure from this particular client since it is the RFS and contains critical data specific to the security world you’ll be creating; i.e. your encryption data. If your HSM dies, or you add more, you need your administrator card set (smart cards) AND the kmdata directory, to add the same security world to another device.
- From the front panel of your Connect, set it to auto update config and give this system’s address. This will let you make some types of changes to the HSM config remotely. Otherwise, for added security or if subject to requirements that do not permit it, you will have to initiate a pull from the front panel, possibly with an operator card or cards.
- Enroll this client to the Connect via front panel. If you use nToken for auth, you can enable that as well, and confirm your nToken credentials spit out by
ntokenenroll -H
on the client side. If not, choose no, and add.
- The ‘enquiry’ command should spit out details of your Connect now:
Server: enquiry reply flags none enquiry reply level Six serial number 1234-abcd-1234 abcd-1234-abcd mode operational version 12.80.4 speed index 9024 rec. queue 842..1194 level one flags Hardware HasTokens SupportsCommandState version string 12.80.4-274-813026a, 12.80.2-140-b43b1b3, 12.80.4-0-f6c0b92, 12.80.2-140-b43b1b3, 12.80.4-0-f6c0b92 checked in 00000000000aaaaa Tue Nov 9 16:43:56 2021 level two flags none max. write size 8192 level three flags KeyStorage level four flags OrderlyClearUnit HasRTC HasNVRAM HasNSOPermsCmd ServerHasPollCmds FastPollSlotList HasSEE HasKLF HasShareACL HasFeatureEnable HasFileOp HasPCIPush HasLongJobs ServerHasLongJobs AESModuleKeys NTokenCmds JobFragmentation LongJobsPreferred Type2Smartcard ServerHasCreateClient HasInitialiseUnitEx Type3Smartcard HasKLF2 module type code 0 product name nFast server device name EnquirySix version 8 impath kx groups feature ctrl flags none features enabled none version serial 0 level six flags none remote port (IPv4) 9004 kneti hash 0000000000000000000000000000000000000000 rec. LongJobs queue 0 SEE machine type None supported KML types active modes none remote port (IPv6) 9004 Module #1: enquiry reply flags Privileged enquiry reply level Six serial number abcd-1234-abcd mode operational version 12.80.2 speed index 4512 rec. queue 19..152 level one flags Hardware HasTokens SupportsCommandState version string 12.80.2-140-b43b1b3, 12.80.4-0-f6c0b92 checked in 00000000000aaaaa Wed Aug 18 15:14:37 2021 level two flags none max. write size 8192 level three flags KeyStorage level four flags OrderlyClearUnit HasRTC HasNVRAM HasNSOPermsCmd ServerHasPollCmds FastPollSlotList HasSEE HasKLF HasShareACL HasFeatureEnable HasFileOp HasPCIPush HasLongJobs ServerHasLongJobs AESModuleKeys NTokenCmds JobFragmentation LongJobsPreferred Type2Smartcard ServerHasCreateClient HasInitialiseUnitEx Type3Smartcard HasKLF2 module type code 7 product name nC2023E/nC3423E/nC4033E device name Rt2 EnquirySix version 7 impath kx groups DHPrime1024 DHPrime3072 DHPrime3072Ex DHPrimeMODP3072 feature ctrl flags LongTerm features enabled StandardKM EllipticCurve ECCMQV AcceleratedECC version serial 29 connection status OK connection info esn = abcd-1234-abcd; addr = INET/192.0.2.1/9004; ku hash = 0000000000000000000000000000000000000000, mech = Any image version 12.80.4-241-f6c0b92 level six flags none max exported modules 100 rec. LongJobs queue 18 SEE machine type PowerPCSXF supported KML types DSAp1024s160 DSAp3072s256 using impath kx grp DHPrimeMODP3072 active modes none hardware status OK
- Depending on how you’ve licensed your nShield’s, you may ultimately need to remove this HSM from your non-transactional systems because they default to only allowing three clients to connect, and if you have three transactional servers, you’ll have no spares to keep the privileged system connected. That can create issues pushing privileged commands to them since those have to come from a privileged system.
If this situation affects you, and you cannot purchase a 100 client license pack (the next level up), you’ll be forced to:
- Complete normal config management and transactional client enrollment from this privileged system
- The last config change will be to replace the privileged client system with your final transactional unprivileged system
- Push the config and confirm it’s written back to the RFS. This is of course dramatically easier if you’ve defined this system to allow config pushes in the config_op section of the nshield config file. If you do not define that, you’d need to physically pull a new config from the front panel, which may be impossible if you’ve also locked the front panel. So there’s a bunch of factors to consider in how you build your environment.
- In the future if privileged commands must be sent, remove a transactional client from production, update the config to replace that system with this privileged client, push it back, run the commands, undo these changes.
- If you see “unsupported firmware” in the above, your nShield probably came with older firmware than the client software. No problem; you extracted what you need earlier, so find it using your RFS server’s IP (not the nShield):
# nethsmadmin --list-images 192.0.2.2 Initiating RFS nethsm image check on 192.0.2.2 Checking the nethsm-firmware directory on the RFS. nethsm-firmware/12.23.1cam3/nCx3N.nff nethsm-firmware/12.23.1cam3-fips/nCx3N.nff Images were successfully found on the RFS (192.0.2.2).
Be careful before proceeding though. If you’re referring to this article for reinstallation purposes, and you have an existing Security World on the HSM in question, see the below warning:
I cannot stress eough the importance of being careful here. Updating the firmware on an nCipher / Thales nShield HSM breaks the Security World installed on it, so if you accidentally flash firmware on a production unit, you’ll take that unit down until you can reinstall the Security World via your administrative card set, which could mean a complete outage, and/or travel if the HSM is not local and you don’t use remote admin cards.
- Now, if you enabled remote upgrade via the front panel of your nShield Connect, and are coming from a privileged client, upgrade it:
nethsmadmin -i nethsm-firmware/12.23.1cam3/nCx3N.nff
- Otherwise, upgrade it from the front panel; it will be able to see the images on the RFS. A word of warning; it takes a LONG time to upgrade. You’ll be staring at a particular downloading file screen on the HSM for what feels like forever; don’t get antsy and pull the plug. Give it a solid 30 minutes. Additionally, after upgrade, the front panel interface will be VERY slow, and if you’re pressing, thinking it didn’t register, and pressing some more, you could easily perform an unintended operation, so be very very slow and patient with the front interface after upgrade. I suspect it’s doing something internally, because even after rebooting it like the user manual says, it’s still slow at the 60 minute post-upgrade mark. I’ll update later if it speeds back up.
- Time to create a security world. Put the HSM into pre-init mode via front panel, or privileged client, where the number of the module is taken from the enquiry command:
nopclearfail -I -m 1
- Once it boots into that mode, enquiry will show ‘pre-initialization’ status. You can create the world now, obviously with arguments that will be specific to your use case and security requirements:
new-world -i -c DLf3072s256mRijndael --acs-quorum=1/1
- This above example creates a simple world using a cipher suite compliant with SP800-131 and having, and requiring, just a single smart card in the admin key set. The utility will do some init stuff:
Create Security World: Module 2: 0 cards of 1 written Module 2 slot 0: empty
and you won’t realize it, but it wants you to input the card in the front of the unit now. Do that, then enter a pass phrase. In my ssh session, since they must use some kind of shell script loop for input, it was flashing like crazy and was very difficult to see what it wanted me to do. It was then showing “(Wait…)” for a few minutes after typing the pass phrase. I ended up giving up, reboot the client computer, tried again, everything went smoothly:
# new-world -i -c DLf3072s256mRijndael --acs-quorum=1/1 Create Security World: Module 1: 0 cards of 1 written Module 1 slot 0: empty Module 1 slot 0: unknown card Module 1 slot 0:- passphrase specified - overwriting card Card writing complete. security world generated on module #0
- If you add a second HSM and need to copy your security world to it:
new-world -l -m 2
where -m 2 specifies the 2nd module from the enquiry command.
Connecting additional HSM clients to both the HSM(s) and RFS server:
- Perform nearly the same nfast software setup on the new client with the exception of the SNMP and firmware file extraction since you’d only be using those on your RFS server where you’d be performing monitoring and upgrade operations against the HSM(s). Run the ./install after extraction. Run ‘enquiry’ after install to ensure this local client is good to go.
- Time to connect the client to the HSMs. Add this client’s IP to the HSM via front panel, or, beyond the scope of this article, you could copy the file /opt/nfast/kmdata/hsm-ESN/config/config on your RFS server to ‘config.new’ in the same directory, add the new clients to the hs_clients section, then use the cfg-pushnethsm command to push the updated config file to your HSMs. Just be careful to not screw it up because pushing a bad config to the HSM could leave your RFS without access to it and requiring a trip to the data center to re-authorize the RFS server as a client. If the push is successful, the HSM will also immediately write back the original config file. You should compare them with diff to ensure it happened successfully. The command would look something like
cfg-pushnethsm --address=192.0.2.1 -f -n /opt/nfast/kmdata/hsm-ESN/config/config.new
- With the client’s IP authorized on the HSM, run this from the new client:
nethsmenroll 192.0.2.1 `anonkneti 192.0.2.1`
- This will enroll the client to the HSM located at IP 192.0.2.1, using the serial number generated by the command “anonkneti 192.0.2.1”. Only do this if the client and HSM talk across a known-secure network. You can enter the serial info manually otherwise.
- On your RFS server, authorize the client IP’s to connect to it without write authorization:
rfs-setup --gang-client --write-noauth 192.0.2.10
where 192.0.2.10 is the IP of the new RFS client #1.
- On the new RFS client #1, give it unauthenticated access to the RFS via:
rfs-sync --setup --no-authenticate 192.0.2.2
where the 192.0.2.2 is the IP of the RFS server.
- If everything was done correctly, you should now be able to run this on the new client:
rfs-sync -U
and it will output the fact that it’s pulling down the config data for the HSM’s, the security world and any previously generated hardware keys located on the RFS server.
- Your client is now connected to the HSM’s and ready to perform operations involving HSM keys.
Good comprehensive guide. Thanks.
My RFS setup always fails… have you run across this issue. everything else works.. Enquiry cmd shows module operational and ok. However, whenever, i try to list the local client ip as the rfs on the HSM.. it spits out error failed to connect to RFS… any suggestion would be great
Hi Michael, any chance your RFS is RHEL/CentOS 7? That would have firewalld enabled by default and would deny the connection even if the software were listening. That’s the only thing I can think of off-hand; rfs-setup has always worked for me.
yes… the operating systems firewall was the culprit. Thanks again Mom :)
Thanks for the article. how do we uninstall SNMP agent that installed on Ubuntu Linux?