Difference between revisions of "Diskless netboot"

 
(37 intermediate revisions by the same user not shown)
Line 1: Line 1:
=Context and aim=
+
[[Category:Linux]]
  
==Aim==
 
  
We aim to use the same distribution everywhere. The client doesn't need any hard drive to run, they will retrieve the file system from the TFTP server and use a NFS share as hard drive. The system will only run in RAM disk.
+
==Target folder tree (server side)==
 
 
 
 
So:
 
 
 
* All the clients will share the installation, configuration files and so on.
 
 
 
* Each client will run a dedicated instance of the operating system in his own RAM disk
 
 
 
* Logs will be centralized on the common NFS server - so we don't loose data on each reboot.
 
 
 
* The user will be able to choose the O.S to run on boot - thanks to a PXE menu
 
 
 
 
 
 
 
Requirements:
 
* [[DHCP server]]
 
* [[NetBoot server]]
 
 
 
 
 
Optional:
 
* [[DNS server]]
 
 
 
 
 
==Target folder tree==
 
  
 
This is how we'll setup our files and folders:
 
This is how we'll setup our files and folders:
  
 
<syntaxhighlight lang="bash">
 
<syntaxhighlight lang="bash">
 
 
# TFTP root
 
# TFTP root
/tftpboot/                                
+
/tftpboot/                                  
 
 
# Ubuntu installation NetBoot disk
 
/tftpboot/rescue/                         
 
  
 
###############
 
###############
Line 58: Line 29:
 
/tftpboot/images/wheezy/initrd.img
 
/tftpboot/images/wheezy/initrd.img
  
# Ubuntu 14.04 [Trusty]  
+
# [X]Ubuntu 14.04 [Trusty]  
 
/tftpboot/images/trusty/   
 
/tftpboot/images/trusty/   
 
/tftpboot/images/trusty/vmlinuz
 
/tftpboot/images/trusty/vmlinuz
Line 72: Line 43:
  
 
# Ubuntu 14.04 [Trusty]  
 
# Ubuntu 14.04 [Trusty]  
/nfs/trusty/                  
+
/nfs/trusty/
 
 
</syntaxhighlight>
 
 
 
 
 
 
 
 
 
=Installation=
 
 
 
 
 
'''NFS support'''
 
<syntaxhighlight lang="bash">
 
apt-get install nfs-kernel-server nfs-common
 
</syntaxhighlight>
 
 
 
'''Debootstrap (manage netboot image)'''
 
<syntaxhighlight lang="bash">
 
apt-get install debootstrap
 
</syntaxhighlight>
 
 
 
 
 
'''Initramfs (to manage "virtual disks")'''
 
<syntaxhighlight lang="bash">
 
apt-get install initramfs-tools
 
</syntaxhighlight>
 
 
 
 
 
 
 
 
 
 
 
=Preparation=
 
 
 
You have to create a dedicated folder on your server where you will host the distributions kernels + Boot settings.
 
 
 
<syntaxhighlight lang="bash">
 
mkdir -p /tftpboot/pxelinux.cfg
 
chmod -R 755 /tftpboot/pxelinux.cfg
 
 
 
mkdir -p /tftpboot/images
 
chmod -R 755 /tftpboot/images
 
</syntaxhighlight>
 
 
 
 
 
The ''pxelinux.cfg/'' folder is mandatory. Inside you can provide:
 
* configuration for a specific IP @ or hostname
 
* configuration for a group
 
* default configuration (required)
 
 
 
=Boot menu and Kernel setup=
 
 
 
The first thing to do is to setup a booting kernel. To do so we'll use the "syslinux" files.
 
 
 
 
 
==Root file: pxelinux.0==
 
 
 
The '''pxelinux.0''' is the root file. That's the file that allows the netboot.
 
 
 
This is the file that is serve by the TFTP server.
 
 
 
<syntaxhighlight lang="bash">
 
cp /usr/lib/syslinux/pxelinux.0 /tftpboot/
 
</syntaxhighlight>
 
 
 
 
 
==Create NetBoot menu | defaults==
 
 
 
Now, we have to specify which kernel to use and which distributions are available for NetBoot.
 
 
 
 
 
Create the default configuration file:
 
 
 
<syntaxhighlight lang="bash">
 
vim /tftpboot/pxelinux.cfg/default
 
</syntaxhighlight>
 
 
 
 
 
Put the following:
 
 
 
<syntaxhighlight lang="bash">
 
# Debian 7.x
 
LABEL wheezy
 
    kernel images/wheezy/vmlinuz
 
    initrd images/wheezy/initrd.img
 
 
 
# Ubuntu 14.04
 
LABEL trusty
 
    kernel images/trusty/vmlinuz
 
    initrd images/trusty/initrd.img
 
 
 
 
 
# Prompt user for selection
 
PROMPT 1
 
# No timeout
 
TIMEOUT 0
 
</syntaxhighlight>
 
 
 
* Each LABEL is a specific configuration that will displayed on the NetBoot menu.
 
* PROMPT 0 = enable user prompt so you can choose the configuration
 
* TIMEOUT 0 = timeout (in seconds) before the default option is chosen. 0 == no timeout
 
 
 
 
 
 
 
Note that I used a reference to "trusty/", that's a folder I need to create later on.
 
 
 
 
 
 
 
==Init Kernel files==
 
 
 
 
 
===Create directories===
 
 
 
Create the target kernel folders. You should create 1 folder for each distribution you'd like to provide in NetBoot.
 
 
 
<syntaxhighlight lang="bash">
 
# Debian 7.x
 
mkdir -p /tftpboot/images/wheezy
 
 
 
# Ubuntu 14.04
 
mkdir -p /tftpboot/images/trusty
 
</syntaxhighlight>
 
 
 
 
 
===Prepare ''initramfs'' to boot over NFS===
 
 
 
'''This step must to be run on the machine that has the kernel you are going to serve to your clients'''.
 
 
 
 
 
>>> In our case it has to be run on the TFTP server
 
 
 
 
 
 
 
Copy initramfs settings for PXE boot
 
 
 
<syntaxhighlight lang="bash">
 
cp -r /etc/initramfs-tools /etc/initramfs-pxe
 
</syntaxhighlight>
 
 
 
 
 
Adjust PXE boot configuration
 
 
 
<syntaxhighlight lang="bash">
 
cd /etc/initramfs-pxe/
 
vim /etc/initramfs-pxe/initramfs.conf
 
 
 
#### Change the following options
 
BOOT=nfs
 
MODULE=netboot
 
</syntaxhighlight>
 
 
 
 
 
===Copy and prepare kernel===
 
 
 
 
 
You have to copy your current kernel files to the boot folder:
 
 
 
<syntaxhighlight lang="bash">
 
# Debian 7.x
 
cp /boot/vmlinuz-`uname -r` /tftpboot/images/wheezy/vmlinuz
 
cp /boot/initrd.img-`uname -r` /tftpboot/images/wheezy/initrd.img
 
 
 
# Ubuntu 14.04
 
cp /boot/vmlinuz-`uname -r` /tftpboot/images/trusty/vmlinuz
 
cp /boot/initrd.img-`uname -r` /tftpboot/images/trusty/initrd.img
 
</syntaxhighlight>
 
 
 
 
 
 
 
Enable NFS boot on target kernel:
 
 
 
<syntaxhighlight lang="bash">
 
mkinitramfs -d /etc/initramfs-pxe -o /tftpboot/images/trusty/initrd.img
 
</syntaxhighlight>
 
 
 
 
 
Adjust rights:
 
 
 
<syntaxhighlight lang="bash">
 
chmod -R 755 /tftpboot/images/
 
</syntaxhighlight>
 
 
 
 
 
 
 
Notes:
 
 
 
* Do NOT use some symlink for "vmlinuz" and "initrd.img" !! It won't work.
 
 
 
* If you don't want to use `uname -r` [current kernel version and architecture] then adjust the values to target kernel number + architecture
 
 
 
* You have to run ''mkinitramfs'' for each kernel you'll provide
 
 
 
* Don't forget to adjust the rights to 755 for every distribution
 
 
 
 
 
 
 
 
 
=NFS server setup=
 
 
 
 
 
==Configuration==
 
 
 
The NFS configuration is done in the '''/etc/exports''' file
 
 
 
<syntaxhighlight lang="bash">
 
vim /etc/exports
 
</syntaxhighlight>
 
 
 
 
 
Add something like that:
 
 
 
<syntaxhighlight lang="bash">
 
  /nfs    192.168.2.0/24(ro,no_root_squash,no_subtree_check,async,insecure)
 
 
</syntaxhighlight>
 
</syntaxhighlight>
  
  
Adjust "192.168.2.0/24" to your own network address
 
 
* rw : Allow clients to read as well as write access
 
* ro : Read only access
 
* insecure : Tells the NFS server to use unpriveledged ports (ports > 1024).
 
* no_subtree_check : If the entire volume (/users) is exported, disabling this check will speed up transfers.
 
* async : async will speed up transfers.
 
* no_root_squash: This phrase allows root to connect to the designated directory.
 
 
 
- NOTE -
 
 
* It's always a good idea to use Read-Only if you plan to share this disk. That will avoid user to mess with your image!
 
 
* There must not be any space between network IP and "("
 
 
 
 
==Security==
 
  
Like TFTP, this part is insecure !
+
==Client overview==
  
You must restrict the access to your NFS server by a firewall script and filtering BEFORE reaching the LAN !
+
Each client must have, at least, 4 Go of RAM.
  
  
 +
===4 GO RAM configuration===
  
NFS is using dynamic ports numbers because it runs over '''rpcbind'''. Making NFS using specifics port is a pain in the ass !! :(
+
This is how we're gonna populate the client:
  
So, instead of that you should allow your LAN communication.  
+
[[File:Client_composition.png|480px|NetBoot client RAM overview - 4Go]]
  
  
<syntaxhighlight lang="bash">
+
As you can see, each client will have some space dedicated for swap + some RAMdisk to allow writing in /var, /tmp and /proc.
    IPTABLES=`which iptables`
 
    LAN_ADDRESS="192.168.2.0/24"
 
  
    # Allow LAN communication
 
    $IPTABLES -A INPUT -s $LAN_ADDRESS -d $LAN_ADDRESS -m state ! --state INVALID -j ACCEPT
 
    $IPTABLES -A OUTPUT -s $LAN_ADDRESS -d $LAN_ADDRESS -m state ! --state INVALID -j ACCEPT
 
</syntaxhighlight>
 
  
 +
Configuration of a '''4Go RAM''' disk:
 +
* No swap
 +
* Local TMPFS (read/write for /dev, /tmp, ...) : 1 Go
 +
** /tmp      = 512 M
 +
** /var/tmp  = 128 M
 +
** /var/log  = 128 M
 +
** /var/run  = 8 M
 +
** /var/lock = 8 M
 +
** /run/shm  = 256 M
 +
* O.S (NFS read only) : all the rest ~ 2.8 Go
 +
* Common share (NFS read write) : ''Remote disk''
  
==Management==
 
 
<syntaxhighlight lang="bash">
 
service nfs-kernel-server {status|start|stop|restart}
 
</syntaxhighlight>
 
  
  
==Test the server==
+
===2 Go===
  
 +
Due to budget restriction we might encounter some low memory machines with only 2 Go...
  
Install the NFS v4 client:
 
<syntaxhighlight lang="bash">
 
apt-get install nfs-common
 
</syntaxhighlight>
 
  
 +
This is how we're gonna populate the client:
  
To mount the default path:
+
[[File:Client_mount_points_2Go.png|480px|NetBoot client RAM overview - 2Go]]
<syntaxhighlight lang="bash">
 
mount -t nfs nfs-server:/ /mnt
 
</syntaxhighlight>
 
  
You'll see: "/nfs"
 
  
 +
In case of '''2Go RAM''' then you have to use some tricks:
 +
* No swap
 +
* O.S (NFS read only) : ~ 1.2 Go
 +
* Common share (NFS read write) : ''Remote disk''
 +
* Local TMPFS (read/write for /dev, /tmp, ...) : all the rest
 +
** /tmp      = 372 M
 +
** /var/tmp  = auto
 +
** /var/log  = 128 M
 +
** /var/run  = auto
 +
** /var/lock = auto
 +
** /run/shm  = auto
  
It's better to do:
 
<syntaxhighlight lang="bash">
 
mount -t nfs nfs-server:/nfs /mnt
 
</syntaxhighlight>
 
  
  
 +
==How big is the client image ?==
  
 +
By default the ''deboostrap'' Ubuntu 14.04 LTS image is 239 Mo. With the applications we're gonna use that size will increase to about 1 or '''1.3 Go''' depending if you copy (or not) the kernel sources. It may even take 1.6 Go if you're using XFCE frontend.
  
  
Line 368: Line 114:
  
 
* Manually
 
* Manually
 
 
** debootstrap
 
** debootstrap
 
 
** copying the install from your server
 
** copying the install from your server
 
 
** Manual install on a client, then, when the system is ready, copy everything to the NFS share
 
** Manual install on a client, then, when the system is ready, copy everything to the NFS share
 
  
 
* Using script and software like "Puppet" or "Chef"
 
* Using script and software like "Puppet" or "Chef"
Line 387: Line 129:
 
mkdir -p /nfs/trusty
 
mkdir -p /nfs/trusty
 
mkdir -p /nfs/wheezy
 
mkdir -p /nfs/wheezy
 +
mkdir -p /nfs/common
 
</syntaxhighlight>
 
</syntaxhighlight>
  
Line 404: Line 147:
 
* Automatic [Puppet || Chef] configuration: [[Diskless image configuration - script setup]]
 
* Automatic [Puppet || Chef] configuration: [[Diskless image configuration - script setup]]
  
 
 
==Kernel modules and source==
 
 
If you're using a local kernel as the default NetBoot kernel, then you need to do copy the ''modules'' + ''kernel source'' to every distribution.
 
 
<syntaxhighlight lang="bash">
 
# Copy kernel modules
 
cp -r /lib/modules/`uname -r` /nfs/trusty/lib/modules
 
 
# Copy kernel sources
 
cp -r /usr/src/linux-headers-`uname -r` /nfs/trusty/usr/src
 
</syntaxhighlight>
 
 
 
Note that '''you have to adjust the /nfs'''/XXX
 
  
  
Line 444: Line 171:
 
</syntaxhighlight>
 
</syntaxhighlight>
  
 +
=PXE interactive menu=
  
 +
You can create interactive NetBoot menus, see:
 +
* [[PXE interactive menu - single level]]
 +
* [[PXE interactive menu - multi level]]
  
 
 
=Custom NetBoot configuration=
 
 
 
==Basic menu==
 
 
See the previous § to learn how to setup a super simple boot menu without any interaction.
 
 
 
 
==Text menu==
 
 
The text menu required a new file: '''menu.c32'''
 
 
<syntaxhighlight lang="bash">
 
cp /usr/lib/syslinux/menu.c32 /tftpboot/pxelinux.cfg/
 
</syntaxhighlight>
 
  
  
  
==Graphic menu==
+
=Local server monitoring=
  
The graphic menu required a new file: '''vesamenu.c32'''
+
Install the following services:
 +
* [[SNMP client]]
 +
* [[Zabbix agent setup]]
  
<syntaxhighlight lang="bash">
 
cp /usr/lib/syslinux/vesamenu.c32 /tftpboot/pxelinux.cfg/
 
</syntaxhighlight>
 
  
  
Then you can add an image file to display as menu background.
 
  
Remember:
+
=Other services=
* You'll have to specify the exact size of the picture later on - so note the picture size !! Recommendation: '''640x480'''
 
* Remember that the SysLinux cannot display more than depth=16 and 65536 colors
 
  
  
ImageMagic command to convert a picture:
+
==File sharing==
  
<syntaxhighlight lang="bash">
+
If you want to expose the NFS common folder as a file-share, you have to install and configure Samba. See: [[Samba server]]
convert -depth 16 -colors 65536 my_custom_image.png splash.png
 
</syntaxhighlight>
 
  
  
Then copy the splash:
+
''Note''
  
<syntaxhighlight lang="bash">
+
Samba is actually better than NFS for the file-share. You should remove Common from /etc/exports and use a samba share instead.  
cp splash.png /tftpboot/pxelinux.cfg/
 
chown -R root:root /tftpboot/pxelinux.cfg/splash.png
 
</syntaxhighlight>
 
  
  
  
 +
==Management UI (webmin)==
  
==Configure / setup menu content==
+
Since there is a lot of services to manage, it's always convenient to use an UI for it. Check [[Webmin]]
  
Edit the PXE NetBoot file:
 
  
<syntaxhighlight lang="bash">
 
vim /tftpboot/pxelinux.cfg/default
 
</syntaxhighlight>
 
  
 +
==VPN server==
  
Put:
+
See [[VPN]]
  
<syntaxhighlight lang="bash">
 
#### GENERIC OPTIONS #####
 
# Enable text menu
 
#DEFAULT pxelinux.cfg/menu.c32
 
# Enable graphical menu
 
DEFAULT pxelinux.cfg/vesamenu.c32
 
# Prompt for user input? (0 = choose from menu, 1 = you can type anything)
 
PROMPT 0
 
# Allow or not the user to left the menu (1 = user is locked to the menu)
 
NOESCAPE 1
 
# Time before using default option
 
TIMEOUT 50
 
  
  
#### Menu settings #####
+
==Apache2 server==
MENU TITLE my super netboot menu
 
# Setup background image (the resolution is $WIDTH $HEIGHT and must match the given picture)
 
MENU RESOLUTION 640 480
 
MENU BACKGROUND pxelinux.cfg/splash.png
 
# Setup text properties
 
MENU WIDTH 80
 
MENU ROWS 14
 
MENU MARGIN 10
 
  
 +
See [[Apache 2]]
  
#### Distributions #####
 
# Debian wheezy
 
LABEL wheezy
 
    MENU LABEL Debian 7.x [wheezy]
 
    MENU DEFAULT
 
    # Kernel and boot files
 
    KERNEL images/wheezy/vmlinuz
 
    APPEND boot=nfs root=/nfs/wheezy initrd=images/wheezy/initrd.img nfsroot=192.168.2.2:/nfs/wheezy
 
  
# Ubuntu 14.04
 
LABEL trusty
 
    MENU LABEL Ubuntu 14.04 [trusty]
 
    # Kernel and boot files
 
    KERNEL images/trusty/vmlinuz
 
    APPEND boot=nfs root=/nfs/trusty initrd=images/trusty/initrd.img nfsroot=192.168.2.2:/nfs/trusty
 
 
</syntaxhighlight>
 
 
 
Note all the "MENU" commands + PROMPT 0
 
 
 
More about the menu configuration: http://www.syslinux.org/doc/menu.txt
 
 
 
 
 
 
 
=Security notes=
 
 
in order to work you must adjust the rights of your "/var/lib/tftpboot/".
 
 
<syntaxhighlight lang="bash">
 
chmod 777 /pxe-boot/*
 
chmod 777 /pxe-boot/pxelinux.cfg/*
 
</syntaxhighlight>
 
 
 
 
 
=DHCP note=
 
 
Don't forget to adjust your DHCP configuration if you plan to serve a specific file for a client!
 
 
<syntaxhighlight lang="bash">
 
vim /etc/dhcp/dhcp.conf
 
</syntaxhighlight>
 
  
  
  
 +
=References=
  
 +
Ubuntu diskless how-to: https://help.ubuntu.com/community/DisklessUbuntuHowto
  
=References=
 
  
Ubuntu diskless how-to: https://help.ubuntu.com/community/DisklessUbuntuHowto
+
Mind reference: http://mindref.blogspot.se/2011/03/debian-diskless.html
  
  
Line 598: Line 235:
 
* http://www.stepladder-it.com/bivblog/14/ to /16/
 
* http://www.stepladder-it.com/bivblog/14/ to /16/
 
* https://blog.dlasley.net/2013/01/pxe-server-ubuntu/
 
* https://blog.dlasley.net/2013/01/pxe-server-ubuntu/
 +
 +
 +
Nice explanation of PXE process: http://www.linux.com/learn/docs/ldp/497-Diskless-root-NFS-HOWTO
 +
 +
* How to improved /etc/fstab: http://www.askapache.com/optimize/super-speed-secrets.html

Latest revision as of 14:37, 21 August 2014


Target folder tree (server side)

This is how we'll setup our files and folders:

# TFTP root
/tftpboot/                                   

###############
# Network bootable image(s) using NFS technology
################       

#### Boot file            
/tftpboot/pxelinux.0                   # Initial boot file - only use to load the PXE NetBoot manager
/tftpboot/{menu.c32 || vesamenu.c32}   # PXE interactive menu managers (text or graphical)
/tftpboot/pxelinux.cfg/                # PXE configuration(s)
/tftpboot/pxelinux.cfg/default         # default PXE configuration

#### Kernel file

/tftpboot/images/      

# Debian 7.x [Wheezy] 
/tftpboot/images/wheezy/   
/tftpboot/images/wheezy/vmlinuz
/tftpboot/images/wheezy/initrd.img

# [X]Ubuntu 14.04 [Trusty] 
/tftpboot/images/trusty/  
/tftpboot/images/trusty/vmlinuz
/tftpboot/images/trusty/initrd.img


#### NFS 
# This is where the runnable will be. Each image will be in a dedicated folder.
/nfs/                    

# Debian 7.x [Wheezy] 
/nfs/wheezy/   

# Ubuntu 14.04 [Trusty] 
/nfs/trusty/


Client overview

Each client must have, at least, 4 Go of RAM.


4 GO RAM configuration

This is how we're gonna populate the client:

NetBoot client RAM overview - 4Go


As you can see, each client will have some space dedicated for swap + some RAMdisk to allow writing in /var, /tmp and /proc.


Configuration of a 4Go RAM disk:

  • No swap
  • Local TMPFS (read/write for /dev, /tmp, ...) : 1 Go
    • /tmp = 512 M
    • /var/tmp = 128 M
    • /var/log = 128 M
    • /var/run = 8 M
    • /var/lock = 8 M
    • /run/shm = 256 M
  • O.S (NFS read only) : all the rest ~ 2.8 Go
  • Common share (NFS read write) : Remote disk


2 Go

Due to budget restriction we might encounter some low memory machines with only 2 Go...


This is how we're gonna populate the client:

NetBoot client RAM overview - 2Go


In case of 2Go RAM then you have to use some tricks:

  • No swap
  • O.S (NFS read only) : ~ 1.2 Go
  • Common share (NFS read write) : Remote disk
  • Local TMPFS (read/write for /dev, /tmp, ...) : all the rest
    • /tmp = 372 M
    • /var/tmp = auto
    • /var/log = 128 M
    • /var/run = auto
    • /var/lock = auto
    • /run/shm = auto


How big is the client image ?

By default the deboostrap Ubuntu 14.04 LTS image is 239 Mo. With the applications we're gonna use that size will increase to about 1 or 1.3 Go depending if you copy (or not) the kernel sources. It may even take 1.6 Go if you're using XFCE frontend.


NFS client image

There are different way to setup a NFS client image.

The main ones are:

  • Manually
    • debootstrap
    • copying the install from your server
    • Manual install on a client, then, when the system is ready, copy everything to the NFS share
  • Using script and software like "Puppet" or "Chef"


Setup client distribution

You have to create one target for each distribution you want to serve:

mkdir -p /nfs/trusty
mkdir -p /nfs/wheezy
mkdir -p /nfs/common


- NOTES -

  • The folder name should match your NetBoot settings. Folder name = a LABEL in the NetBoot config.
  • The folder name should match a Linux (Debian like) distribution name


Configure client distribution



Backup distribution

You can create an archive of your current distribution for later restore / re-use.


Compression

cd /nfs
tar cvpjf trusty.tar.bz2 ./trusty


Restoration

cd /nfs
tar -xvjf trusty.tar.bz2

PXE interactive menu

You can create interactive NetBoot menus, see:



Local server monitoring

Install the following services:



Other services

File sharing

If you want to expose the NFS common folder as a file-share, you have to install and configure Samba. See: Samba server


Note

Samba is actually better than NFS for the file-share. You should remove Common from /etc/exports and use a samba share instead.


Management UI (webmin)

Since there is a lot of services to manage, it's always convenient to use an UI for it. Check Webmin


VPN server

See VPN


Apache2 server

See Apache 2



References

Ubuntu diskless how-to: https://help.ubuntu.com/community/DisklessUbuntuHowto


Mind reference: http://mindref.blogspot.se/2011/03/debian-diskless.html


Super video tutorials:


Nice explanation of PXE process: http://www.linux.com/learn/docs/ldp/497-Diskless-root-NFS-HOWTO