Wednesday, November 19, 2014

continuous-integration-bamboo-on-ubuntu-14.04-with-tomcat7-libapr

Bamboo14.04 on 172.24.32.5 2vCPU 8GB RAM 16GB Disk
ubuntu / hipchat UTC single bootable / ext4 partition (no swap on a VM) as OpenSSH server
login; vi /etc/network/interfaces

 iface eth0 inet static
        address 172.24.32.128
        netmask 255.255.254.0
        network 172.24.32.0
        broadcast 172.24.32.255
        gateway 172.24.32.1

        dns-nameservers 172.24.32.10 172.24.32.11
        dns-search hipchat.atlassian.com

 # ensured that DNS Server is updated with the new DNS name bamboo.example.com and static ip address

 **sudo su**

    apt-get update; apt-get install libssl-dev openssl
 > for the many libssl and openssl security vulnerabilities

    /usr/sbin/useradd --create-home --home-dir /usr/local/bamboo --shell /bin/bash bamboo
    passwd bamboo
    ps a
    apt-get install ntp unzip unrar curl git-core openjdk-7-jdk
    git --version; java -version
    apt-get install tomcat7 

 apt-get install libtcnative-1

### Installing libtcnative manually

**Do this to ensure you are not running openssl and libssl with security vulnerabilities like heartbleed**

    apt-get install make
    https://apr.apache.org/download.cgi
    wget http://apache.claz.org//apr/apr-1.5.1.tar.gz
    tar -xf apr.1.5.1.tar.gz
    cd apr-1.5.1
    sudo ./configure ; sudo make ; sudo make install
    ls -ahl /usr/local/apr/lib
 > libapr-1.so.0.5.1

 http://tomcat.apache.org/download-native.cgi
    wget http://mirror.reverse.net/pub/apache/tomcat/tomcat-connectors/native/1.1.31/source/tomcat-native-1.1.31-src.tar.gz
    tar -xf tomcat-native-1.1.31-src.tar.gz
    cd tomcat-native-1.1.31-src/jni/native
    sudo ./configure --with-apr=/usr/local/apr --with-java-home=/usr/lib/jvm/java-7-openjdk-amd64 ; sudo make ; sudo make install
    ls -ahl /usr/local/apr/lib
 > libtcnative-1.so.1.31

    echo "export JAVA_HOME='/usr/lib/jvm/java-7-openjdk-amd64'" >> /usr/share/tomcat7/bin/setenv.sh
    echo "export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/usr/local/apr/lib' >> /usr/share/tomcat7/bin/setenv.sh

 chmod +x /usr/share/tomcat7/bin/setenv.sh
 > aka $CATALINA_HOME/bin/setenv.sh

### Configuring tomcat7 to use SSL

vi /var/lib/tomcat7/conf/server.xml
    

    

### removing annoying warnings

sudo ln -s /var/lib/tomcat7/common/ common
sudo ln -s /var/lib/tomcat7/server/ server
sudo ln -s /var/lib/tomcat7/shared/ shared

sudo /etc/init.d/tomcat7 restart

less /var/log/tomcat7/catalina.out


Jul 16, 2014 10:01:00 PM org.apache.catalina.core.AprLifecycleListener init
INFO: Loaded APR based Apache Tomcat Native library 1.1.31 using APR version 1.5.1.
Jul 16, 2014 10:01:00 PM org.apache.catalina.core.AprLifecycleListener init
INFO: APR capabilities: IPv6 [true], sendfile [true], accept filters [false], random [true].
Jul 16, 2014 10:01:01 PM org.apache.catalina.core.AprLifecycleListener initializeSSL
INFO: OpenSSL successfully initialized (OpenSSL 1.0.1f 6 Jan 2014)
Jul 16, 2014 10:01:01 PM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["http-apr-8080"]
Jul 16, 2014 10:01:01 PM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["http-apr-8443"]
Jul 16, 2014 10:01:01 PM org.apache.catalina.startup.Catalina load
INFO: Initialization processed in 702 ms
Jul 16, 2014 10:01:01 PM org.apache.catalina.core.StandardService startInternal
INFO: Starting service Catalina
Jul 16, 2014 10:01:01 PM org.apache.catalina.core.StandardEngine startInternal
INFO: Starting Servlet Engine: Apache Tomcat/7.0.52 (Ubuntu)
Jul 16, 2014 10:01:01 PM org.apache.catalina.startup.HostConfig deployDirectory
INFO: Deploying web application directory /var/lib/tomcat7/webapps/ROOT
Jul 16, 2014 10:01:02 PM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ["http-apr-8080"]
Jul 16, 2014 10:01:02 PM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ["http-apr-8443"]
Jul 16, 2014 10:01:02 PM org.apache.catalina.startup.Catalina start
INFO: Server startup in 1376 ms

https://bamboo.example.com:8443/

### Ruby
`sudo apt-get install ruby2.0`
`gem install rake`

### Bamboo
    https://www.atlassian.com/software/bamboo/download
    tar -xf atlassian-bamboo-5.5.1.tar.gz
    sudo mv atlassian-bamboo-5.5.1 /usr/local/bamboo
    sudo chown bamboo:bamboo /usr/local/bamboo/atlassian-bamboo-5.5.1
    sudo echo "bamboo.home=/usr/local/bamboo" >> /usr/local/bamboo/atlassian-bamboo-5.5.1/atlassian-bamboo/WEB-INF/classes/bamboo-init.properties


### MySQL
    sudo apt-get install mysql-server-5.5
 > enter a password for the default root user
 mysql -uroot -p

    > create an empty database and a bamboo user
    CREATE DATABASE bamboo CHARACTER SET utf8 COLLATE utf8_bin;
 GRANT ALL PRIVILEGES ON bamboo.* TO 'bamboouser'@'localhost' IDENTIFIED BY 'password';
 FLUSH PRIVILEGES;
 QUIT

    > download and install the mysql java connector from http://dev.mysql.com/downloads/connector/j/
    tar -xf mysql-connector-java-5.1.31.tar.gz
 cp -a mysql-connector-java-5.1.31/mysql-connector-java-5.1.31-bin.jar /usr/local/bamboo/atlassian-bamboo-5.5.1/lib/

vi /etc/init.d/bamboo

#!/bin/sh -e
# bamboo startup script
#chkconfig: 2345 80 05
#description: bamboo
 
# Define some variables
# Name of app ( bamboo, Confluence, etc )
APP=bamboo
# Name of the user to run as
USER=bamboo
# Location of application's bin directory
BASE=/usr/local/bamboo/atlassian-bamboo-5.5.1
# Location of Java JDK
export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64
 
case "$1" in
  # Start command
  start)
    echo "Starting $APP"
 $BASE/bin/start-bamboo.sh
    # /bin/su -m $USER -c "cd $BASE/logs && $BASE/bin/start-bamboo.sh &> /dev/null"
    ;;
  # Stop command
  stop)
    echo "Stopping $APP"
 $BASE/bin/stop-bamboo.sh
    $ /bin/su -m $USER -c "$BASE/bin/stop-bamboo.sh &> /dev/null"
    echo "$APP stopped successfully"
    ;;
   # Restart command
   restart)
        $0 stop
        sleep 5
        $0 start
        ;;
  *)
    echo "Usage: /etc/init.d/$APP {start|restart|stop}"
    exit 1
    ;;
esac
 
exit 0


`chmod +x /etc/init.d/bamboo`

cd /etc/init.d
update-rc.d bamboo defaults
> *fix some things*
> vi bamboo
> update-rc.d -f bamboo remove
> update-rc.d bamboo defaults


`/etc/init.d/bamboo start`

`tail -f /usr/local/bamboo/atlassian-bamboo-5.5.1/bin/logs/catalina.out`

### Bamboo Setup Wizard

http://myatlassian.com
> copy your license

http://bamboo1.example.com:8085/
> wait about 30 seconds

Setup Wizard: License -> Directories -> Database (bamboouser / password) -> administrator user
https://confluence.atlassian.com/display/BAMBOO/Running+the+Setup+Wizard


### More Info

**For parallel builds, add multiple local agents** , i.e. http://bamboo.example.com:8085/admin/agent/configureAgents!default.action

If you want to use submodules, e.g. `git clone --recursive git@bitbucket.org:USER/REPO.git`

> there is a checkbox for Repository Source but **HTTPS does not seem to work**

> RESOLUTION:

> For the user that runs bamboo (e.g. root)
> ensure there's an SSH key (/root/.ssh/id_rsa; chmod 400) is installed 
> git clone --recursive once to add the fingerprint of the remote git server to authorized

Sunday, November 16, 2014

Monitoring Your Windows Server Metrics with WMI

WMI Query Structure

Each WMI query is made in three phases as follows:
  1. Connecting to WMI on a local or remote computer
  2. Executing the query
  3. Reading the returned data
The query language is WMI Query Language (WQL) and is a subset of SQL.  A sample script to gather WMI data looks like the following:
computer = "."
Set oWMI = GetObject("winmgmts:{impersonationLevel=impersonate}!\\" + computer + "\root\cimv2")
Set oRes = oWMI.ExecQuery("select PercentDiskTime from Win32_PerfFormattedData_PerfDisk_PhysicalDisk where Name = '0 C:'")
For each oEntry in oRes
wscript.echo oEntry.PercentDiskTime
next

You can find a similar structure in the scripts describing how to create custom monitors and feed them with data.
Since WMI data is highly dependent on your system configuration (such as disk partitions or network adapter names), consider the sample queries in this article just as a sample, and be sure to verify how the data are structured on your particular system.
From here, you can verify your queries using WMI Explorer by clicking on the following link:  http://www.ks-soft.net/hostmon.eng/wmi/index.htm

CPU Monitoring

select PercentProcessorTime from Win32_PerfFormattedData_PerfOS_Processor where Name = '_Total'
“PercentProcessorTime,” as appears above, is the percentage of the time the processor is busy doing non-idle threads.

Physical Disk monitoring

select PercentDiskTime from Win32_PerfFormattedData_PerfDisk_PhysicalDisk where Name = '0 C:'
“PercentDiskTime,” as appears above, is the percentage of the time the selected physical disk is busy servicing read or write requests.

Memory monitoring

select AvailableBytes from Win32_PerfFormattedData_PerfOS_Memory
“AvailableBytes,” also featured above, is the amount of memory immediately available for allocation to a process or for system use.

Memory swap monitoring

select PagesPersec from Win32_PerfFormattedData_PerfOS_Memory
Is the current rate (“PagesPersec”) at which memory pages are read from or written to disk to resolve hard page faults. A high rate of memory operations involving disk swap are symptoms of memory shortage and affects system performance.

Free space on disks monitoring

select FreeSpace from Win32_LogicalDisk where DeviceID = 'C:'
“FreeSpace” is the available storage space in bytes on the specified logical disk.

Network monitoring

select BytesReceivedPersec, BytesSentPersec from Win32_PerfFormattedData_Tcpip_NetworkInterface where Name = 'D-Link DFE-538TX 10_100 Adapter'
“BytesReceivedPersec” and “BytesSentPersec” are the current transmission rates for the specified adapter.

TCP/IP re-transmissions

select SegmentsRetransmittedPersec from Win32_PerfFormattedData_Tcpip_TCPv4
“SegmentsRetransmittedPersec” is the current rate of TCP/IP segments re-transmission. It gives you an idea of how well your network connection is performing.

Server monitoring

select FilesOpen, ServerSessions from Win32_PerfFormattedData_PerfNet_Server
“FilesOpen” is the current number of files opened by users over the network.  “ServerSession” is the current number of users connected to the server.

TCP/IP Connections monitoring

select ConnectionsEstablished from Win32_PerfFormattedData_Tcpip_TCPv4
“ConnectionEstablished,” seen in the code above, is the current number of Established connections, inbound and outbound.

Terminal server monitoring

select ActiveSessions from Win32_PerfFormattedData_LocalSessionManager_TerminalServices
“ActiveSessions” is the number of currently active sessions connected to your Terminal Server.
Keep the link to this instructional post handy. It will help you determine what you need to query in WML, and what you find out as you’re monitoring your systems and servers may just keep you stay more competitive and efficient in your business.

Simple and advanced Shell (Terminal) tutorial.

System:
Running kernel and system information:

# uname -a                                  # Get the kernel version (and BSD version)
# lsb_release -a                         # Full release info of any LSB distribution
# cat /etc/debian_version         # Get Debian version
Use /etc/DISTR-release with DISTR= lsb (Ubuntu) /etc/issue.
# uptime                                      # Show how long the system has been running + load
# hostname                                # system's host name
# hostname -i                            # Display the IP address of the host.
# man hier                                 # Description of the file system hierarchy
# last reboot                              # Show system reboot history

Hardware Informations:
Kernel detected hardware:

# dmesg                               # Detected hardware and boot messages
# lsdev                                  # information about installed hardware
# dd if=/dev/mem bs=1k skip=768 count=256 2>/dev/null | strings -n 8 # Read BIOS

# cat /proc/cpuinfo                               # CPU model
# cat /proc/meminfo                             # Hardware memory
# grep MemTotal /proc/meminfo       # Display the physical memory
# watch -n1 'cat /proc/interrupts'        # Watch changeable interrupts continuously
# free -m                                                # Used and free memory (-m for MB)
# cat /proc/devices                              # Configured devices
# lspci -tv                       # Show PCI devices
# lsusb -tv                      # Show USB devices
# lshal                            # Show a list of all devices with their properties
# dmidecode                # Show DMI/SMBIOS: hw info from the BIOS

Load, statistics and messages:
The following commands are useful to find out what is going on on the system.

# top                                                   # display and update the top cpu processes
# mpstat 1                                         # display processors related statistics
# vmstat 2                                         # display virtual memory statistics
# iostat 2                                           # display I/O statistics (2 s intervals)
# systat -vmstat 1                            # BSD summary of system statistics (1 s intervals)
# systat -tcp 1                                  # BSD tcp connections (try also -ip)
# systat -netstat 1                           # BSD active network connections
# systat -ifstat 1                               # BSD network traffic through active interfaces
# systat -iostat 1                              # BSD CPU and and disk throughput
# tail -n 500 /var/log/messages    # Last 500 kernel/syslog messages
# tail /var/log/warn                          # System warnings messages see syslog.conf

Users:

# id                                                                     # Show the active user id with login and group
# last                                                                  # Show last logins on the system
# who                                                                 # Show who is logged on the system
# groupadd admin                                           # Add group "admin" and user colin
# useradd -c "Colin Barschel" -g admin -m colin
# usermod -a -G                                               # Add existing user to group (Debian)
# userdel colin                                                  # Delete user colin
# pw groupmod admin -m newmembe r      # Add a new member to a group
# pw useradd colin -c "Colin Barschel" -g admin -m -s /bin/tcsh
# pw userdel colin; pw groupdel admin

Kernel modules:

# lsmod                                      # List all modules loaded in the kernel
# modprobe isdn                      # To load a module (here isdn)

Compile Kernel

# cd /usr/src/linux
# make mrproper                      # Clean everything, including config files
# make oldconfig                      # Reuse the old .config if existent
# make menuconfig                 # or xconfig (Qt) or gconfig (GTK)
# make                                       # Create a compressed kernel image
# make modules                      # Compile the modules
# make modules_install         # Install the modules
# make install                           # Install the kernel
# reboot

Repair grub:
So you broke grub? Boot from a live cd, [find your linux partition under /dev and use fdisk to find the linux partion] mount the linux partition, add /proc and /dev and use grub-install /dev/xyz. Suppose linux lies on /dev/sda4:

# mount /dev/sda6 /mnt                   # mount the linux partition on /mnt
# mount --bind /proc /mnt/proc       # mount the proc subsystem into /mnt
# mount --bind /dev /mnt/dev          # mount the devices into /mnt
# chroot /mnt                                      # change root to the linux partition
# grub-install /dev/sda                     # reinstall grub with your old settings

Listing and PIDs:
Each process has a unique number, the PID. A list of all running process is retrieved with ps.
# ps -auxefw                         # Extensive list of all running process
However more typical usage is with a pipe or with pgrep:


# ps axww | grep cron
  586  ??  Is     0:01.48 /usr/sbin/cron -s
# ps axjf                                     # All processes in a tree format
# ps aux | grep 'ss[h]'               # Find all ssh pids without the grep pid
# pgrep -l sshd                         # Find the PIDs of processes by (part of) name
# echo $$                                  # The PID of your shell
# fuser -va 22/tcp                     # List processes using port 22 (Linux)
# pmap PID                               # Memory map of process (hunt memory leaks) (Linux)
# fuser -va /home                     # List processes accessing the /home partition
# strace df                                  # Trace system calls and signals
# truss df                                    # same as above

Signals/Kill:
Terminate or send a signal with kill or killall.

# kill -s TERM 4712                  # same as kill -15 4712
# killall -1 httpd                          # Kill HUP processes by exact name
# pkill -9 http                              # Kill TERM processes by (part of) name
# pkill -TERM -u www              # Kill TERM processes owned by www
# fuser -k -TERM -m /home     # Kill every process accessing /home (to umount)

Important signals are:
1       HUP (hang up)
2       INT (interrupt)
3       QUIT (quit)
9       KILL (non-catchable, non-ignorable kill)
15     TERM (software termination signal)

Permissions:
Change permission and ownership with chmod and chown. The default umask can be changed for all users in /etc/profile for Linux. The default umask is usually 022. The umask is subtracted from 777, thus umask 022 results in a permission 0f 755.

1 --x execute                        # Mode 764 = exec/read/write | read/write | read
2 -w- write                          # For:       |--  Owner  --|   |- Group-|   |Oth|
4 r-- read
  ugo=a                              u=user, g=group, o=others, a=everyone
# chmod [OPTION] MODE[,MODE] FILE    # MODE is of the form [ugoa]*([-+=]([rwxXst]))
# chmod 640 /var/log/maillog                      # Restrict the log -rw-r-----
# chmod u=rw,g=r,o= /var/log/maillog       # Same as above
# chmod -R o-r /home/*                                # Recursive remove other readable for all users
# chmod u+s /path/to/prog                           # Set SUID bit on executable (know what you do!)
# find / -perm -u+s -print                               # Find all programs with the SUID bit
# chown user:group /path/to/file                  # Change the user and group ownership of a file
# chgrp group /path/to/file                             # Change the group ownership of a file
# chmod 640 `find ./ -type f -print`                # Change permissions to 640 for all files
# chmod 751 `find ./ -type d -print`               # Change permissions to 751 for all directories

Disk information:

# hdparm -I /dev/sda                 # information about the IDE/ATA disk (Linux)
# fdisk /dev/ad2                          # Display and manipulate the partition table
# smartctl -a /dev/ad2                # Display the disk SMART info

System mount points/Disk usage

# mount | column -t                   # Show mounted file-systems on the system
# df                                              # display free disk space and mounted devices
# cat /proc/partitions                # Show all registered partitions

# du -sh *                                 # Directory sizes as listing
# du -csh                                 # Total directory size of the current directory
# du -ks * | sort -n -r              # Sort everything by size in kilobytes

Who has which files opened:
This is useful to find out which file is blocking a partition which has to be unmounted and gives a typical error of:

# umount /home/
umount: unmount of /home             # umount impossible because a file is locking home
   failed: Device busy
# ls -lSr                                               # Show files, biggest last

Find opened files on a mount point with fuser or lsof:

# fuser -m /home                     # List processes accessing /home
# lsof /home

COMMAND   PID    USER   FD   TYPE DEVICE    SIZE     NODE NAME
tcsh    29029 eedcoba  cwd    DIR   0,18   12288  1048587 /home/cipi (cipi:/home)
lsof    29140 eedcoba  cwd    DIR   0,18   12288  1048587 /home/cipi (cipi:/home)
About an application:

ps ax | grep Xorg | awk '{print $1}'
3324
# lsof -p 3324
COMMAND   PID    USER   FD   TYPE DEVICE    SIZE    NODE NAME
Xorg    3324 root    0w   REG        8,6   56296      12492 /var/log/Xorg.0.log
About a single file:
# lsof /var/log/Xorg.0.log
COMMAND  PID USER   FD   TYPE DEVICE  SIZE  NODE NAME
Xorg    3324 root    0w   REG    8,6 56296 12492 /var/log/Xorg.0.log

Mount/remount a file system
For example the cdrom. If listed in /etc/fstab:

# mount /cdrom
# mount -t auto /dev/cdrom /mnt/cdrom             # typical cdrom mount command
# mount /dev/hdc -t iso9660 -r /cdrom               # typical IDE
# mount /dev/scd0 -t iso9660 -r /cdrom             # typical SCSI cdrom
# mount /dev/sdc0 -t ntfs-3g /windows              # typical SCSI
Entry in /etc/fstab:
/dev/cdrom   /media/cdrom  subfs noauto,fs=cdfss,ro,procuid,nosuid,nodev,exec 0 0

Add swap on-the-fly
Suppose you need more swap (right now), say a 2GB file /swap2gb .

# dd if=/dev/zero of=/swap2gb bs=1024k count=2000
# mkswap /swap2gb                                            # create the swap area
# swapon /swap2gb                                             # activate the swap. It now in use
# swapoff /swap2gb                                             # when done deactivate the swap
# rm /swap2gb

Mount an SMB share
Suppose we want to access the SMB share myshare on the computer smbserver, the address as typed on a Windows PC is \\smbserver\myshare\. We mount on /mnt/smbshare. Warning> cifs wants an IP or DNS name, not a Windows name.

# smbclient -U user -I 192.168.16.229 -L //smbshare/        # List the shares
# mount -t smbfs -o username=winuser //smbserver/myshare /mnt/smbshare
# mount -t cifs -o username=winuser,password=winpwd //192.168.16.229/myshare /mnt/share
Additionally with the package mount.cifs it is possible to store the credentials in a file, for example /home/user/.smb:
username=winuser
password=winpwd
And mount as follow:
# mount -t cifs -o credentials=/home/user/.smb //192.168.16.229/myshare /mnt/smbshare

Mount an image:

# mount -t iso9660 -o loop file.iso /mnt                # Mount a CD image
# mount -t ext3 -o loop file.img /mnt                     # Mount an image with ext3 fs

Create a memory file system:
A memory based file system is very fast for heavy IO application. How to create a 64 MB partition mounted on /memdisk:

# mount -t tmpfs -osize=64m tmpfs /memdisk

Disk performance:
Read and write a 1 GB file on partition ad4s3c (/home)

# time dd if=/dev/ad4s3c of=/dev/null bs=1024k count=1000
# time dd if=/dev/zero bs=1024k count=1000 of=/home/1Gb.file
# hdparm -tT /dev/hda      # Linux only

Networking:

# ethtool eth0                                           # Show the ethernet status (replaces mii-diag)
# ethtool -s eth0 speed 100 duplex full # Force 100Mbit Full duplex
# ethtool -s eth0 autoneg off # Disable auto negotiation
# ethtool -p eth1                                      # Blink the ethernet led - very useful when supported
# ip link show                                           # Display all interfaces on Linux (similar to ifconfig)
# ip link set eth0 up                                # Bring device up (or down). Same as "ifconfig eth0 up"
# ip addr show                                        # Display all IP addresses on Linux (similar to ifconfig)
# ip neigh show                                      # Similar to arp -a

Ports in use:
Listening open ports:

# netstat -an | grep LISTEN
# lsof -i                                         # List all Internet connections
# socklist                                     # Display list of open sockets
# netstat -anp --udp --tcp | grep LISTEN    
# netstat -tup                              # List active connections to/from system
# netstat -tupl                             # List listening ports from system

Firewall
Check if a firewall is running (typical configuration only):

# iptables -L -n -v                                 # For status Open the iptables firewall
# iptables -P INPUT       ACCEPT     # Open everything
# iptables -P FORWARD     ACCEPT
# iptables -P OUTPUT      ACCEPT
# iptables -Z                                         # Zero the packet and byte counters in all chains
# iptables -F                                         # Flush all chains
# iptables -X                                         # Delete all chains

IP Forward for routing
Check and then enable IP forward with :
# cat /proc/sys/net/ipv4/ip_forward  # Check IP forward 0=off, 1=on
# echo 1 > /proc/sys/net/ipv4/ip_forward
or edit /etc/sysctl.conf with:
net.ipv4.ip_forward = 1

Network Address Translation

# iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE    # to activate NAT
# iptables -t nat -A PREROUTING -p tcp -d 78.31.70.238 --dport 20022 -j DNAT \
--to 192.168.16.44:22           # Port forward 20022 to internal IP port ssh
# iptables -t nat -A PREROUTING -p tcp -d 78.31.70.238 --dport 993:995 -j DNAT \
--to 192.168.16.254:993-995     # Port forward of range 993-995
# ip route flush cache
# iptables -L -t nat            # Check NAT status

DNS
The DNS entries are valid for all interfaces and are stored in /etc/resolv.conf. The domain to which the host belongs is also stored in this file. A minimal configuration is:

nameserver 66.63.128.84
search cipi.net intern.lab
domain cipi.org
Check the system domain name with:
# hostname -d                # Same as dnsdomainname

DHCP

# dhcpcd -n eth0           # Trigger a renew (does not always work)
# dhcpcd -k eth0           # release and shutdown
The lease with the full information is stored in:
/var/lib/dhcpcd/dhcpcd-eth0.info

tar
The command tar (tape archive) creates and extracts archives of file and directories. The archive .tar is uncompressed, a compressed archive has the extension .tgz or .tar.gz (zip) or .tbz (bzip2). Do not use absolute path when creating an archive, you probably want to unpack it somewhere else. Some typical commands are:
Create

# cd /
# tar -cf home.tar home/         # archive the whole /home directory (c for create)
# tar -czf home.tgz home/      # same with zip compression
# tar -cjf home.tbz home/       # same with bzip2 compression
Only include one (or two) directories from a tree, but keep the relative structure. For example archive /usr/local/etc and /usr/local/www and the first directory in the archive should be local/.
# tar -C /usr -czf local.tgz local/etc local/www
# tar -C /usr -xzf local.tgz      # To untar the local dir into /usr
# cd /usr; tar -xzf local.tgz     # Is the same as above

Extract

# tar -tzf home.tgz               # look inside the archive without extracting (list)
# tar -xf home.tar                # extract the archive here (x for extract)
# tar -xzf home.tgz             # same with zip compression (-xjf for bzip2 compression)
                                # remove leading path gallery2 and extract into gallery
# tar --strip-components 1 -zxvf gallery2.tgz -C gallery/
# tar -xjf home.tbz home/colin/file.txt    # Restore a single file

More advanced

# tar c dir/ | gzip | ssh user@remote 'dd of=dir.tgz' # arch dir/ and store remotely.
# tar cvf - `find . -print` > backup.tar                 # arch the current directory.
# tar -cf - -C /etc . | tar xpf - -C /backup/etc      # Copy directories
# tar -cf - -C /etc . | ssh user@remote tar xpf - -C /backup/etc      # Remote copy.
# tar -czf home.tgz --exclude '*.o' --exclude 'tmp/' home/

Find

Some important options:
-x (on BSD) -xdev (on Linux)       Stay on the same file system (dev in fstab).
-exec cmd {} \;       Execute the command and replace {} with the full path
-iname       Like -name but is case insensitive
-ls       Display information about the file (like ls -la)
-size n       n is +-n (k M G T P)
-cmin n       File's status was last changed n minutes ago.
# find . -type f ! -perm -444        # Find files not readable by all
# find . -type d ! -perm -111        # Find dirs not accessible by all
# find /home/user/ -cmin 10 -print   # Files created or modified in the last 10 min.
# find . -name '*.[ch]' | xargs grep -E 'expr' # Search 'expr' in this dir and below.
# find / -name "*.core" | xargs rm   # Find core dumps and delete them (also try core.*)
# find / -name "*.core" -print -exec rm {} \;  # Other syntax
# Find images and create an archive, iname is not case sensitive. -r for append
# find . \( -iname "*.png" -o -iname "*.jpg" \) -print -exec tar -rf images.tar {} \;
# find . -type f -name "*.txt" ! -name README.txt -print  # Exclude README.txt files
# find /var/ -size +10M -exec ls -lh {} \;     # Find large files > 10 MB
# find /var/ -size +10M -ls           # This is simpler
# find . -size +10M -size -50M -print
# find /usr/ports/ -name work -type d -print -exec rm -rf {} \;  # Clean the ports
# Find files with SUID; those file are vulnerable and must be kept secure
# find / -type f -user root -perm -4000 -exec ls -l {} \;

Miscellaneous

# which command                      # Show full path name of command
# time command                         # See how long a command takes to execute
# time cat                                     # Use time as stopwatch. Ctrl-c to stop
# set | grep $USER                    # List the current environment
# cal -3                                         # Display a three month calendar
# date [-u|--utc|--universal] [MMDDhhmm[[CC]YY][.ss]]
# date 10022155                       # Set date and time
# whatis grep                              # Display a short info on the command or word
# whereis java                            # Search path and standard directories for word
# setenv varname value           # Set env. variable varname to value (csh/tcsh)
# export varname="value"        # set env. variable varname to value (sh/ksh/bash)
# pwd                                # Print working directory
# mkdir -p /path/to/dir                 # no error if existing, make parent dirs as needed
# mkdir -p project/{bin,src,obj,doc/{html,man,pdf},debug/some/more/dirs}
# rmdir /path/to/dir                     # Remove directory
# rm -rf /path/t


Original Source: http://community.linuxmint.com/tutorial/view/454

Sunday, November 9, 2014

How to Migrate Users and Computers to an entirely new 2008R2 AD Domain/Forest from an old 2003/2008/2008R2 Domain/Forest

Since completing a number of successful AD Creation/Migration projects, I am frequently asked for particulars of successful User and Computer migration. Embarking on this project for the first time can be daunting in the amount of research and preparation required to be successful. Here I've simplified and laid out the basic steps required without overly exhausting the details, which can be found by googling or technetting the appropriate keywords pertaining to each step. (I have included some of the relevant links inside the steps when I had them readily available)
These notes are the end result of much research and trial and error, and are designed to ensure that those who follow them will avoid the many pitfalls waiting for the unfamiliar or unprepared.
Please note that on the instructions, where I refer to “NEW Domain”, I am talking about your new target domain that you are migrating “to”, which, for example, might be NEW.local, ABCDEF.com, COMPANY01.com, etc. Where I refer to “OLD Domain”, I am talking about the domain you are migrating “from”. Where I enter a value holder in “carrots”, such as , you should really enter your new domain, ie: NEW.local or whatever, without the carrots. Same goes for password holders in carrots.
This how-to could also be helpful with migrations between 2003 or 2008 forests, with minor changes.
Good Luck!
1.

Set up NEW Domain and Create Network between OLD and NEW Domains

1. Load a server with 2008 R2 and install the Domain Controller role, choosing domain and server naming as appropriate.
2. Create Admin and other needed Service Accounts in the NEW domain according to company policies
3. Communication between domains Varies by site (typically VPN for remote, Local Layer-3 routing for local)
4. Create appropriate DNS Stub Zones in each Domain for the other
5. Verify proper DNS resolution across Domains
2.

Create Trusts between Domains

1. Two-way Transitive Trust between OLD domain and NEW domain.
2. Forest-Wide Authentication
3. Verify Trusts
3.

Prepare NEW (Target) Domain (Note: do not include the “<” or “>” symbols around the examples)

1. Create DNS Suffix Settings / DNS Suffix Search List GPO at Domain Root
2. Create an ADMT Server on NEW domain/forest.
a. Can be physical or virtual – 64 bit recommended (Temporary Server – can be decommissioned at completion of Migration project)
b. Must load the appropriate version of Microsoft ADMT software depending upon Forest levels of OLD and NEW domains (compare needs to checklist on Microsoft download site)
c. Should create an AD user in NEW domain for ADMTadmin that is also a member of the Domain Admins Group and the Administrators group. Should also be a member of the local Administrators group on this ADMT server machine.
d. Allow program to install its own SQL Express database, which will be referred to as \SQLExpress in SQL Management Studio 2008
e. Install SQL Management Studio (Management Tools only, NO Database)
3. Disable SID Filtering:
a. From “elevated” Command Prompt on PDC Emulator in NEW Target Domain, enter the following command:
i. Netdom trust /domain: /quarantine:No /userD: /passwordD:
ii. Change source domain, user, and password above as required for each domain
4. Enable SID History:
a. From “elevated” Command Prompt on PDC Emulator in NEW Target Domain, enter the following command:
i. Netdom trust /domain: /enableSIDhistory:yes /userD: /passwordD:
ii. Change source domain, user, and password above as required for each domain
4.

Prepare OLD (Source) Domain (Note: do not include the “<” or “>” symbols around the examples)

1. Disable SID Filtering:
a. From “elevated” Command Prompt on PDC Emulator in OLD Source Domain, enter the following command:
Netdom trust /domain: /quarantine:no /userD:admtadmin /passwordD:
b. Change source domain above as required for each domain
2. Enable SID History: (Note: do not include the “<” or “>” symbols around the examples)
a. From “elevated” Command Prompt on PDC Emulator in OLD Source Domain, enter the following command:
Netdom trust /domain: /enableSIDhistory:yes /userD:admtadmin /passwordD:
b. Change source domain above as required for each domain
3. Load Windows 2008 R2 Schema Extensions (2008 R2 Install DVD\Support\Adprep) (Unless already 2008 R2)
a. Perform one of the following:
i. On a 32-bit PDC Emulator machine in Source Domain
1. Insert Server 2008 R2 DVD into drive
2. From elevated Command Prompt, navigate to X:\support\adprep (Where X is drive letter)
3. Enter, in this order, the following commands
a. Run adprep32 /forestprep
b. Run adprep32 /domainprep /gpprep
c. Run adprep32 /rodcprep
ii. On a 64-bit PDC Emulator machine in Source Domain
1. Insert Server 2008 R2 DVD into drive
2. From elevated Command Prompt, navigate to X:\support\adprep (Where X is drive letter)
3. Enter, in this order, the following commands
a. Run adprep /forestprep
b. Run adprep /domainprep /gpprep
c. Run adprep /rodcprep
4. If using local Exchange Servers, Load Exchange Schema Extensions on Target Domain (unless already using the appropriate version of Exchange on the Target Domain)
a. On 2008 R2 Target DC
i. Insert Exchange DVD
ii. From elevated Command Prompt enter, in this order, the following commands
1. Run setup /PS
2. Run setup /preparead /”organizational name” (ie – “NEW.local”)
3. Run setup /pad
5. Set GPO Audit Policy, for Domain Root and for Domain Controllers OU. (Computer, Windows, Security, Local, Audit Policy) to audit Success/Failure of:
a. Account Logon Events
b. Account Management
c. Logon Events
d. Policy Change
5.

Prepare Cross-Domain permissions

1. In Source Domain:
a. Add user NEW\admtadmin to the “Administrators” group in the Builtin OU
b. Add group NEW\Domain Admins to the “Administrators” group in the Builtin OU
2. In Target Domain:
a. Add User OLD\xxxxxx to “Administrators” group in the Builtin OU (where xxxxxx is a user that is BOTH a Domain Admin in the Source domain AND a Local Admin on all Workstations in the domain
b. Add group OLD\Domain Admins to the “Administrators” group in the Builtin OU
6.

Prepare for Password Export (Note: do not include the “<” or “>” symbols around the examples)

1. Create PES Key on Target ADMT Server for Source Domain.
a. On ADMT Server in NEW domain, enter the following command at the Command Prompt:
i. C:> admt key /option:create /sourcedomain: /keyfile:”c:\ PES Key\PES” /keypassword:
ii. Change the sourcedomain and folder information as appropriate
2. Copy the PES.pes Key file to PDC Emulator DC on Source Domain and put it in C:\PES Key\PES.pes
3. Download and Install PES Service from Microsoft on Source Domain PDC Emulator using PES.pes file you just copied over
a. Browse to c:\PES Key\PES.pes
b. Enter the SAME password you used when creating the Key file on the ADMT Server
c. Set “Log on as” to: NEW\admtadmin with correct user password
d. Reboot Source PDC Emulator Domain Controller you just installed PES on
4. ALWAYS Start and/or Restart the PES Service on Source PDC Emulator prior to actual USER migrations
7.

Pre-Migrate Groups with SID History (will automatically avoid transfer of Built-in goups)

1. On ADMT server, start Active Directory Migration Tool
2. Right-click on Group Migration Wizard
3. Type in appropriate Source domain and DC info (should be the PDC Emulator)
4. Type in appropriate Target domain info
5. Destination should be an OU you’ve created for “Groups from Old Domain”
6. Follow prompts to enter information as appropriate
8.

Pre-Migrate Users with SID History (disabled in Target Domain, enabled in Source Domain)

1. Ensure that you have followed STEP 6, Number 4
2. On ADMT server, start Active Directory Migration Tool
3. Right-click on User Migration Wizard
4. Type in appropriate Source domain and DC info (should be the PDC Emulator)
5. Type in appropriate Target domain info
6. Select users to pre-migrate
7. Select “Disable Target Accounts” and “Migrate SID Histories”
8. If migrating FROM more than one domain to the NEW domain, Select “On Conflicts, do not migrate” so that you can determine if there are any username conflicts between the current migrating domain users and users that have previously been migrated from other domains
9. Follow prompts to enter information as appropriate
10. If there were any username conflicts, change the username of the “to-be-merged” user and re-run this process for that new username until all conflicts are resolved.
9.

Prepare Computers for Migration

1. In source domain Active Directory, move computer to “no policy” computers group, if needed
2. Log into the workstation as the regular user of the machine to be migrated
3. Apply all patches and updates from Microsoft, Adobe, Java, etc., even if they happen later in the process
4. Run the latest version of CCleaner against the registry with “backup” and fix, re-run until clean
5. Clean up all temp files and empty recycle bin, etc., using either CCleaner or Disk Cleanup tool. (For IE, do not delete Cookies, History, Recent Files, or Recent Documents, except for Temporary Internet Files)
6. Ensure that the NIC properties show, at a minimum:
a. Client for Microsoft Networks
b. QoS Packet Scheduler
c. File and Printer Sharing for Microsoft Networks
d. Internet Protocol Version 4
e. Link-Layer Topology Discovery Mapper (Windows Vista / Windows 7 only)
f. Link-Layer Topology Discovery Responder (Windows Vista / Windows 7 only)
7. Verify that the following services are set to “automatic” and are “started”:
a. Windows XP:
i. DCOM Server Process Launcher
ii. Remote Procedure Call (RPC)
b. Windows Vista:
i. DCOM Server Process Launcher
ii. Remote Procedure Call (RPC)
c. Windows 7:
i. DCOM Server Process Launcher
ii. Remote Procedure Call (RPC)
iii. RPC Endpoint Mapper
d. Ensure that the service “Remote Procedure Call (RPC) Locator” is stopped and/or does not exist
8. Verify that the following Registry Keys exist:
a. Windows XP:
i. HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\RpcSs
b. Windows Vista:
i. HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\RpcSs
ii. HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\DcomLaunch
c. Windows 7:
i. HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\RpcSs
ii. HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\DcomLaunch
iii. HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\RpcEptMapper
d. If one or more of these keys do not exist, export them from another machine with the same OS, and import them to the registry on this machine
9. Disable Windows Firewall (either directly or via GPO)
10. Disable any other Firewalls if present
11. If the machine has any “Manufacturer-provided” security software (HP Security, etc., completely disable it)
12. From a command prompt, run the command “verifier” with default values, click next to fix drivers
13. Reboot and Login as an admin user (not the regular user of the machine) and Backup the main User Profile just in case (use Transwiz, USMT, etc.)
14. Reboot computer and Log in as the regular user of the computer
15. On Windows 7 machines, it has been shown to migrate better if, using MSCONFIG, all non-Microsoft services are temporarily disabled, to be re-enabled after migration. You will have to reboot again at this point
16. Allow startup to completely finish to ensure that no automatic updates are trying to run
17. Close any open programs
18. Log out and leave computer in this state
19. Ensure that the user is not logged directly into any other domain machines during the migration process
10.

Migrate Computers

1. Log into Target Domain ADMT server with Source Domain Admin credentials that are also a local Computer Admin on the workstations for that site
2. Ensure that the regular workstation user has already been pre-migrated (IMPORTANT)
3. Run Computer Migration Wizard with full agent operation, and disabling post-check retries
4. Computer should automatically reboot if migration is successful (or if the agent “Completes with Errors”)
5. May require a couple more manual reboots during the next couple of steps before it is complete
6. New NEW Domain user will not be able to log into their machine until user account is re-migrated and enabled in Target Domain
11.

Re-Migrate Groups (only if groups have changed since last user migration)

1. Follow same rules as in STEP 7
12.

Re-Migrate Users of recently migrated Computers (enabled in Target Domain, disabled in Source Domain)

1. Ensure that you have followed STEP 6, Number 4
2. Select “Enable in Target Domain” and “Disable in Source Domain”
3. Select “Migrate SID Histories”
4. Select “Migrate and Merge conflicting users”
13.

After Migration is successfully completed (before allowing User to log in)

1. On Target Domain DC, uncheck the “required to change password” box on user account.
2. On Target DC, add user to appropriate Target Domain groups if not already there.
IF CONTINUING TO USE OLD EXCHANGE SERVER IN SOURCE DOMAIN:
3. Repair email mailbox:
a. In Exchange Management Console of Source Domain, “Disable” User’s mailbox (Do NOT “Remove”, NOTE: This step will remove any non-default additional email addresses for the user. If this is a problem, you will need to notate what they are so that you can put them back in, OR, you will need to ignore this Step a, and Step b below, and instead use the Exchange Management Shell to link the mailboxes.
b. In Disconnected Mailbox window, Re-connect mailbox:
i. Select “Linked” mailbox, click next
ii. Select “Existing User”, and find disabled user in Source Domain, click next
iii. Select “Trusted Forest or Domain” and select NEW Domain
iv. Select the NEW Domain DC as the Linked Domain DC (using the local will eliminate the need to wait for AD Replication to occur)
v. Select the same username in the Target Domain as the Linked Master User, click next, then connect, then finish
IF MOVING TO AN EXCHANGE SERVER IN THE TARGET DOMAIN:
4. Migrate mailboxes from old exchange to new exchange using normal procedures for migrating mailboxes, which is a separate process from this document
5. Login to migrated system as local admin and check computer properties to ensure that Domain is the NEW domain
6. Run GPUPDATE /FORCE from Command Prompt
7. Reboot Computer
8. Login to local system as local admin and check computer name to ensure that DNS Suffix is set to NEW Domain
9. If not, Run GPUPDATE /FORCE from Command Prompt again, reboot, and continue this process until DNS Suffix IS set to the NEW Domain.
10. Reboot Computer
11. Allow User to log in to computer on the NEW domain. (If there are errors, see Troubleshooting below)
12. Ensure proper operation of all of User’s legacy applications
13. Ensure proper access to all of User’s legacy resources
a. Folders / Shares –
i. Add NEW domain users /groups to appropriate NEW domain access groups appropriately
ii. Add NEW domain users /groups to appropriate “share” permissions appropriately
14.

Troubleshooting

1. If, at first user logon, you receive the following message:
a. “The Trust between this workstation and the primary domain failed”, then:
i. “reset” the new computer account in active directory on the new domain, and try logging in again. If this does not work, then,
ii. Log in on the local machine as a local machine admin and remove the computer from the domain, then reboot and rejoin the computer to the domain and try logging in again
2. If, while trying to log on, you receive a message that states something along the lines of:
a. “The NEW Domain is not available”, or
b. “Unable to log you in because the Domain Controller or Domain is unavailable”, etc.
i. Log in locally to the machine and verify that the Domain is set to the NEW Domain
ii. Verify that the Full Computer Name is XXX-XXXXX.NEW Domain, if it still has the old DNS suffix,
1. Open a Command Prompt and enter “gpupdate /force” to force a Policy update from the new domain
2. Reboot Computer and try again to log into the new domain
3. Try logging into the old domain, then retry logging into the new domain
4. You may need to let it sit for 10 to 15 minutes before this will work
3. If machine Passed Pre-Test, and Completed with Errors on Agent-Operation, you can log in and check to see if the new “NEW Domain” profile was created on the machine. If it was created, check to see if it is exactly the same size and the “OLD Domain” profile. If it is, then check to see if both profiles have equal access to the “User” folder for the user on the local machine. If so, then check to see if the domain is still set to the OLD Domain. If all of this exists, then you can manually join the computer to the NEW Domain with no problems.
4. NOTE: If continuing to utilize shares on the OLD Domain, remember that on File Shares, users now located on the NEW Domain will need to be added to the “Sharing” permissions on the Share itself, before their SID History will allow them to utilize their File and Folder permissions.

by Mini_magick20120309-28733-1i7m8l8_small ByronHay