Bootcamp adventures

I needed to replace a drive in a Mac mini with a bigger one. The drive had Mac OS X 10.9 (Mavericks) and Bootcamp with Windows 7. After using Clonezilla to backup the drive and restore it to the bigger one, the partitions were obviously still the same size. There was just a lot of free unpartitioned space at the end of the new drive.

How to resize and move all the partitions (including the hidden EFI and Recovery partitions), to fill the free space?

Disk Utility will not let you touch the Bootcamp partition. Windows 7 looked like it could resize it, but not move it. Resizing it with Win7 created a mess: the Mac would still see the original size.

The heart of the problem seems to be that the Mac wants a GPT partition table, but for Bootcamp, it creates a hybrid MBR partition which is what Win7 sees. Win7 would have no problem with a GPT-only partition, but Bootcamp makes a hybrid MBR anyway. Win7 then resizes that MBR partition, but doesn't update the GPT partition table, which is what the Mac sees. And the Mac doesn't let you fix it either.

At this point, I tried Gparted, but it wouldn't touch this mess (giving some error which I forgot).

Paragon's Camptune X looked like the best solution. However, after paying $20 for it, it turned out it couldn't do anything either. All it does is to let you move a cursor for the relative sizes of the Mac and Windows partitions. But you cannot increase the size to use the free space.

Finally, Rod Smith's Gdisk saved the day again.

What I ended up doing worked in the end:

• Booted a Gparted USB key, and resized the Windows partition to fill the entire disk.
• Booted to Mac, and used Camptune X to enlarge the Mac partition while reducing the Windows one.
• Now, Windows would not boot.
• Used gdisk to re-create the hybrid MBR, and mark the Windows partition as bootable, as explained in detail in this post.

rsync server daemon on Mac OS X with launchctl

(Update: Added the --no-detach option to the rsync command. Newer MacOS versions wouldn't start the daemon without it. With the added argument, it now works again in Sierra.)

There are many web pages describing how to enable the rsync daemon on Mac OS X using launchd/launchctl mechanism. But I had to use a different (and simpler) plist file in LaunchDaemons to make it work across reboots on Lion (10.7.4).

(I started by following this guide , and this very similar one. And I also read this and this. In the end, what helped me getting the plist file right was this thread. Particularly this post: "For one you have both a Program and a ProgramArguments key, when you should have only one or the other (you use Program if there is just one element to the command, or ProgramArguments if there are multiple." And this one.)

This is the .plist file I used in /Library/LaunchDaemons/org.samba.rsync.plist : 

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>Disabled</key>
    <false/>
    <key>Label</key>
    <string>org.samba.rsync</string>
    <key>ProgramArguments</key>
    <array>
        <string>/usr/bin/rsync</string>
        <string>--daemon</string>
        <string>--no-detach</string>
    </array>
    <key>RunAtLoad</key>
    <true/>
    <key>KeepAlive</key>
    <dict>
        <key>SuccessfulExit</key>
        <false/>
    </dict>
</dict>
</plist>

This is an example /etc/rsyncd.conf file:

secrets file = /etc/rsyncd.secrets
hosts allow = 192.168.1.0/24 10.0.0.1 *.cust.isp.tld

uid = nobody
gid = nobody
list = yes
read only = yes

[shared]
path = /Users/Shared
comment = Users-Shared
uid = someuser
gid = admin
auth users = user_in_secrets

The file /etc/rsyncd.secrets looks like:

some_rsync_user:password
other_user:other_password

To install it:

sudo -s
chown root:wheel /etc/rsyncd.*
chmod 644 /etc/rsyncd.conf
chmod 600 /etc/rsyncd.secrets
launchctl load /Library/LaunchDaemons/org.samba.rsync.plist
launchctl start org.samba.rsync ## (this last command is probably unneeded)

To check if it is installed and running:

launchctl list | grep rsync
808  -    0x7fddb4806c10.anonymous.rsync
-    0    org.samba.rsync

ps ax | grep [r]sync
  808   ??  Ss     0:00.00 /usr/bin/rsync --daemon

rsync --stats someuser@localhost::

To remove it:

sudo launchctl unload /Library/LaunchDaemons/org.samba.rsync.plist
sudo killall rsync

For logging transfers, add

log file = /var/log/rsyncd.log
transfer logging = yes

to /etc/rsyncd.conf. And to have the log rotated, create a file like /etc/newsyslog.d/rsyncd.conf and add

# logfilename          [owner:group]    mode count size when  flags [/pid_file] [sig_num]
/var/log/rsyncd.log   644  5    5000 *     J

 

USB OSX installer for the impatient

To make a bootable USB disk with the Mac OS X installer, the guides I found are much too verbose for my taste, and have too many cute screenshots and ads. Here is a summary for the impatient.

For Mavericks, Yosemite, El Capitan, Sierra

There is a handy "createinstallmedia" command.

The only difficulty is getting the installer, which must be downloaded from the App store. If you need an older installer than the current version, the only way seems to be to find it in the "purchased" page.

For the Sierra (10.12) installer, try this App Store link: https://itunes.apple.com/us/app/macos-sierra/id1127487414?mt=12

The downloaded installer image is automatically started. If you proceed with the install, it will be deleted afterwards. So copy it before installing or just close the installer.

• Get a USB disk of 8GB or more.
• Create a single GPT (GUID) partition on the USB key, and format it. This can be done in Disk Utility, but command line junkies can also do it this way:

  diskutil list ## check which is the device name to format
  disk=/dev/diskX ## USE CORRECT DISK found with previous command
  echo "This will completely destroy '$disk'"
  # diskutil partitionDisk $disk GPT hfs+ Untitled 100% ## Remove leading dash once you are sure
  

• Define variables for the installer location and your USB disk:
  ("/Volumes/Untitled" is the mount point of your USB key, which will be erased.)
  

  installer="/Applications/Install OS X Yosemite.app"
  USBdisk="/Volumes/Untitled"

• Then run:

sudo "$installer/Contents/Resources/createinstallmedia" --volume "$USBdisk" --applicationpath "$installer" --nointeraction

That's it.

For older versions like (Mountain) Lion

The installer disk image can be found in Applications / Install Mac OS X Lion.app (right-click -> Show Package Contents) / Contents / SharedSupport / InstallESD.dmg

• Open InstallESD.dmg. You get a "Mac OSX Install ESD" disk on the desktop
• Partition and format the (8 GB.) USB key as standard Mac OSX Extended with journal. (The partition table defaults to MBR for USB drives; that's OK)
• In the "Restore" tab of Disk Utility:
• the source is the mounted image on your desktop: "Mac OS X Install ESD" (NOT the .dmg file)
• the destination is your new USB Mac partition (not the drive itself)

Other instructions suggest using the InstallESD.dmg file as the source, and the USB key itself (not the partition it contains) as the destination. That may work too. Just don't mix both methods. I had tried that and failed, but maybe it was because I had first made a GPT partition table instead of MBR?

If you only have a 4GB key, it seems to work using Carbon Copy Cloner and de-selecting all unneeded language packs. But I haven't tried an install from such a key.

Mac and OpenLDAP: Local homes for network users

I wanted a Mac to authenticate users against our Debian OpenLDAP server, but to create a local home directory on the Mac (see here for more details). The usual configuration for network users on the Mac is to mount their homes from the server over NFS. There are many excellent instructions on the net on how to do that. But finding help on how to have them use a local home instead was much more difficult.

It turns out it can be done very simply, by disabling one line in /etc/auto_master on the Mac. By default, it contains +auto_master, which tells the Mac's automounter to look for an automount map in LDAP. If this line is disabled, the Mac will create a local home for network users the first time they log in. Since our userHomes in LDAP are defined as /home/username, the Mac home is created under /home instead of /Users, which is fine.

So for such a setup, you do NOT need to import an Apple schema into your LDAP directory. (That was quite a hassle because you need to tweak the original schema which is not quite kosher; but it was unnecessary).

All you need to do is comment out this single line in /etc/auto_master to make it

#+auto_master  # Use directory service

Or copy/paste this:

sudo perl -i.orig -pe 's/^(\+auto_master.*)/## $1/' /etc/auto_master

Kill the Final Cut registration screen

I came across this nicely detailed post explaining how to get rid of the forced registration screen of Final Cut Pro/Studio, which always pops up when you really don't want to be bothered with this idiocy.

But I felt the solution was worse than the problem. It involved far too much clicking around for my taste. And you need the Property List Editor. You only have that once you have installed over 1 GB (!!) of developer tools. If you can remember where you put your OS X disk, that is.

Surely, there must be a better way to do it, by just copying a command from some web page and pasting it into Terminal?

It turned out to be 3 commands. And getting them right was much worse than the solution I din't like. You need your machine ID, which is in an XML file that defaults read doesn't want to read. And in that file it is encoded in Base64. You need to put this ID into a property list file as data. That can be done with defaults write, but the data needs to be in hex. I should just have registered, I guess...

Anyway, the detailed explanations are in the link of the first sentence, and the 3 ridiculous commands to paste into Terminal are here:

id=$(perl -MMIME::Base64 -ne '/^\s+(\S{64})\s*$/ && print unpack("H*",decode_base64($1));' "/Library/Application Support/ProApps/Final Cut Studio System ID"|tail -1)
sudo defaults write /Library/Preferences/com.apple.RegFinalCutStudio "{ AECoreTechRegister=1; AECoreTechRegSent=1; }"
sudo defaults write /Library/Preferences/com.apple.RegFinalCutStudio AECoreTechRegInfo -data "$id"

Hard drive partitions and file system essentials

What most normal users need to know about hard disk partitions and filesystems to be able to move hard disks between various operating systems like Mac or Windows.

Hard disks contain 1 or more partitions. To the user, each partition appears as if it were a separate hard disk.

(In Windows, each partition receives a separate drive letter like C:, D:, etc.; on a Mac, you see a separate icon on the desktop for each partition; in Linux, each is a device like /dev/sdb1, /dev/sdb2, etc.)

Every partition needs to be formatted with a file system to let the operating system store and retrieve files. (On Mac, this formatting process is called "erasing")

There are many different types of file systems. Your system needs to understand these file systems to be able to use them. Unfortunately, various operating systems use different file systems. The problem is to find which one will be understood by all the systems you intend to connect your drive to. Also, some systems only support reading some file systems, not writing to them.

Summary

Below is a table trying to summarize the compatibility between the 3 main operating systems and the 4 main file system types. There are many others, but if you know about them, you probably don't need this page.

	Windows	Mac OS X	Linux
FAT32 or DOS	Native support Max. 4GB. file size	Read/Write Max. 4GB. file size	Read/Write Max. 4GB. file size
NTFS	Native support	Read only. Write support through external drivers. 1	Read/Write on recent distributions. 2
HFS+ or "Mac OS extended"	Requires third party programs for reading and writing. 3	Native support	Read only. Write if the journaling feature has been turned off in Mac OS X. 4
Ext2 or Ext3	Free drivers allow Read/Write access.5	Requires commercial driver. 6	Native support

FAT or FAT32 (named "MS-DOS" in Macs)

This the oldest of the file systems commonly used today. As such, it has the greatest compatibility and the least functionality. It is a sort of lowest common denominator.

All operating systems can read and write to it. It is the file system generally used on USB flash drives, memory cards for photo cameras, etc.

It cannot store files greater than 4 Gigabytes. It is also the least reliable of the current file systems, and has many other drawbacks (fragmentation, no support for permission, time stamps in local time and with only 2 seconds resolution, etc.)

The Windows disk manager refuses to format a FAT32 partition greater than 32 GB. But it can be formatted in Windows with the free fat32format.exe utility, or can be formatted to the wanted size on Mac or Linux.

NTFS

Is the native file system of Windows.

Macs can read it, but cannot write to it.

However, there is a Mac version of the open source NTFS-3G driver which can write to NTFS. ¹

Recent Linux versions can both read it and write to it (thes usually have this NTFS-3G driver installed by default). ²

HFS aka. "Mac OS X" HFS+ aka. "Mac OS X Extended (journaled)"

Is the native file system on Macs.The Mac default is the HFS+ journaled variant.

Windows needs special programs installed to be able to read or write it. ³

Linux can read it when it has the hfsutils package installed. It can also write to it if journaling has been disabled. ⁴

Ext2 or Ext3

is the most common file system on Linux.

(If you wonder why you would need to know anything about Linux: while it is not very common as a desktop operating system, it is the system used in almost all your non-computer devices which contain a hard disk, like your NAS backup disk, your media player, etc. If that device breaks, you may be able to recover the files from it's hard disk by connecting it to your main computer and installing the driver for the ext2 file system)

Windows can read and write to it using free drivers. ⁵

There is a Mac driver, but it may be problematic. ⁶

Footnotes:

1. Mac -> NTFS : The free and open source ntfs-3g driver for Mac is available on http://sourceforge.net/projects/catacombae/files/. The commercial version is based on the same code, but improves speed. You may also want to have a look at the user guide and the macntfs-3g blog.

2. Linux -> NTFS : In case you have an older distribution which doesn't have it pre-installed, you can normally install "ntfs-3g" using your distribution's package manager. Or have a look at their availability page.

3. Windows -> HFS : If you only need to copy files from a Mac disk to your Windows machine, you can use the free HFSExplorer, which will open your drive in a Windows Explorer-like window and let you copy files from there.

For full support, you may need commercial software like MacDrive or similar.

4. Linux -> HFS : If it isn't already on your system, you will need to install the "hfsutils" package.

If you need to write to the HFS disk, journaling must be disabled. You need to do this on a Mac. Afterwards, you can re-enable journaling (again on a Mac). To disable journaling on a Mac, open Disk Utility, select the volume, hold the Option (or Alt) key while opening the File menu, which will make the "Disable Journal" menu entry appear in the menu. Alternatively, you can enter diskutil disableJournal "/Volumes/YOUR_VOLUME_NAME" in Terminal

5. Windows -> ext2/3 : There are 2 free drivers. The open source one is at http://www.ext2fsd.com/ and the closed source one is at http://www.fs-driver.org/.

6. Mac -> Ext2/3 : You can try the commercial ExtFS for Mac OS X. Or the open source fuse-ext2 which I have never tried. (There is also another free open source driver (http://sourceforge.net/projects/ext2fsx), but that project doesn't seem to be actively maintained. It may have worked well on older Mac OS versions, but when I tried a simple folder move with the current version 1.4d4 on a Mac OS X 10.5 system, it made the system crash hard, and left a badly corrupted drive, which I had to repair using e2fsck on Linux.)

Mac install

Update (May 2012): this entry is not quite up-to-date. The "Mac Install notes" on Google docs may be much better.

Software:

A few links to (free) software I want on the Mac when doing a fresh install:

• Firefox (the standard web browser)
• Smultron (text editor. After install, in Smultron's help menu, also do: "install command-line utility")
• Fink (essential open source shell tools like wget, etc.)
• NeoOffice or OpenOffice.org (Office documents like Word, Excel, etc.)
• Videolan VLC player (plays most audio/video formats which QT or iTunes don't)
• Lingon (launchd configuration editor)
• muCommander (File manager (and FTP client). Not as great as Total Commander, but better than nothing) (or the shareware Disk Order?)
• MediInfo (displays all the gory details about audio/video files)

A few other things that may be needed or wanted:

• ExpressCard SATA driver (OS X 10.5 driver for Silicon Image SiI3132 chipset, Vendor ID 0x1095, Device ID 0x3132. For 10.6 or other systems look here)
• Carbon Copy Cloner (disk cloning)
• Cyberduck (another FTP client for people who don't like muCommander or need more FTP features)
• DiffMerge (visually compare and merge files)
• Timecode Calculator Widget (a dashboard widget)

And some configuration notes:

File locations:

On the Mac, many standard Unix files are in different locations than expected:

• /var/db/smb.conf for the Samba configuration
• /private/etc/sshd_config for the SSH server config.
• /private/var/at/tabs/$USER for the user's crontab
• /private/var/mail/$USER for system mail (like cron output)

Configuration:

To change the silly default computer name: System Preferences -> Sharing. If it is not enough, try

$ sudo hostname my-permanent-name
$ sudo scutil -set LocalHostName $(hostname)
$ sudo scutil -set HostName $(hostname)

Configure Terminal

$ mkdir ~/bin
$ echo "export PATH=\$PATH:~/bin" >>~/.profile
$ echo "export EDITOR=nano" >>~/.profile
$ echo "export PS1='\[\e]2; \$PWD \a\e[32;1m\]\$PWD/\$\[\e[0m\] '" >>~/.profile

These preferences use the "Grass" look with a 12pt Andale Mono font. To use them:

cp Downloads/com.apple.Terminal.plist Library/Preferences/

Then "Force Quit" Terminal. (otherwise, Terminal will overwrite them on exit with it's current settings)

I also like a longer history:

$ echo "export HISTSIZE=2000" >>~/.profile

I use many external disks and Time Machine constantly asking me if I want to use that disk for backups annoys me. To disable it:

$ defaults write com.apple.TimeMachine DoNotOfferNewDisksForBackup -bool YES

Fix slow ssh response

When logging into an ssh server you may experience a long delay after authentication, and before you get the prompt. This will happen with most sshd servers on a home network or on a new network while it is being set up.

The reason is that sshd tries to do a reverse lookup on the connecting IP, which takes a while to time out.

To speed up the initial response of ssh, the solution is to prevent these reverse lookups, at least until the network has a working DNS which can resolve the connecting IPs to names. To do this, set "UseDNS no" in your sshd_config file and force sshd to re-read it's configuration.

sudo -s # if you are not root, like on Ubuntu or Mac

file=/etc/sshd_config
# or
file=/etc/ssh/sshd_config
# or on a Mac
file=/private/etc/sshd_config

perl -i.bak -pe 's/^\s*#?UseDNS\s+.*/UseDNS no/i' $file
grep -qi 'UseDNS no' $file || echo UseDNS no >> $file
# on Linux:
kill -HUP `cat /var/run/sshd.pid`

(the last kill line to force sshd re-read it's config file doesn't work on Mac)
While searching for this solution, I came across other configuration settings. They didn't apply to my case, but if you still have problems, you may want to set "GSSAPIKeyExchange no" in your client configuration file which is usually in /etc/ssh_config (ssh_ , not sshd_ !). Or look into IPv6 problems.

Now I have to find an equivalent solution for rsync.