El cheapo HSM on Linux using LVM

Needed

Linux kernel with support for device mapper and mirror target
LVM userland tools

Given

/dev/loop0, backed by a file on a SSD
/dev/loop1, backed by a file on a rotating hard disk

Create physical volumes:

# pvcreate /dev/loop0    
  Writing physical volume data to disk "/dev/loop0"
  Physical volume "/dev/loop0" successfully created
# pvcreate /dev/loop1
  Writing physical volume data to disk "/dev/loop1"
  Physical volume "/dev/loop1" successfully created

Create the volume group:

# vgcreate vgtest /dev/loop0 /dev/loop1
  Volume group "vgtest" successfully created

Create a logical volume on the SSD:

# lvcreate -n lvtest1 -L 64MB vgtest /dev/loop0
  Logical volume "lvtest1" created

Create another logical volume on the hard disk:

# lvcreate -n lvtest2 -L 64MB vgtest /dev/loop1
  Logical volume "lvtest2" created

Move logical volume lvtest1 to the hard disk:

# pvmove -n vgtest/lvtest1 /dev/loop0 /dev/loop1
  /dev/loop0: Moved: 25.0%
  /dev/loop0: Moved: 100.0%

Move logical volume lvtest2 to the SSD:

# pvmove -n vgtest/lvtest2 /dev/loop1 /dev/loop0
  /dev/loop1: Moved: 75.0%
  /dev/loop1: Moved: 100.0%

Check the distribution of the logical volumes among the physical volumes:

# pvdisplay -m /dev/loop{0,1}
  --- Physical volume ---
  PV Name               /dev/loop0
  VG Name               vgtest
  PV Size               128.00 MiB / not usable 4.00 MiB
  Allocatable           yes 
  PE Size               4.00 MiB
  Total PE              31
  Free PE               15
  Allocated PE          16
  PV UUID               hEIiA9-nq60-bCJI-OaPe-jSw3-YOqp-fFcbqY
   
  --- Physical Segments ---
  Physical extent 0 to 15:
    Logical volume	/dev/vgtest/lvtest2
    Logical extents	0 to 15
  Physical extent 16 to 30:
    FREE
   
  --- Physical volume ---
  PV Name               /dev/loop1
  VG Name               vgtest
  PV Size               256.00 MiB / not usable 4.00 MiB
  Allocatable           yes 
  PE Size               4.00 MiB
  Total PE              63
  Free PE               47
  Allocated PE          16
  PV UUID               gtf7QG-Etc5-u2Br-KEHP-LDdN-BhqJ-J9OKQd
   
  --- Physical Segments ---
  Physical extent 0 to 15:
    FREE
  Physical extent 16 to 31:
    Logical volume	/dev/vgtest/lvtest1
    Logical extents	0 to 15
  Physical extent 32 to 62:
    FREE

Move just the first five physical extents of the logical volume lvtest1 to the SSD:

# pvmove -n vgtest/lvtest1 /dev/loop1:16-20 /dev/loop0
  /dev/loop1: Moved: 100.0%

Further thoughts

Replace /dev/loop0 and /dev/loop1 by real block devices, e.g. /dev/sda1 (SSD) and /dev/sdb1 (hard disk)
Better: replace /dev/loop0 and /dev/loop1 by /dev/md0 and /dev/md1, where md0 is a RAID1 of SSDs and md1 is a RAID1 of hard disks. Otherwise, the volume group vgtest is just like a JBOD
One might use SystemTap to gather statistics about which blocks are more often accessed than others and to move just these blocks to the SSDs.