Recovering an Oracle Cloud instance

in Cloud & Serverless

Use case: you have an Oracle Cloud instance but lost SSH access to it. To recover access, you need to know which pubkey you used on it, or which port SSH is listening on. Or, you don't want to terminate it before you make sure it doesn't contain important data.

Solution: you will have to attach the (problematic) boot volume of that instance to another instance, either some other already existing instance or a 'dummy' instance created specifically for the purpose.

Getting boot volume ready

You have two options: A. re-attach the problematic boot volume itself, or B. create a clone.

A. Re-attach boot volume without cloning

Open the problematic instance

  1. Stop the instance
  2. Go to Storage → Boot Volumes → Detach

When the boot volume is detached and is no longer a boot volume of some instance, it can be attached to another instance as storage volume.

B. Clone boot volume

  1. Go to https://cloud.oracle.com/block-storage/boot-volumes
  2. Find the boot volume of the instance you need to recover. Click the three dots on it and Create Clone. Give the new clone a name that makes sense (so you can recognise it later when cleaning up).

(Optional) Create an empty dummy instance

If you don't have another instance which you could use temporarily, create a new one.

Attach volume to temporary instance

  1. Open your temporary instance and under Storage (previously Resources) click Attached block volumes → Attach block volume
  2. In the pop-up that appears:
    • Volume: choose your cloned volume
    • Attachment type: Custom / Paravirtualized
    • Access: Read/write

At this point, your dummy instance should see the newly attached volume without rebooting

Mount problematic volume on temporary instance

SSH into your temporary instance and elevate to root (sudo su -)

  1. Run lsblk -o NAME,FSTYPE,LABEL,SIZE,MOUNTPOINT (alternatively, fdisk -l) to list the volumes attached to your instance. The cloned volume you just attached is likely to be listed last and it's partitions won't have a mountpoint. For example:

    root:~# lsblk -o NAME,FSTYPE,LABEL,SIZE,MOUNTPOINT 
NAME    FSTYPE LABEL   SIZE MOUNTPOINT
sda                   46.6G 
├─sda1  ext4          46.5G /
└─sda15 vfat           127M /boot/efi
sdb                   46.6G 
├─sdb1  vfat   EFISYS 32.5M 
├─sdb2                   1G 
└─sdb3  ufs    rootfs 45.5G
    • sdb1 is the EFI system partition (bootloader)
    • sdb2 is likely a swap or boot partition
    • sdb3 (ufs, rootfs, 45.5G) is the main root filesystem (containing the OS) that we will mount below
  2. Create a temporary directory to mount into, e.g. mkdir /mnt/recovery
  3. mount -t auto -v /dev/sdb3 /mnt/recovery The sdb3 here was obtained in step one, your volume may differ. Use the volume that contains the main root filesystem. Note: Your new OS may not recognise the filesystem type of the problematic volume. In our case, we're mounting a UFS filesystem on Debian. To make that work, you will have to pass the type explicitly: mount -t ufs -o ro,ufstype=ufs2 /dev/sdb3 /mnt/recovery (-o ro means read-only - Linux can't write to UFS)

The files from the attached volume are available at /mnt/recovery. You can diagnose the problem, check logs and if you mounted with rw (read, write) support, you can also edit the necessary files to fix the issue on the problematic volume.

Cleaning up

Make sure you cd outside mounted volume (else you will get target is busy error) and then unmount.

cd ~ && umount /mnt/recovery

Unmounting is neccessary to save your changes. Modern filesystems cache writes in memory. When you umount, the kernel ensures everything is physically written.

You can now reattach the problematic volume as boot volume back to the original instance.

Oracle may charge you for the extra resources you just created. Clean up to avoid unnecessary costs

  1. Terminate the dummy instance
  2. Terminate the boot volume for the dummy instance
  3. Terminate the cloned volume

Additional resources:

#oracle-cloud-infrastructure