Block size limitations of a VMFS datastore

Symptoms

• Deploying a template or clone of a virtual machine fails with the error:
    
  A general system error occurred: File *.vmdk is larger than the maximum size supported by datastore
   
• Creating a virtual disk larger than a certain size fails with these errors:
  
  
  • In the ESX host command line:
     
    Failed to create virtual disk: There is not enough space on the file system for the selected operation.
     
  • In the Virtual Infrastructure (VI) Client for ESX 3.5 with Virtual Center 2.5, and in vSphere Client for ESX 4.x with vCenter Server 4.x, you see the warning:
    
    Create New Virtual Disk: The disk capacity entered was not a properly formed number or was out of range. It has been replaced with the nearest acceptable value. 

 Note: After you click OK, the number is adjusted to the largest file size possible.

• Physical to virtual or virtual to virtual machine conversions may fail with the error:
  The file system on the destination does not support the files

• In the Client VI for ESX 3.0 with VirtualCenter 2.0, you see the warning:
  The disk capacity specified is greater than the amount left on the datastore when it was last updated. Unless more space has become available since that update, creation of the virtual machine will fail. Do you wish to submit this task anyway?

Purpose

This article provides information on VMFS block sizes and the advantages and disadvantages associated with the various block sizes you can use to create a datastore.

Note: This article does not apply to datastores located on NFS volumes.

Resolution

VMFS Block Sizes

The block size on a VMFS datastore defines two things:

• The maximum file size
• The amount of space a file occupies

To determine the block size used by a datastore:

From the VI/vSphere Client:

1. Select an ESX host that contains the datastore.
2. Click the Configuration tab.
3. Click Storage.
4. Select the datastore.
5. The block size is identified in the Details window under the Formatting subheading.

From the ESX console:

1. Log in to the ESX console.
2. Run the command:
   
   # vmkfstools -P <path to datastore>
   
   The block size is in bold in this line:
   
   Capacity 429228294144 (409344 file blocks * 1048576), 8896118784 (8484 blocks) available
   Where 1048576 is equivalent to a 1 MB block size on a VMFS datastore.
   
   This table reports the various block sizes that can be found in this output:
   
   

   Block Size Value	Actual Block Size
   1048576	1 MB
   2097152	2 MB
   4194304	4 MB
   8388608	8 MB

   
   Note: vSphere 5.0 and later releases have a block size of 1 MB only.
   

VMFS-2 and VMFS-3 Size Limitations

There is no noticeable I/O performance difference when using a larger block size on thick provisioned virtual disks (VMDK).

Thin provisioned virtual disk (VMDK) performance, especially the performance of first writes, will be reduced as the block size of the VMFS datastore is increased, but subsequent writes to thin VMDK on any block size will be equivalent to eagerzeroedthick.

Therefore VMware recommends using a 1 MB block size when creating VMFS datastores for thin provisioned virtual disks for optimal performance. The benefits of using a smaller block size are:

• Better performance during first write to thin provisioned virtual disks.
• Minimize internal fragmentation, as the reason for using thin provisioned virtual disks is to save space. If a bigger block size is used, there will be more space wasted because of internal fragmentation.

The block size of the datastore also has an impact on the maximum file size (including virtual disk size being added to a virtual machine) that can be created on it. Therefore you must also consider the largest file or virtual disk size that you want to use when creating the VMFS datastore.

Note: Choose the VMFS block size carefully when creating VMFS datastores, because there is no way to change the block size of a VMFS datastore once it has been created. If you require a larger block size then the datastore will need to be recreated (this procedure is covered later in this article).

This table lists the maximum file and virtual disk sizes that are supported depending on the block size of the VMFS datastore:

Block Size	Largest virtual disk on VMFS-2	Largest virtual disk on VMFS-3
1 MB	456 GB	256 GB*
2 MB	912 GB	512 GB*
4 MB	1.78 TB	1 TB*
8 MB	2 TB	2 TB minus 512 B*
16 MB	2 TB	Invalid block size
32 MB	2 TB	Invalid block size
64 MB	2 TB	Invalid block size

* Overhead required to take snapshots of a virtual machine.

VMFS-3 uses sub-blocks for directories and small files with a size smaller than 64 KB. When the VMFS uses 1 sub-block of 64 KB, we switch to file blocks. The size of the file block depends on the block size you selected when the datastore was created.

In vSphere 4.0 and later, if snapshots are intended to be used, there are further size limitations on the virtual disk size as snapshots require an additional overhead. For more information, see Creating a snapshot for a virtual machine fails with the error: File is larger than maximum file size supported (1012384).

VMFS-5 Size Limitations

With VMFS-5, we use a unified 1 MB block size which is no longer configurable, but we can address larger files than a VMFS-3 1 MB block size can due to enhancements to the VMFS file system. Therefore a 1 MB VMFS-3 block size is not the same as a 1 MB VMFS-5 block size regarding file sizes.

The limits that apply to VMFS-5 datastores are:

• The maximum virtual disk (VMDK) size is 2 TB minus 512 B.
• The maximum virtual-mode RDM size is 2 TB minus 512 B.
• Physical-mode RDMs are supported up to 64 TB.

In VMFS-5, very small files (that is, files smaller than 1 KB) will be stored in the file descriptor location in the metadata rather than using file blocks. Once the file size increases beyond 1 KB, sub-blocks are used. After one 8 KB sub-block is used, 1 MB file blocks are used. As VMFS-5 uses sub-blocks of 8 KB rather than 64 KB (as in VMFS-3), this reduces the amount of disk space being used by small files. For more information on VMFS-5, see vSphere 5 FAQ: VMFS-5 (2003813).

Note on upgrading an existing datastore from VMFS-3 to VMFS-5
When upgrading a VMFS datastore from VMFS-3 to VMFS-5, you can extend a datastore past 2 TB - 512 B. The caveat to upgrading a VMFS-3 datastore to VMFS-5 is that it will inherit the block size properties of the original VMFS-3 datastore.

If you upgrade to VMFS-5 from VMFS-3 then regardless of the block size, VMFS-5 uses double-indirect addressing to cater for large files (up to a size of 2 TB - 512 B) on upgraded VMFS-3 volumes. For example, if the VMDK goes beyond 512 GB it will switch to using double-indirect addressing, which will allow for VMDKs up to 2 TB - 512 B.

Example of upgrading an existing VMFS-3 datastore to VMFS-5

Note: For this example, we will create a VMFS-3 volume and attempt to create a large (513 GB) file on that datastore. Then we will upgrade the datastore to VMFS-5 and test creating a large file on the upgraded datastore.

1. Create a VMFS-3 datastore:
   
   vmkfstools -Ph -v10 /vmfs/volumes/cs -ee-d67-local/
   
   VMFS-3.54 file system spanning 1 partitions.
   File system label (if any): cs-ee-d67-local
   Mode: public
   Capacity 837 GB, 836.4 GB available, file block size 2 MB
   
2. Test creation of a large file (a VMDK larger than 512 GB):
   
   # vmkfstools -c 513G /vmfs/volumes/cs-ee-d67-local/513GB.vmdk
   Failed to create virtual disk: The destination file system does not support large files (12).
   
3. Upgrade the datastore to VMFS-5:
   
   vmkfstools -T /vmfs/double indirect addressingvolumes/cs-ee-d67-local/
   
   Please ensure that the VMFS-3 volume /vmfs/volumes/502369f3-0f06dcd7-ee21-0024e84b3c30 is not in active use by any local or remote ESX 3.x/4.x server.
   
   We recommend the following:
   
   Back up data on your volume as a safety measure.
   Take precautions to ensure no ESX 3.x/4.x servers are accessing this volume.
   
   Continue converting VMFS-3 to VMFS-5?
   
   0) _Yes
   1) _No
   
   Select a number from 0-1:
   
   After entering 0 at the prompt, you see:
   
   Checking if remote hosts are using this device as a valid file system. This may take a few seconds...
   Upgrading file system /vmfs/volumes/cs-ee-d67-local/...
   done.
   
4. After the upgrade completes, verify that the block size is still 2 MB:
   
   vmkfstools -Ph -v10 /vmfs/volumes/cs-ee-d67-local/
   
   VMFS-5.54 file system spanning 1 partitions.
   File system label (if any): cs-ee-d67-local
   Mode: public
   Capacity 837 GB, 836.4 GB available, file block size 2 MB
   
5. Test creation of a large file again:
   
   vmkfstools -c 513G /vmfs/volumes/cs-ee-d67-local/513GB.vmdk
   
   Create: 100% done.

Note: For more information on upgrading to VMFS-5 and other criteria, see the VMFS-5 Upgrade Considerations Guide.

To recreate a datastore with a different block size

The block size on a datastore cannot be automatically changed as it is a file system property that can only be specified when the datastore is initially created.

The only way to increase the block size is to move all data off the datastore and recreate it with the larger block size. The preferred method of recreating the datastore is from a console or SSH session, as you can simply recreate the file system without having to make any changes to the disk partition.

Note: All data on a VMFS volume is lost when the datastore is recreated. Migrate or move all virtual machines and other data to another datastore. Back up all data before proceeding.

From the ESX/ESXi console:

Note: This procedure should not be performed on a local datastore on an ESX host where the operating system is located, as it may remove the Service Console privileged virtual machine which is located there.

1. Storage vMotion, move, or delete the virtual machines located on the datastore you would like to recreate with a different block size.
2. Log into the Local Tech Support Mode console of the ESX/ESXi host. For more information, see:
• Unable to connect to an ESX host using Secure Shell (SSH) (1003807)
• Tech Support Mode for Emergency Support (1003677)
• Using Tech Support Mode in ESXi 4.1 and ESXi 5.x (1017910)
  
3. Use the esxcfg-scsidevs command to obtain the disk identifier (mpx, naa, or eui) for the datastore you want to recreate:
   
   # esxcfg-scsidevs -m
   
4. Use vmkfstools to create a new VMFS datastore file system with a different block size over the existing one:
   
   # vmkfstools -C VMFS-type -b Block-Size -S Datastore-Name/vmfs/devices/disks/Disk-Identifier:Partition-Number
   
   Example command to create a datastore called NewDatastore with an 8 MB block size:
   
   # vmkfstools -C vmfs3 -b 8m -S NewDatastore /vmfs/devices/disks/naa.60901234567890123456789012345678:1
   
5. Rescan from all other ESX hosts with the vmkfstools -V command.

From the VI / vSphere Client:

Note: This procedure should not be performed on a LUN containing the ESX/ESXi operating system, as it may require additional effort to recreate the partition table.

1. Storage vMotion, move, or delete the virtual machines located on the datastore you would like to recreate with a different block size.
2. Select the ESX/ESXi host in the inventory and click the Configuration tab.
3. Select the Storage under hardware, right-click the datastore and choose Delete.
   
   Note: Do not do this on a datastore located on the same disk/LUN as the ESX/ESXi operating system.
   
4. Rescan for VMFS volumes from the other hosts that can see the datastore.
5. Create the new datastore with the desired block size on one of the hosts using the Add Storage Wizard.
6. Rescan for VMFS volumes from all other hosts that can see the datastore.

Note: If extending a VMDK over 1 TB (1023 GB) on an ESX 3.5 host using the GUI fails, see Cannot extend a vmdk over 1TB (1008528).

Additional Information

To create a VMFS datastore larger than 2 TB, you must extend the VMFS datastore over another LUN.

• For information on increasing the capacity of the datastore in ESX 3.x, see Adding Extents to Datastores in the ESX Server 3 Configuration Guide.
• For information on increasing the capacity of the datastore in ESX 4.x, see Increase VMFS Datastores in the ESX 4.x Configuration Guide.
• For vSphere 5.0 storage configuration maximums pertaining to 2 TB LUN support, see Configuration Maximums for VMware vSphere 5.0.

Source:-
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1003565