This means that if the namenode metadata is large, restarts will be slower. When a NameNode restarts, it must load the filesystem metadata from local disk into memory. Hence, we can see that we require more than 100x memory on the Namenode heap to store the multiple small files as opposed to one big 192MB file. The total memory required by the Namenode to store the metadata of these files = 150 x (192 + (192 x 3)) = 115200 Bytes.

These files are then replicated across the cluster. In contrast, scenario 2 has 192 1 MB files. of blocks x Replication Factor)).Īccording to this calculation, the total memory required to store the metadata of this file on the Namenode = 150 x (1 + (2 x 3)) = 1050 Bytes. After replication, the total memory required to store the metadata of a file is = 150 bytes x (1 file inode + (No. Scenario 1 has one file which is 192MB which is broken down to 2 blocks of size 128MB and 64MB. The two scenarios below illustrate the small files issue: This arrangement is described in more detail in the public documentation here. The HDFS namespace tree and associated metadata are maintained as objects in the NameNode’s memory (and backed up to disk), each of which occupies approximately 150 bytes, as a rule of thumb. Files are split into blocks that are stored and replicated on the DataNodes’ local file systems across the cluster. In HDFS, data and metadata are separate entities.

Before we get to the implications of this, let’s review how HDFS stores files. Hadoop’s storage and application layers are not designed to function efficiently with a large number of small files. Hence, in this section, we shall discuss why it is a good goal to have a file size as close to a multiple of the HDFS block size as possible. But when small files become a significant part of datasets, the problems arise. These are files like library jars, XML configuration files, temporary staging files, and so on. One should note that it is expected and inevitable to have some small files on HDFS. In this blog post, we will define the issue of small file storage and examine ways to tackle it while keeping the complications at bay.Ī small file is one which is significantly smaller than the default Apache Hadoop HDFS default block size (128MB by default in CDH). However, HDFS stores small files inefficiently, leading to inefficient Namenode memory utilization and RPC calls, block scanning throughput degradation, and reduced application layer performance. The Apache Hadoop Distributed File System (HDFS) was developed to store and process large data sets over the range of terabytes and petabytes. Small files are a common challenge in the Apache Hadoop world and when not handled with care, they can lead to a number of complications.

Small filesafe