Configuration Settings
DFS-Perf has two categories of configuration parameters. One is the Environment Configurations which is for the DFS-Perf framework. Another is the Workloads Configurations which is used to custom the workloads.
Environment Configurations
These configuration contains the environment settings to run DFS-Perf. First, in conf/slaves, you can configure the nodes to launch DFS-Perf process, and you can repeat the node name to launch multi-process, e.g.
slaveA
slaveA
slaveB
slaveB
means to launch two DFS-Perf processes on slaveA and slaveB.
Then, the environment configurations are set in conf/dfs-perf-env.sh
| Property Name | Example | Meaning |
|---|
| JAVA |
/usr/lib/jvm/bin/java |
Java installation path. |
| DFS_PERF_DFS_ADRESS |
alluxio://master:19998 |
The URI address of the targeted DFS. |
| DFS_PERF_WORKSPACE |
/tmp/dfs-perf-workspace |
The workspace folder's path of DFS-Perf in targeted DFS. |
| DFS_PERF_OUT_DIR |
$DFS_PERF_HOME/result |
The performance result output dir. |
| DFS_PERF_MASTER_HOSTNAME |
master |
The DFS-Perf master service address. |
| DFS_PERF_MASTER_PORT |
23333 |
The DFS-Perf master service port. |
| DFS_PERF_THREADS_NUM |
4 |
The thread num of each DFS-Perf process. |
| DFS_PERF_UNREGISTER_TIMEOUT_MS |
10000 |
The slave is considered to be failed if not register in this time. |
| DFS_PERF_STATUS_DEBUG |
false |
If true, it will print the names of those running and remaining nodes. |
| DFS_PERF_FAILED_ABORT |
true |
If true, the test will abort when the number of failed nodes more than the threshold. |
| DFS_PERF_FAILED_PERCENTAGE |
1 |
The threshold percentage for failure abort. |
Workloads Configurations
Each workload has its own configurations and can be modified in conf/testsuite/{workload.name}
| Property Name | Example | Meaning |
|---|
| clients.per.thread |
20 |
the number of clients to connect to the file system for each thread |
| op.second.per.thread |
5 |
the metadata operations time for each thread, in seconds |
SimpleWrite
| Property Name | Example | Meaning |
|---|
| files.per.thread |
2 |
the number of files to write for each write thread |
| block.size.bytes |
134217728 |
the block size of a file |
| file.length.bytes |
134217728 |
the file size of write test, in bytes |
| buffer.size.bytes |
4194304 |
the size of the buffer write once |
| write.type |
ASYNC_THROUGH |
the WriteType of the write operation, now only used for Alluxio |
SimpleRead
| Property Name | Example | Meaning |
|---|
| files.per.thread |
2 |
the number of files to read for each read thread |
| read.mode |
SEQUENCE |
the read mode of read test, should be RANDOM or SEQUENCE. This is used to choose which files to read |
| buffer.size.bytes |
4194304 |
the size of the buffer read once |
| read.type |
CACHE_PROMOTE |
the ReadType of the write operation, now only used for Alluxio |
SkipRead
| Property Name | Example | Meaning |
|---|
| files.per.thread |
2 |
the number of files to read for each read thread |
| read.mode |
SEQUENCE |
the read mode of read test, should be RANDOM or SEQUENCE. This is used to choose which files to read |
| buffer.size.bytes |
4194304 |
the size of the buffer read once |
| read.type |
CACHE_PROMOTE |
the ReadType of the write operation, now only used for Alluxio |
| read.bytes |
16777216 |
the read bytes for each skip-and-read operator |
| skip.bytes |
16777216 |
the skip bytes for each skip-and-read operator |
| skip.mode |
FORWARD |
the skip mode, should be FORWARD or RANDOM |
| skip.times.per.file |
2 |
the skip-and-read times for each read file |
Mixture
| Property Name | Example | Meaning |
|---|
| basic.files.per.thread |
2 |
the number of basic files to write at the beginning for each thread |
| read.files.per.thread |
4 |
the number of files to read for each thread |
| write.files.per.thread |
1 |
the number of files to write for each thread |
| block.size.bytes |
134217728 |
the block size of a file |
| file.length.bytes |
134217728 |
the file size in bytes |
| buffer.size.bytes |
4194304 |
the size of the buffer read and write once |
| read.type |
CACHE_PROMOTE |
the ReadType of the read operation, now only used for Alluxio |
| write.type |
ASYNC_THROUGH |
the WriteType of the write operation, now only used for Alluxio |
Iterate
| Property Name | Example | Meaning |
|---|
| iterations |
2 |
the number of the write-read iteration |
| shuffle.mode |
false |
shuffle mode means it may read remotely |
| read.files.per.thread |
2 |
the number of files to read for each thread |
| write.files.per.thread |
2 |
the number of files to write for each thread |
| block.size.bytes |
134217728 |
the block size of a file |
| file.length.bytes |
134217728 |
the file size in bytes |
| buffer.size.bytes |
4194304 |
the size of the buffer read and write once |
| read.type |
CACHE_PROMOTE |
the ReadType of the read operation, now only used for Alluxio |
| write.type |
ASYNC_THROUGH |
the WriteType of the write operation, now only used for Alluxio |
Massive
| Property Name | Example | Meaning |
|---|
| time.seconds |
5 |
the time to do the global read/write |
| shuffle.mode |
false |
shuffle mode means it may read remotely |
| basic.files.per.thread |
2 |
the number of global files to write at the beginning for each thread |
| block.size.bytes |
134217728 |
the block size of a file |
| file.length.bytes |
134217728 |
the file size in bytes |
| buffer.size.bytes |
4194304 |
the size of the buffer read and write once |
| read.type |
CACHE_PROMOTE |
the ReadType of the read operation, now only used for Alluxio |
| write.type |
ASYNC_THROUGH |
the WriteType of the write operation, now only used for Alluxio |
