Viewing Dynamic Resource Pool Configuration

To view dynamic resource pool configuration:

1. Select Clusters > Cluster name > Dynamic Resource Pool Configuration. If the cluster has a YARN service, the YARN > Resource Pools tab displays. If the cluster has an Impala service enabled for dynamic resource pools, the Impala Admission Control > Resource Pools tab displays.
2. Click the YARN or Impala Admission Control tab.

Creating a YARN Dynamic Resource Pool

There is always a resource pool named root.default. By default, all YARN applications run in this pool. You create additional pools when your workload includes identifiable groups of applications (such as from a particular application, or a particular group within your organization) that have their own requirements.

1. Select Clusters > Cluster name > Dynamic Resource Pool Configuration. If the cluster has a YARN service, the YARN > Resource Pools tab displays. If the cluster has an Impala service enabled for dynamic resource pools, the Impala Admission Control > Resource Pools tab displays.
2. Click the YARN tab.
3. Click Create Resource Pool. The Create Resource Pool dialog box displays, showing the Resource Limits tab.
4. Specify a name and resource limits for the pool:
   • In the Resource Pool Name field, specify a unique pool name containing only alphanumeric characters. If referencing a user or group name that contains a ".", replace the "." with "_dot_".
   • To make the pool a parent pool, select the Parent Pool checkbox.
   • Define a configuration set by assigning values to the listed properties. Specify a weight that indicates that pool's share of resources relative to other pools, minimum and maximum for virtual cores and memory, and a limit on the number of applications that can run simultaneously in the pool. If this is a parent pool, you can also set the maximum resources for undeclared children.
   • Click the Scheduling Policy tab and select a policy:
     • Dominant Resource Fairness (DRF) (default) - An extension of fair scheduling for more than one resource. DRF determines CPU and memory resource shares based on the availability of those resources and the job requirements.
     • Fair (FAIR) - Determines resource shares based on memory.
     • First-In, First-Out (FIFO) - Determines resource shares based on when a job was added.
   • If you have enabled Fair Scheduler preemption, click the Preemption tab and optionally set a preemption timeout to specify how long a job in this pool must wait before it can preempt resources from jobs in other pools. To enable preemption, follow the procedure in Enabling and Disabling Fair Scheduler Preemption.
   • If you have enabled ACLs and specified users or groups, optionally click the Submission and Administration Access Control tabs to specify which users and groups can submit applications and which users can view all and kill applications. By default, anyone can submit, view all, and kill applications. To restrict these permissions, select Allow these users and groups and provide a comma-delimited list of users and groups in the Users and Groups fields respectively.
5. Click Create.
6. Click Refresh Dynamic Resource Pools.

Enabling and Disabling Dynamic Resource Pools for Impala

By default, admission control and dynamic resource pools for Impala are disabled. Only when both are enabled does the Impala Admission Control tab appear in the Dynamic Resource Pool Configuration tab. To enable and disable Impala dynamic resource pools, follow the procedure in Enabling and Disabling Impala Admission Control Using Cloudera Manager.

Creating an Impala Dynamic Resource Pool

There is always a resource pool designated as root.default. By default, all Impala queries run in this pool when the dynamic resource pool feature is enabled for Impala. You create additional pools when your workload includes identifiable groups of queries (such as from a particular application, or a particular group within your organization) that have their own requirements for concurrency, memory use, or service level agreement (SLA). Each pool has its own settings related to memory, number of queries, and timeout interval.

1. Select Clusters > Cluster name > Dynamic Resource Pool Configuration. If the cluster has a YARN service, the YARN > Resource Pools tab displays. If the cluster has an Impala service enabled for dynamic resource pools, the Impala Admission Control > Resource Pools tab displays.
2. Click the Impala Admission Control tab.
3. Click Create Resource Pool.
4. Specify a name and resource limits for the pool:
   • In the Resource Pool Name field, type a unique name containing only alphanumeric characters.
   • Optionally click the Submission Access Control tab to specify which users and groups can submit queries. By default, anyone can submit queries. To restrict this permission, select the Allow these users and groups option and provide a comma-delimited list of users and groups in the Users and Groups fields respectively.
5. Click Create.
6. Click Refresh Dynamic Resource Pools.

Choosing Settings for an Impala Dynamic Resource Pool

Impala dynamic resource pools support the following settings:

Max Memory

The maximum amount of aggregate memory, cluster-wide, that can be used by all queries running concurrently in the pool. In conjunction with Default Query Memory Limit and the number of Impala hosts in the cluster, Impala determines the expected maximum memory used by all queries in the pool and holds back any further queries once the estimated upper limit is reached.

  Note: If you specify Max Memory for an Impala dynamic resource pool, you must also specify the Default Query Memory Limit. Max Memory relies on the Default Query Memory Limit to produce a reliable estimate of overall memory consumption for a query.

For example, consider the following scenario:

• The cluster is running impalad daemons on five DataNodes.
• A dynamic resource pool has Max Memory set to 100 GB.
• The Default Query Memory Limit for the pool is 10 GB. Therefore, any query running in this pool could use up to 50 GB of memory (default query memory limit * number of Impala nodes).
• The maximum number of queries that Impala executes concurrently within this dynamic resource pool is two, which is the most that could be accomodated within the 100 GB Max Memory cluster-wide limit.
• There is no memory penalty if queries use less memory than the Default Query Memory Limit per-host setting or the Max Memory cluster-wide limit. These values are only used to estimate how many queries can be run concurrently within the resource constraints for the pool.

Max Running Queries

The maximum number of queries that can run concurrently in this pool. The default value is unlimited. Any queries for this pool that exceed Max Running Queries are added to the admission control queue until other queries finish. You can use Max Running Queries in the early stages of resource management, when you do not have extensive data about query memory usage, to determine if the cluster performs better overall if throttling is applied to Impala queries.

For a workload with many small queries, you typically specify a high value for this setting, or leave the default setting of "unlimited". For a workload with expensive queries, where some number of concurrent queries saturate the memory, I/O, CPU, or network capacity of the cluster, set the value low enough that the cluster resources are not overcommitted for Impala.

Once you have enabled memory-based admission control using other pool settings, you can still use Max Running Queries as a safeguard. If queries exceed either the total estimated memory or the maximum number of concurrent queries, they are added to the queue.

Max Queued Queries

The maximum number of queries that can be in the admission control queue for a pool at any one time. Additional queries cannot be added to the queue until other queries begin running, reducing the queue size. The default value is 200. Typically, this value does not need to be adjusted. If a large number of queries are queued, you can change other parameters such as the timeout interval, or you can increase the pool capacity.

Queue Timeout

The amount of time, in milliseconds, that a query waits in the admission control queue for this pool before being canceled. The default value is 60,000 (60 seconds).

It the following cases, Queue Timeout is not significant and you can specify a high value to avoid canceling queries unexpectedly:

• In a low-concurrency workload, where few or no queries are queued
• In an environment without a strict SLA, where it does not matter if queries occasionally take longer than usual because they are held in admission control

You might also need to increase the value to use Impala with some business intelligence tools that have their own timeout intervals for queries.

In a high-concurrency workload, especially for queries with a tight SLA, long wait times in admission control can cause a serious problem. For example, if a query needs to run in 10 seconds, and you have tuned it so that it runs in 8 seconds, it violates its SLA if it waits in the admission control queue longer than 2 seconds. In a case like this, set a low timeout value and monitor how many queries are cancelled because of timeouts. This technique helps you to discover capacity, tuning, and scaling problems early, and helps avoid wasting resources by running expensive queries that have already missed their SLA.

If you identify some queries that can have a high timeout value, and others that benefit from a low timeout value, you can create separate pools with different values for this setting.

Default Query Memory Limit

The equivalent of setting the MEM_LIMIT query option for each query that runs in this pool. This value represents the maximum memory the query can use on each host. If the query exceeds that memory on a host, it activates spill-to-disk, and could be canceled if available memory is too low. Impala multiplies this default memory limit value by the number of Impala hosts in the cluster to estimate how many queries fit within the total RAM represented by Max Memory, which represents a cluster-wide limit.

Cloning a Resource Pool

Deleting a Resource Pool

Editing Dynamic Resource Pools

1. Select Clusters > Cluster name > Dynamic Resource Pool Configuration. If the cluster has a YARN service, the YARN > Resource Pools tab displays. If the cluster has an Impala service enabled for dynamic resource pools, the Impala Admission Control > Resource Pools tab displays.
2. Click the YARN or Impala Admission Control tab.
3. Click Edit at the right of a resource pool row. Edit the properties.
4. If you have enabled ACLs and specified users or groups, optionally click the Submission and Administration Access Control tabs to specify which users and groups can submit applications and which users can view all and kill applications. By default, anyone can submit, view all, and kill applications. To restrict these permissions, select Allow these users and groups and provide a comma-delimited list of users and groups in the Users and Groups fields respectively.
5. Click Save.
6. Click Refresh Dynamic Resource Pools.

Viewing Dynamic Resource Pool Status

Configuring Default YARN Fair Scheduler Properties

For information on the default properties, see Configuring the Fair Scheduler.

1. Select Clusters > Cluster name > Dynamic Resource Pool Configuration. The YARN > Resource Pools tab displays.
2. Click the YARN tab.
3. Click the Default Settings button.
4. Specify the default scheduling policy, maximum applications, and preemption timeout properties.
5. Click Save.
6. Click Refresh Dynamic Resource Pools.

Creating a Resource Subpool

YARN resource pools can form a nested hierarchy. To manually create a subpool:

1. Select Clusters > Cluster name > Dynamic Resource Pool Configuration. The YARN > Resource Pools tab displays.
2. Click at the right of a resource pool row and select Create Subpool. Configure subpool properties.
3. Click Create.
4. Click Refresh Dynamic Resource Pools.

Setting YARN User Limits

Pool properties determine the maximum number of applications that can run in a pool. To limit the number of applications specific users can run at the same time:

1. Select Clusters > Cluster name > Dynamic Resource Pool Configuration. The YARN > Resource Pools tab displays.
2. Click the User Limits tab. The table displays a list of users and the maximum number of jobs each user can submit.
3. Click Add User Limit.
4. Specify a username. containing only alphanumeric characters. If referencing a user or group name that contains a ".", replace the "." with "_dot_".
5. Specify the maximum number of running applications.
6. Click Create.
7. Click Refresh Dynamic Resource Pools.

Configuring ACLs

To configure which users and groups can submit and kill applications in any resource pool:

1. Enable ACLs.
2. Select Clusters > Cluster name > Dynamic Resource Pool Configuration. The YARN > Resource Pools tab displays.
3. Click the Access Control Settings button.
4. In the Admin ACL property, specify which users and groups can submit and kill applications.
5. Click Save Changes to commit the changes.
6. Return to the Home page by clicking the Cloudera Manager logo.
7. Click the icon next to any stale services to invoke the cluster restart wizard.
8. Click Restart Stale Services.
9. Click Restart Now.
10. Click Finish.

Enabling ACLs

To specify whether ACLs are checked:

1. Select Clusters > Cluster name > Dynamic Resource Pool Configuration. The YARN > Resource Pools tab displays.
2. Click the Access Control Settings button.
3. In the Enable ResourceManager ACLs property, select the YARN service.
4. Click Save Changes to commit the changes.
5. Return to the Home page by clicking the Cloudera Manager logo.
6. Click the icon next to any stale services to invoke the cluster restart wizard.
7. Click Restart Stale Services.
8. Click Restart Now.
9. Click Finish.

Specifying Resource Limit Properties

1. Select Clusters > Cluster name > Dynamic Resource Pool Configuration. If the cluster has a YARN service, the YARN > Resource Pools tab displays. If the cluster has an Impala service enabled for dynamic resource pools, the Impala Admission Control > Resource Pools tab displays.
2. Click the Resource Pools tab.
3. For each resource pool, click Edit.
   1. Select a configuration set name.
   2. Edit the configuration set properties and click Save.
4. Click Refresh Dynamic Resource Pools.

Viewing the Properties of a Configuration Set

1. Select Clusters > Cluster name > Dynamic Resource Pool Configuration. If the cluster has a YARN service, the YARN > Resource Pools tab displays. If the cluster has an Impala service enabled for dynamic resource pools, the Impala Admission Control > Resource Pools tab displays.
2. In the Configuration Sets drop-down list, select a configuration set. The properties of each pool for that configuration set display.

Adding a Scheduling Rule

1. Select Clusters > Cluster name > Dynamic Resource Pool Configuration. If the cluster has a YARN service, the YARN > Resource Pools tab displays. If the cluster has an Impala service enabled for dynamic resource pools, the Impala Admission Control > Resource Pools tab displays.
2. Click the Scheduling Rules tab.
3. Click Create Scheduling Rule.
4. In the Configuration Set field, choose the configuration set to which the rule applies. Select Create New or Use Existing.
5. If you create a new configuration set, type a name in the Name field.

   If you use an existing configuration set, select one from the drop-down list.

6. Configure the rule to repeat or not:
   • To repeat the rule, keep the Repeat field selected and specify the repeat frequency. If the frequency is weekly, specify the repeat day or days.
   • If the rule does not repeat, clear the Repeat field, click the left side of the on field to display a drop-down calendar where you set the starting date and time. When you specify the date and time, a default time window of two hours is set in the right side of the on field. Click the right side to adjust the date and time.
7. Click Create.
8. Click Refresh Dynamic Resource Pools.

Reordering Scheduling Rules

1. Select Clusters > Cluster name > Dynamic Resource Pool Configuration. If the cluster has a YARN service, the YARN > Resource Pools tab displays. If the cluster has an Impala service enabled for dynamic resource pools, the Impala Admission Control > Resource Pools tab displays.
2. Click the Scheduling Rules tab.
3. Click Reorder Scheduling Rules.
4. Click Move Up or Move Down in a rule row.
5. Click Save.
6. Click Refresh Dynamic Resource Pools.

Editing a Scheduling Rule

1. Select Clusters > Cluster name > Dynamic Resource Pool Configuration. If the cluster has a YARN service, the YARN > Resource Pools tab displays. If the cluster has an Impala service enabled for dynamic resource pools, the Impala Admission Control > Resource Pools tab displays.
2. Click Scheduling Rules.
3. Click Edit at the right of a rule.
4. Edit the rule.
5. Click Save.
6. Click Refresh Dynamic Resource Pools.

Deleting a Scheduling Rule

1. Select Clusters > Cluster name > Dynamic Resource Pool Configuration. If the cluster has a YARN service, the YARN > Resource Pools tab displays. If the cluster has an Impala service enabled for dynamic resource pools, the Impala Admission Control > Resource Pools tab displays.
2. Click Scheduling Rules.
3. Click at the right of a rule and select Delete.
4. Click OK.
5. Click Refresh Dynamic Resource Pools.

Placement Rule Ordering and Evaluation

Pool placement rules are evaluated in the order in which they appear in the placement rule list. When a job is submitted, the rules are evaluated, and the first matching rule is used to determine the pool in which the job is run.

If a rule is always satisfied, subsequent rules are not evaluated. Rules that are never evaluated appear in struck-though gray text. For an example see Example Placement Rules.

By default, pool placement rules are ordered in reverse order of when they were added; the last rule added appears first. You can easily reorder rules.

Creating Pool Placement Rules

Placement rules determine the pools to which applications and queries are assigned. At installation, Cloudera Manager provides a set of default rules and rule ordering.

Default Placement Rules and Order

Reordering Pool Placement Rules

To change the order in which pool placement rules are evaluated:

1. Select Clusters > Cluster name > Dynamic Resource Pool Configuration. If the cluster has a YARN service, the YARN > Resource Pools tab displays. If the cluster has an Impala service enabled for dynamic resource pools, the Impala Admission Control > Resource Pools tab displays.
2. Click the YARN or Impala Admission Control tab.
3. Click the Placement Rules tab.
4. Click Reorder Placement Rules.
5. Click Move Up or Move Down in a rule row.
6. Click Save.
7. Click Refresh Dynamic Resource Pools.