Effectively leveraging the ServiceNow platform requires a deep understanding of the complex interactions between rules and processes that can directly impact performance and usability. Specifically, the execution order of Transaction Quota Rules can present unexpected challenges when processing large data loads causes delays. In this article, Appmore’s ServiceNow consultant Niko Sääski shares practical insights on how Transaction Quota Rule settings affect performance and offers tips on customizing these rules to meet the specific needs of different users.
Timeout challenges in REST table API integration
I faced a problem where an integration partner was experiencing timeout issues using the REST Table API. They query our table for a big data load at set intervals. As time has gone by, there are more records in the table, and the queries are taking longer to complete. There is an OOB (out-of-the-box) Transaction Quota Rule, “REST Table API request timeout,” which catches REST Table API transactions. The timeout is set to 60 seconds. Logs revealed this to be the culprit, as the transactions were taking over 60 seconds and were thus getting timed out by the rule.
How I addressed the timeout issue with a targeted exception for Integration Users
I figured the easy solution would be to set an exception for this specific integration user, allowing their transactions to last longer than 60 seconds. I did not want to modify OOB records or raise the timeout limit for all REST Table API transactions. The “Transaction Quota Rule” table has a handy field, “Execution Order,” so I could create a copy of the OOB rule with a lower order to target the integration user. However, it doesn’t work as one might expect. In this article, I’ll demonstrate the issue and provide examples for re-creating it.
1. We need an API call that should take some time to execute. The lowest timeout limit (max_duration) that can be set in the sysrule_quota is 5 seconds. I’m using the “Incident” table, to which I created thousands of new records with a script. My example API call is at the end of this article, which selects many fields to ensure the call takes a long time.
2. Create an integration user so we can target that user in the sysrule_quota conditions. My user has the username special.table.api.user.
3. Create the new Transaction Quota Rule. You can use the OOB rule as a base /sysrule_quota.do?sys_id=55841f53ff2102003434ffffffffff39. Here are the values I set:
name: Custom REST Table API request timeout
order: -9,001
max_duration: 300
condition: type=rest^urlMATCH_RGX.*/api/now(/v[0-9]+)?/table.*^user=special.table.api.user^EQ
The rule condition differs from the OOB rule by specifically catching my API user, having a much lower order, and a higher timeout.
4. Optional: I changed the max_duration of the OOB rule “REST Table API request timeout” to 10 seconds to make my testing faster.
5. Create a system property glide.quota.manager.debug for debugging and set it to true. Documentation link.
6. Consume the API (I used Postman) and check the results in “Node Log File Browser” (ui_page_process). Filter by thread name “glide.quota.manager” to reduce the number of rows. Here are the interesting parts:
02:37:35.055 - glide.quota.manager - QuotaFinder: > Assigning quota "Custom REST API request timeout" with filter: type=rest^userSTARTSWITHmagic_user^EQ to transaction: URL= /api/now/table/redacted, THREAD= API_INT-thread-2, FG= true, TYPE= 7, STATE= 1, USER= magic_user, TIME= 1,116, MEM= 0, ATTRIBUTES= {X-Transaction-Source=Session-Type=non-interactive, user-agent=PostmanRuntime/7.40.0}console.log( 'Code is Poetry' );
02:37:35.055 - glide.quota.manager - QuotaFinder: > Assigning quota "REST Table API request timeout" with filter: type=rest^urlMATCH_RGX.*/api/now(/v[0-9]+)?/table.*^EQ to transaction: URL= /api/now/table/redacted, THREAD= API_INT-thread-2, FG= true, TYPE= 7, STATE= 1, USER= magic_user, TIME= 1,116, MEM= 0, ATTRIBUTES= {X-Transaction-Source=Session-Type=non-interactive, user-agent=PostmanRuntime/7.40.0}
02:37:35.057 - glide.quota.manager - QuotaFinder: > Assigning quota "REST and JSON Catch All" with filter: type=rest^ORtype=json^urlNOT LIKEsysparm_payload_type=edge_encryption_distribution^urlNOT LIKEsysparm_payload_type=machine_learning_download^urlNOT LIKEsysparm_payload_type=distribution_download^EQ to transaction: URL= /api/now/table/redacted, THREAD= API_INT-thread-2, FG= true, TYPE= 7, STATE= 1, USER= magic_user, TIME= 1,118, MEM= 0, ATTRIBUTES= {X-Transaction-Source=Session-Type=non-interactive, user-agent=PostmanRuntime/7.40.0}
02:37:35.057 - glide.quota.manager - QuotaManager: Quota exceeded for "REST Table API request timeout", requesting cancel
From `syslog` table
Cancelling transaction #386824 /api/now/table/incident (maximum execution time exceeded): Thread API_INT-thread-2 (special.table.api.user, [redacted]), after 10632ms
Why the order field Is ineffective in transaction quota rules?
It turns out that the QuotaFinder continues to match the transaction to rules even after a match is found. The documentation reveals the following:
“Note: The Order field on a quota rule affects the order in which the quota rules are checked. The Quota Manager checks lower-order rules first (for example, order 80 before order 90), but ultimately checks all rules. This might have marginal performance implications, depending on the conditions of the rules involved.” Source: Documentation link
“Order: A number that represents the priority of the quota transaction in relation to other quotas. The transaction quota with the lowest timeout, the lowest order value, and matching conditions determines the applied quota policy.” Source: Documentation link
From my perspective, this effectively makes the order field useless in cases where one wants to have a higher timeout limit, since all matching rules are checked anyways, and the lowest limit values prevail. There’s no field akin to “Stop processing” as in “Inbound Email Actions.” Also, the rules are evaluated within milliseconds of each other.
Clarifications from ServiceNow support and conclusions
I contacted ServiceNow Support, and they confirmed as much:
“The order means which condition will be checked first, and that is it. If other conditions match and have lower limits, then order becomes irrelevant… For transaction quotas, if there is any quota rule that the transaction has violated, it is canceled. The order has little practical consequence: the transaction is canceled based on the “first” rule that it violates, in order, but a non-violated rule coming “before” a violated one doesn’t prevent the violation.”
There is something else to mention: the “Transaction Quota Rules” table contains an inherited field “Overrides” (sys_overrides), which is hidden by default. The tooltip for the field states, “Rule being overridden by the current record.” It sounds very promising, but unfortunately it has nothing to do with this. I also asked about that field from ServiceNow Support:
“Like order, the sys_overrides field is inherited from sysrule and has different semantics in other tables. It’s for Domain Separation. It seems to be especially significant in sys_choice, sys_ui_section, and other application-level metadata. However, it has no role in the quota manager.”
In conclusion the solutions to my problem were either:
1. Limit the number of rows per call when consuming the API
2. Modify the OOB rule:
- Either raise the timeout limit for all
- Exclude my integration user from the OOB rule and create a separate rule which will target the integration user as explained in a KB article.
These tests were performed in build glide-washingtondc-12-20-2023__patch4-hotfix1-06-10-20241.
Example API call:
https://[instance].service-now.com/api/now/table/incident?sysparm_display_value=all&sysparm_exclude_reference_link=false&sysparm_suppress_pagination_header=false&sysparm_fields=parent%2Ccaused_by%2Cwatch_list%2Cupon_reject%2Csys_updated_on%2Corigin_table%2Capproval_history%2Cskills%2Cnumber%2Cstate%2Csys_created_by%2Cknowledge%2Corder%2Cdelivery_plan%2Ccmdb_ci%2Cimpact%2Ccontract%2Cactive%2Cwork_notes_list%2Cpriority%2Csys_domain_path%2Cgroup_list%2Cbusiness_duration%2Capproval_set%2Cneeds_attention%2Cuniversal_request%2Cshort_description%2Ccorrelation_display%2Cdelivery_task%2Cwork_start%2Cadditional_assignee_list%2Cnotify%2Cservice_offering%2Csys_class_name%2Cfollow_up%2Cclosed_by%2Cparent_incident%2Creopened_by%2Creassignment_count%2Cassigned_to%2Csla_due%2Ccomments_and_work_notes%2Cescalation%2Cupon_approval%2Ccorrelation_id%2Cmade_sla%2Cchild_incidents%2Chold_reason%2Ctask_effective_number%2Cresolved_by%2Csys_updated_by%2Cuser_input%2Copened_by%2Csys_created_on%2Csys_domain%2Croute_reason%2Ccalendar_stc%2Cclosed_at%2Cbusiness_service%2Cbusiness_impact%2Crfc%2Ctime_worked%2Cexpected_start%2Copened_at%2Cwork_end%2Creopened_time%2Cresolved_at%2Ccaller_id%2Csubcategory%2Cwork_notes%2Cclose_code%2Cassignment_group%2Cbusiness_stc%2Ccause%2Cdescription%2Corigin_id%2Ccalendar_duration%2Cclose_notes%2Csys_id%2Ccontact_type%2Cincident_state%2Curgency%2Cproblem_id%2Ccompany%2Cactivity_due%2Caction_status%2Cseverity%2Ccomments%2Capproval%2Cdue_date%2Csys_mod_count%2Creopen_count%2Csys_tags%2Clocation%2Ccategory