See our latest leap: End-to-End SAP Job Monitoring and Management
Job Monitoring in SAP is probably as old of an issue as monitoring SAP itself! See our post on What to Monitor When You're Not Monitoring SAP.
The nature of background jobs is such that they need to be monitored and there are several ways to do so:
1. Monitoring by exception (Alerts)
This is a natural SAP method - when the job completes and an alert is generated. Green alert for success and Red for failure. Simple enough. This probably works in 80% of cases, let alone the way who and how is notified. Unfortunately, SAP is a bit of a let-down here - some jobs are critical and the failure notification should go to the operations team, and some are not and are the responsibility of the users who ran them. Then there is a question of how - sending to an e-mail? opening a ticket in the Service Desk? Last but not least, what do you do in response? Cry? Laugh? Restart the job? Fix the problem?
2. Hitting the refresh button
This method is often applied to critical jobs - keep checking until it ends - then do what you may with the results. In SAP it means running SM37 and hitting refresh every few minutes, when it failed you can click on the job and dig into the logs, to figure out the problem...
It all sounds natural and familiar... maybe annoying...a lot ... and we keep missing the alerts and don't have time to look at SM37... and job runs get deleted and we lost the forensic for the troubleshooting...
Tools
Here come the tools to take the annoyance away. I will leave off the so-called Job Scheduling tools as they are about the complete automation of background jobs. These tools, in essence, extend SAP's own Background Processing and while solving many limitations introduce its own problems. In any case, let's leave it to the CIOs and CFOs to worry about hiring for the job scheduling team and financing multi-million license payments and focus on what worries the Basis and Operations teams on the ground - Monitoring Tools.
1. CCMS
Yes the good old CCMS that is being retired... for the last 10 years... without success...
There is a little-known facility in CCMS that allows you to monitor the jobs - you define the Job Name criteria and CCMS will monitor the matching job statuses and duration. If you use RZ20 you can look at the history and associated alerts, it will do long-running monitoring as well, just define the threshold in CCMS and it will generate alerts. Sounds easy, only the reality is a bit more complicated: The configuration interface is archaic, to say the least - you need to run SE16 and edit a table ALBTCMON. If you are activating it for the first time there is even more esoteric stuff to do, culminating in "Reset Segment to WARMUP Status". See here for the complete details: https://help.sap.com/saphelp_nw70/helpdata/en/1c/48803d48de0610e10000000a114084/content.htm
After you have spent your lunchtime configuring and ready to roll, you find out that it acts somewhat strange - all jobs that matched the Job Name criteria you defined are lumped together and the last job's info overrides the previous one... Say you define criteria and named it FI_MonthEnd to include all jobs with the Name FI_MonthEnd_* - the CCMS will create an MTE (Tree node) with the name FI_MonthEnd which will aggregate all the FI_MonthEnd_* jobs ... not good! Now you need to fire up SE16 again and define separate criteria for each FI_MonthEnd_* job so you know which one is running too long and which one failed... you may end up with hundreds of records in the ALBTCMON table... don't forget to back it up!
This feature has been around for a while but SAP never improved it, not even a better configuration interface... so I would not bet on any improvements in the next 10 years. Time to move on - join the 21st century:
2. SAP Solution Manager
This thing is solid! Install, Patch, Patch ... Patch...Configure Technical Monitoring (Job Monitoring Scenario) and a few weeks later you have the most complete SAP Job monitoring!
https://service.sap.com/~sapidb/011000358700000012372014E/Job_Monitoring_Setup_Guide.pdf
3. External Monitoring Tools
Enterprise-wide monitoring does not run on SAP, there are thousands of servers and applications that are already monitored and the monitoring team will not use Solution Manager, but rather something else, homegrown or off-the-shelf from the vendors big and small. Some of the vendor tools cover SAP monitoring. How decent is the SAP monitoring part? This actually has not been improving ...quite an opposite... with the exception of the few focused players ... like OZSoft-Consulting and IT-Conductor.
OZSoft-Consulting offers SAP Management Pack for System Center Operations Managers. This is a plug-in for Microsoft's enterprise-wide monitoring tool a.k.a. SCOM.
OZSoft incorporated a lot of monitoring features in its Management Pack and covers most of the requirements, including Job Monitoring.
The SAP Management Pack Batch Monitoring facility does pretty much everything
1. You define the job criteria
The beauty of it is that you can use a wild card in each column and have multiple criteria.
2. The MP will start pulling the info on all jobs matching one of the criteria
Unlike CCMS these are the actual individual jobs, although it only keeps one entry for all JobCounts, so periodic jobs will not create too many entries.
3. For each job several Performance Counters are pulled
And they are Monitored by SCOM - exceeding the threshold will change the state of the Job object to Red and optionally generate an Alert.
Of course, you can override the threshold either for an individual job, SAP system, or a group of objects (these can be manual or dynamic with selection criteria).
IT-Conductor represents a different take on batch monitoring:
1. It relies on the Alerts, generated off the CCMS Alerts for the exceptions and notifications while taking a more analytical approach to job performance
You configure the job criteria including the Name, Group, and User name all taking the Regex and IT-Conductor starts collecting the matching jobs history which you can visualize and apply thresholds:
You can zoom in and see the spread of the Runtime.
Click on the green bars or the numbers in the table to see what jobs were running in this time interval:
Click further to see the job details.
2. IT-Conductor lets you create targeted Job monitors
Choose your criteria to target the critical jobs and let the user subscribe to them to get notified if the performance is a problem. All of IT-Conductor monitors can be hooked up to the monitoring grid:
Here all of your system(s) health is at your fingertips!
Key Takeaway
I hope this blog post provided you with some 'background' on job monitoring and options available to you along with the features/benefits of each. Please feel free to comment and share your thoughts, challenges, and concerns - we created IT-Conductor to make IT folks' life easier! If IT-Conductor sounds interesting go ahead and sign up for a free trial:
You can find more info on OZSoft SAP Management Pack here: http://www.ozsoft-consulting.com/mp4sap/